Support parallel indexing #849

JoaquinPolonuer · 2025-02-04T18:31:54Z

No description provided.

… from manifest if available

… test to support concurrency

paperqa/agents/search.py

tests/test_agents.py

This reverts commit 6d76d53, reversing changes made to aa29496.

paperqa/agents/search.py

.gitignore

main.py

paperqa/settings.py

paperqa/agents/search.py

…d unnecessary files

…aper-qa into parallel_indexing

paperqa/agents/search.py

tests/test_agents.py

Co-authored-by: James Braza <[email protected]>

…aper-qa into parallel_indexing

paperqa/agents/search.py

tests/test_agents.py

…aper-qa into parallel_indexing

paperqa/agents/search.py

…aper-qa into parallel_indexing

paperqa/agents/search.py

Co-authored-by: James Braza <[email protected]>

jamesbraza

LGTM, excellent work here and thanks for working through all the comments

paperqa/agents/search.py

Co-authored-by: James Braza <[email protected]>

JoaquinPolonuer · 2025-02-10T20:57:26Z

Thank you!!

JoaquinPolonuer added 4 commits February 4, 2025 09:08

cache writer

aa29496

Added the get_manifest_kwargs function to get more than doi and title…

b1aec30

… from manifest if available

Merge branch 'use_manifest_kwargs' into parallel_indexing

6d76d53

Make read_docs async, added release_lock before indexing and adjusted…

2c7489a

… test to support concurrency

dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Feb 4, 2025

[pre-commit.ci lite] apply automatic fixes

2f10564

JoaquinPolonuer commented Feb 4, 2025

View reviewed changes

paperqa/agents/search.py Outdated Show resolved Hide resolved

JoaquinPolonuer commented Feb 4, 2025

View reviewed changes

tests/test_agents.py Outdated Show resolved Hide resolved

JoaquinPolonuer force-pushed the parallel_indexing branch 2 times, most recently from 296f6c7 to 2f10564 Compare February 4, 2025 19:08

Revert "Merge branch 'use_manifest_kwargs' into parallel_indexing"

0486f04

This reverts commit 6d76d53, reversing changes made to aa29496.

JoaquinPolonuer commented Feb 4, 2025

View reviewed changes

paperqa/agents/search.py Outdated Show resolved Hide resolved

jamesbraza reviewed Feb 4, 2025

View reviewed changes

JoaquinPolonuer and others added 10 commits February 4, 2025 11:43

keep previous default concurrency

1c69418

handle any exception when building the index

af50ce7

Add concurrency speed test

821ebda

[pre-commit.ci lite] apply automatic fixes

26ada26

added batch size to settings, changed commit to checkpoint and remove…

6e549c7

…d unnecessary files

Merge branch 'parallel_indexing' of https://github.com/Future-House/p…

f991395

…aper-qa into parallel_indexing

[pre-commit.ci lite] apply automatic fixes

c8cb4c3

no need to wipe the searcher

f070a48

Merge branch 'parallel_indexing' of https://github.com/Future-House/p…

d706910

…aper-qa into parallel_indexing

use counter instead of global

1d1c3c7

JoaquinPolonuer commented Feb 4, 2025

View reviewed changes

paperqa/agents/search.py Outdated Show resolved Hide resolved

[pre-commit.ci lite] apply automatic fixes

9433e81

jamesbraza reviewed Feb 4, 2025

View reviewed changes

paperqa/agents/search.py Show resolved Hide resolved

paperqa/agents/search.py Outdated Show resolved Hide resolved

paperqa/agents/search.py Outdated Show resolved Hide resolved

JoaquinPolonuer and others added 3 commits February 4, 2025 14:24

added too-many-public-methods to toml's disable

348a8f8

resolved comments

320e49f

[pre-commit.ci lite] apply automatic fixes

60a4e10

jamesbraza reviewed Feb 5, 2025

View reviewed changes

paperqa/agents/search.py Outdated Show resolved Hide resolved

tests/test_agents.py Outdated Show resolved Hide resolved

JoaquinPolonuer and others added 4 commits February 5, 2025 08:04

Update paperqa/agents/search.py

42e384f

Co-authored-by: James Braza <[email protected]>

use mock_save_index instead of increment count

85f5e03

Merge branch 'parallel_indexing' of https://github.com/Future-House/p…

4d8f284

…aper-qa into parallel_indexing

[pre-commit.ci lite] apply automatic fixes

e51114b

jamesbraza reviewed Feb 5, 2025

View reviewed changes

paperqa/agents/search.py Outdated Show resolved Hide resolved

tests/test_agents.py Outdated Show resolved Hide resolved

JoaquinPolonuer and others added 8 commits February 7, 2025 14:58

raise exception if not one of ValueError or ImpossibleParsingError

0100564

[pre-commit.ci lite] apply automatic fixes

4ae9065

raise e instead of Exception(e)

9b162f7

[pre-commit.ci lite] apply automatic fixes

faa63ba

removed unnecessary ()

771ce69

added save_index() to delete_document

c59d5bb

Merge branch 'parallel_indexing' of https://github.com/Future-House/p…

cbdeb79

…aper-qa into parallel_indexing

[pre-commit.ci lite] apply automatic fixes

d16e1c8

jamesbraza reviewed Feb 7, 2025

View reviewed changes

paperqa/agents/search.py Outdated Show resolved Hide resolved

paperqa/agents/search.py Outdated Show resolved Hide resolved

paperqa/agents/search.py Show resolved Hide resolved

JoaquinPolonuer added 6 commits February 7, 2025 15:35

I was missing an async

4ffbf85

Merge branch 'parallel_indexing' of https://github.com/Future-House/p…

bfc4553

…aper-qa into parallel_indexing

removing unnecessary parameter

fcd6e22

no need to check writer before commiting

9b2de50

improve documentation in checkpoint function

d068698

comments on silencing exceptions

80a3c4b

jamesbraza reviewed Feb 10, 2025

View reviewed changes

paperqa/agents/search.py Outdated Show resolved Hide resolved

paperqa/agents/search.py Outdated Show resolved Hide resolved

JoaquinPolonuer and others added 2 commits February 10, 2025 08:38

Update paperqa/agents/search.py

fc36fc1

Co-authored-by: James Braza <[email protected]>

use context manager to encapsulate writer logic

543b4f6

jamesbraza approved these changes Feb 10, 2025

View reviewed changes

paperqa/agents/search.py Show resolved Hide resolved

dosubot bot added the lgtm This PR has been approved by a maintainer label Feb 10, 2025

Update paperqa/agents/search.py

25b1a23

Co-authored-by: James Braza <[email protected]>

JoaquinPolonuer merged commit f8d364a into main Feb 10, 2025
5 checks passed

JoaquinPolonuer deleted the parallel_indexing branch February 10, 2025 21:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support parallel indexing #849

Support parallel indexing #849

JoaquinPolonuer commented Feb 4, 2025

jamesbraza left a comment

JoaquinPolonuer commented Feb 10, 2025

Support parallel indexing #849

Support parallel indexing #849

Conversation

JoaquinPolonuer commented Feb 4, 2025

jamesbraza left a comment

Choose a reason for hiding this comment

JoaquinPolonuer commented Feb 10, 2025