Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for shards in SOLR #620

Closed
jnioche opened this issue Oct 1, 2018 · 7 comments · Fixed by #1343
Closed

Add support for shards in SOLR #620

jnioche opened this issue Oct 1, 2018 · 7 comments · Fixed by #1343
Assignees
@mvolikas
Labels
enhancement good first issue help wanted SOLR

Comments

@jnioche
Copy link
Contributor

jnioche commented Oct 1, 2018 •
edited
Loading

Just like it's done in ES, we could route the documents in the statusupdaterbolt based on the host / name or IP and in the spouts check that the number of instances is equal to the # of shards and filter the queries per shard accordingly.

At the moment, we can have only one instance of a spout.
@jnioche
Copy link
Contributor Author

jnioche commented Jun 21, 2022

https://solr.apache.org/guide/solr/latest/deployment-guide/solrcloud-shards-indexing.html#document-routing

@mvolikas mvolikas mentioned this issue Jun 9, 2024
Closed
@mvolikas
Copy link
Contributor

This one is really interesting. Will it be up to the user to correctly create the status core with the same number of shards as the parallelism: *** used in the crawler.flux?

@jnioche
Copy link
Contributor Author

jnioche commented Jun 24, 2024

This one is really interesting. Will it be up to the user to correctly create the status core with the same number of shards as the parallelism: *** used in the crawler.flux?

yes. we should set it to a reasonable default value (10?) but then it is up to the user to manage it

@mvolikas mvolikas mentioned this issue Jun 26, 2024
Merged
@mvolikas
Copy link
Contributor

mvolikas commented Jul 26, 2024 •
edited
Loading

@jnioche can we merge branch 851 into main at this point? This way we can update / add tests along with the new functionality.

@jnioche
Copy link
Contributor Author

jnioche commented Jul 26, 2024 •
edited
Loading

Hasn't that been done in #1240? Is there more in that branch that hasn't been merged? If so, would you mind creating a PR from that branch? Thanks!

@mvolikas
Copy link
Contributor

mvolikas commented Aug 4, 2024

Hasn't that been done in #1240? Is there more in that branch that hasn't been merged? If so, would you mind creating a PR from that branch? Thanks!

The changes from #1240 were merged into branch apache:851 but not into main. Should I open an additional PR from apache:851 to main?

@jnioche
Copy link
Contributor Author

jnioche commented Aug 27, 2024

Hi @mvolikas - sorry about the delayed response, I have just returned from holidays

Should I open an additional PR from apache:851 to main?

yes please, that would be great

@mvolikas mvolikas self-assigned this Sep 29, 2024
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Oct 5, 2024
@mvolikas
apache#620 update spout to fetch from the corresponding shard
7cd06f9
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Oct 5, 2024
@mvolikas
apache#620 add Solr scripts
e33c2bb
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Oct 13, 2024
@mvolikas
apache#620 fix tests to operate in cloud mode
8ebd0d1
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Oct 13, 2024
@mvolikas
apache#620 fix code format
b5bd556
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Oct 14, 2024
@mvolikas
apache#620 add Solr spout test
e1ee25d
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Oct 14, 2024
@mvolikas
apache#620 add license
27d71c9
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Oct 14, 2024
@mvolikas
apache#620 improve the Solr related scripts
d22c325
@mvolikas mvolikas mentioned this issue Oct 18, 2024
Merged
3 tasks
@mvolikas mvolikas linked a pull request Oct 18, 2024 that will close this issue
#620 Add support for shards - SolrSpout #1343
Merged
3 tasks
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Oct 27, 2024
@mvolikas
apache#620 add solr archetype, update readmes
e7add5e
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Nov 2, 2024
@mvolikas
apache#620 minor fixes
49f4556
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Nov 2, 2024
@mvolikas
apache#620 do not set the 'shard' query parameter when we have a sing…
686a83a 
…le shard
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Nov 3, 2024
@mvolikas
apache#620 fix archetype includes, improve scripts and configuration …
c581b91 
…files
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Nov 3, 2024
@mvolikas
apache#620 fix java topologies
291013e
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Nov 3, 2024
@mvolikas
apache#620 add 'injection.flux' topology
99b01cd
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Nov 9, 2024
@mvolikas
apache#620 bring in change from apache#1390
6cdffd6
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Nov 9, 2024
@mvolikas
apache#620 update sample flux topologies and readme
6532dc8
mvolikas added a commit to mvolikas/incubator-stormcrawler that referenced this issue Nov 9, 2024
@mvolikas
apache#620 minor comments and readme changes
7640e01
jnioche pushed a commit that referenced this issue Nov 10, 2024
@mvolikas
#620 Add support for shards - SolrSpout (#1343)
f53e89a 
* #620 update spout to fetch from the corresponding shard

* #620 add Solr scripts

* #620 fix tests to operate in cloud mode

* #620 fix code format

* #620 add Solr spout test

* #620 add license

* #620 improve the Solr related scripts

* #620 add solr archetype, update readmes

* #620 minor fixes

* #620 do not set the 'shard' query parameter when we have a single shard

* #620 fix archetype includes, improve scripts and configuration files

* #620 fix java topologies

* #620 add 'injection.flux' topology

* #620 bring in change from #1390

* #620 update sample flux topologies and readme

* #620 minor comments and readme changes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mvolikas mvolikas

Labels
enhancement good first issue help wanted SOLR
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

#620 Add support for shards - SolrSpout mvolikas/incubator-stormcrawler
2 participants
@jnioche @mvolikas