LLM scraper #971
LLM scraper #971
Conversation
| add_reference :events, :llm_object, foreign_key: true | ||
| add_column :events, :open_science, :string, array: true, default: [] | ||
| add_column :materials, :llm_processed, :bool, default: false |
There was a problem hiding this comment.
removed materials for now.
lib/modules/llm_service.rb
Outdated
| def post_process_func(event) | ||
| response = process(event) | ||
| event = unload_json(event, response) | ||
| event.llm_object_attributes = llm_object_attributes |
There was a problem hiding this comment.
This overwrites the data from the "scrape" LLM interaction, right? Is there ever a need to retain the data for both?
There was a problem hiding this comment.
Currently not I think. I'd like to still have data for the scrape in case we ever decide not to do post processing for whatever reason
|
Will try and review this next week. Had a busy week due to various projects wrapping up at the end of June |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
test/test_helper.rb
Outdated
| @@ -233,22 +235,22 @@ | |||
|
|
|||
| def mock_biotools | |||
| biotools_file = File.read("#{Rails.root}/test/fixtures/files/annotation.json") | |||
| WebMock.stub_request(:get, /data.bioontology.org/). | |||
| to_return(:status => 200, :headers => {}, :body => biotools_file) | |||
| WebMock.stub_request(:get, /data.bioontology.org/) | |||
Check failure
Code scanning / CodeQL
Incomplete regular expression for hostnames High test
test/test_helper.rb
Outdated
|
|
||
| WebMock.stub_request(:get, /nominatim.openstreetmap.org/). | ||
| to_return(:status => 200, :headers => {}, :body => nominatim_file) | ||
| WebMock.stub_request(:get, /nominatim.openstreetmap.org/) |
Check failure
Code scanning / CodeQL
Incomplete regular expression for hostnames High test
Summary of changes
Added functionality for using automated LLM scraping and metadata processing
Motivation and context
Robustness, metadata parsing, (potential) deduplication, collection management
Checklist
to license it to the TeSS codebase under the
BSD license.