Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add forge methods to acces/query datasets from external sources #367

Draft
wants to merge 50 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
cc8b557
Add web service
crisely09 Dec 12, 2023
f6f853e
More changes
crisely09 Dec 13, 2023
47e8464
Clean code and add tests
crisely09 Dec 13, 2023
2ff9d7d
Added methods for forge and configuration file
crisely09 Dec 14, 2023
380eb07
Load local configuration when configuration tests (#370)
ssssarah Jan 10, 2024
b173780
Pass view when forge.sparql, forge.elastic, forge.search call (#373)
ssssarah Feb 2, 2024
16e1d94
filter exception fix (#375)
ssssarah Feb 2, 2024
5fc964c
added timeouts to every requests. call (#368)
ssssarah Feb 6, 2024
caab280
resolving returns also the hasleafregionpart attribute of the resource
NicoRicardi Feb 5, 2024
d6388b8
deflatten is independent of order of items. If both prop and prop.som…
NicoRicardi Feb 8, 2024
e358acf
removed prints and fixed linting
NicoRicardi Feb 9, 2024
9ca0398
comments for clarity as suggested by Sarah Mouffok
NicoRicardi Feb 9, 2024
922bfd3
previously to deflatten, we check all columns names are strings
NicoRicardi Feb 12, 2024
659bc1d
Update resource schema (#374)
ssssarah Feb 12, 2024
76e2f7e
Rm metadata fetch when forge.retrieve (#376)
ssssarah Feb 12, 2024
41503dc
re-do metadata fetch until endpoint is fixed (#381)
ssssarah Feb 13, 2024
647912a
return resources as dictionaries when forge.elastic if as_resource =…
ssssarah Feb 19, 2024
c38b3c4
Set jsonld context version to 1.1 (#387)
MFSY Feb 23, 2024
a050dc0
Use as default Aggregate SPARQL view (#391)
crisely09 Mar 5, 2024
d6f38de
reset resolver sparql view to default, change store sparql view to ag…
ssssarah Mar 5, 2024
ede675a
Fixes pyshacl deep nodeshape path eval error (#393)
MFSY Apr 2, 2024
6b666f1
Fixed coupled of issues (#394)
MFSY Apr 3, 2024
85d6c6f
improved deflatten function (lower time complexity)
NicoRicardi Apr 11, 2024
a36f42c
linting
NicoRicardi Apr 11, 2024
baf26f0
Validation with ontology (#396)
MFSY Apr 19, 2024
f12b465
Increased request timeout constant (#400)
MFSY Apr 25, 2024
7342b58
Propagated inference value to model when using forge.validate (#399)
ssssarah Apr 25, 2024
f31e6cc
Expand jsonld context in load_resource_graph_from_source (#401)
MFSY Apr 25, 2024
8106ed8
Added AttributeError exception when a None resolver is provided to re…
MFSY May 3, 2024
29280b6
Add alternateName to agent resolver (#404)
crisely09 May 14, 2024
3d8a37c
Add `add_image` method to `Dataset` entity (#389)
crisely09 May 21, 2024
8db6ae8
Update prod-forge-nexus.yml
NicoRicardi May 21, 2024
eb6a56d
now it works. (has_rank.id and expanded NCBITaxon)
NicoRicardi May 21, 2024
89634dd
example of resolving with strain ResolvingStrategies.ipynb
NicoRicardi May 21, 2024
c5538fe
Add method when initializing forge to export pyshacl environment vari…
crisely09 May 24, 2024
c895b7c
Remove nexus_sdk from nexus store when uploading files and add `x-nxs…
crisely09 May 31, 2024
9476d95
File fetch (#408)
ssssarah May 31, 2024
536995b
rm nexus sdk usage and dependency (#407)
ssssarah Jun 1, 2024
c198389
Fix org and project metadata (#409)
crisely09 Jun 14, 2024
559e3bd
Added v0.8.2 release notes (#410)
MFSY Jun 18, 2024
aa0cb9a
Fix readthedocs yml connfig error: Value build not found (#411)
MFSY Jun 18, 2024
50130ec
Fixed building a SPARQL query from a Filter with __ne__ operator and …
ssssarah Jun 19, 2024
0b3b91f
Store initialization configuration in forge._config method (#333)
crisely09 Jun 19, 2024
7e3844b
Refactor batch request mechanism (#383)
ssssarah Jun 20, 2024
7848a54
rewrite uri as static method (#413)
ssssarah Jun 21, 2024
42b6f8c
Check for schema id to use schema endpoint (#415)
crisely09 Jun 28, 2024
6777f70
Use schema_id in prepare_update (#416)
crisely09 Jun 28, 2024
3e26c2e
Merge master in dev-datasets
MFSY Jul 5, 2024
e5fcf75
Merge branch 'master' into dev-datasets
MFSY Jul 5, 2024
b1f8d09
Fixed tox errors
MFSY Jul 9, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 98 additions & 0 deletions examples/configurations/dataset-store/prod_nexus_datasets.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
Model:
name: RdfModel
origin: store
source: BlueBrainNexus
context:
iri: "https://bbp.neuroshapes.org"
bucket: "neurosciencegraph/datamodels"
Store:
name: BlueBrainNexus
endpoint: https://bbp.epfl.ch/nexus/v1
model:
name: RdfModel
searchendpoints:
sparql:
endpoint: "https://bluebrain.github.io/nexus/vocabulary/defaultSparqlIndex"
elastic:
endpoint: "https://bbp.epfl.ch/neurosciencegraph/data/views/aggreg-es/dataset"
mapping: "https://bbp.epfl.ch/neurosciencegraph/data/views/es/dataset"
default_str_keyword_field: "keyword"
vocabulary:
metadata:
iri: "https://bluebrain.github.io/nexus/contexts/metadata.json"
local_iri: "https://bluebrainnexus.io/contexts/metadata.json"
namespace: "https://bluebrain.github.io/nexus/vocabulary/"
deprecated_property: "https://bluebrain.github.io/nexus/vocabulary/deprecated"
project_property: "https://bluebrain.github.io/nexus/vocabulary/project"
max_connection: 50
versioned_id_template: "{x.id}?rev={x._store_metadata._rev}"
file_resource_mapping: https://raw.githubusercontent.com/BlueBrain/nexus-forge/master/examples/configurations/nexus-store/file-to-resource-mapping.hjson

Resolvers:
ontology:
- resolver: OntologyResolver
origin: store
source: BlueBrainNexus
targets:
- identifier: terms
bucket: neurosciencegraph/datamodels
searchendpoints:
sparql:
endpoint: "https://bluebrain.github.io/nexus/vocabulary/defaultSparqlIndex"
result_resource_mapping: https://raw.githubusercontent.com/BlueBrain/nexus-forge/master/examples/configurations/nexus-resolver/term-to-resource-mapping.hjson
agent:
- resolver: AgentResolver
origin: store
source: BlueBrainNexus
targets:
- identifier: agents
bucket: bbp/agents
searchendpoints:
sparql:
endpoint: "https://bluebrain.github.io/nexus/vocabulary/defaultSparqlIndex"
result_resource_mapping: https://raw.githubusercontent.com/BlueBrain/nexus-forge/master/examples/configurations/nexus-resolver/agent-to-resource-mapping.hjson

Formatters:
identifier: https://bbp.epfl.ch/neurosciencegraph/data/{}/{}
identifier_neuroelectro: http://neuroelectro.org/api/1/{}/{}
identifier_neurolex: http://neurolex.org/wiki/{}

Datasets:
UniProt:
origin: store
source: SPARQLStore
searchendpoints:
sparql:
endpoint: "https://sparql.uniprot.org/sparql"
model:
name: RdfModel
origin: directory
source: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/UniProt
context:
iri: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/UniProt/jsonld_context.json
NeuroElectro:
origin: store
source: BlueBrainNexus
bucket: bbp/neuroelectro
model:
name: RdfModel
origin: directory
source: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/NeuroElectro
context:
iri: /Users/cgonzale/Documents/code/nexus-forge/examples/models/neuroshapes_context.json
NeuroMorpho:
origin: web_service
source: http://neuromorpho.org/api/neuron
health: http://neuromorpho.org/api/health
files_download:
endpoint: https://neuromorpho.org/dableFiles
Accept: "*/*"
max_connection: 50
content_type: application/json;charset=UTF-8
accept: "*/*"
model:
name: RdfModel
origin: directory
source: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/Neuromorpho
context:
iri: /Users/cgonzale/Documents/code/nexus-forge/examples/database-sources/Neuromorpho/jsonld_context.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
NeuronMorphology
Entity
]
id: forge.format('identifier', 'neuronmorphologies/neuromorpho', f'{x.bbpID}')
brainLocation: {
type: BrainLocation
brainRegion: forge.resolve(x.brain_region[0], scope='ontology', strategy='EXACT_CASE_INSENSITIVE_MATCH')
Expand Down
Loading
Loading