Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow OpenNeuro setup from alternative GitHub urls for important datasets like ds003194 #287

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Allow OpenNeuro setup from alternative GitHub urls for important datasets like ds003194 #287

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
088f8c9
use GraphQL to check snapshot
MontrealSergiy Jun 14, 2024
a97172f
use GraphQL to check OpenNeuro snapshot - add headers
MontrealSergiy Jun 14, 2024
b5f94e2
Update open_neuro.rb copyright year
MontrealSergiy Jun 16, 2024
8f3afa0
change comments for name and version validation method
MontrealSergiy Jun 17, 2024
5443756
Allow alternative GitHub repos for OpenNeuro
MontrealSergiy Jun 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 42 additions & 8 deletions openneuro/app/models/open_neuro.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
#
# CBRAIN Project
#
# Copyright (C) 2008-2023
# Copyright (C) 2008-2024
# The Royal Institution for the Advancement of Learning
# McGill University
#
Expand Down Expand Up @@ -38,8 +38,10 @@ class OpenNeuro
DATA_PROVIDER_OWNER = User.admin
USERFILES_OWNER = User.admin

DATALAD_REPO_URL_PREFIX = 'https://github.com/OpenNeuroDatasets'
GITHUB_VALIDATION_URL = 'https://api.github.com/repos/OpenNeuroDatasets/:name/git/ref/tags/:version'
DATALAD_REPO_URL_PREFIX = 'https://github.com/OpenNeuroDatasets'
DATALAD_ALT_URL_PREFIXES = ['https://github.com/MontrealSergiy', 'https://github.com/aces']
GITHUB_VALIDATION_URL = 'https://api.github.com/repos/OpenNeuroDatasets/:name/git/ref/tags/:version'
OPENNEURO_API_URL = 'https://openneuro.org/crn/graphql'

# Creates an OpenNeuro object that represents
# the dataset internally as a pair, a WorkGroup
Expand All @@ -57,7 +59,7 @@ def self.find(name, version)
groupquery = work_group_builder(name, version)
group = groupquery.first
dpquery = data_provider_builder(name, version, group&.id) # group.id can be nil
dp = dpquery.first
dp = data_provider_builder(name, version, group&.id, alt_urls = true).first

self.new.tap do |open_neuro|
open_neuro.name = name
Expand Down Expand Up @@ -282,12 +284,14 @@ def self.work_group_builder(name, version)

# Returns a DataladDataProvider fetcher or constructor
# representing an OpenNeuro dataset. (ActiveRecord query)
def self.data_provider_builder(name, version, group_id)
def self.data_provider_builder(name, version, group_id, alt_urls=false)
urls = "#{DATALAD_REPO_URL_PREFIX}/#{name}"
urls = DATALAD_ALT_URL_PREFIXES.map {|p| "#{p}/#{name}"}.unshift urls if alt_urls
DataladDataProvider.where(
:name => data_provider_name_builder(name, version),
:user_id => DATA_PROVIDER_OWNER.id,
:group_id => group_id,
:datalad_repository_url => "#{DATALAD_REPO_URL_PREFIX}/#{name}",
:datalad_repository_url => urls,
:containerized_path => version,
:online => true,
)
Expand All @@ -309,7 +313,37 @@ def self.data_provider_name_builder(name, version)
"OpenNeuro-#{name}-#{version.gsub('.','_')}"
end

# Validation of a pair [ dataset, version ] performed with:
# Validation of a pair [ dataset, version ] with OpenNeuro GraphQL API
#
# The result is sometimes different
# "https://api.github.com/repos/OpenNeuroDatasets/
#
# Some datasets or dataset versions, availabe on OpenNeuro portal or with OpenNeuro client are not always
# on GitHub
#
# At the moment the method just returns true or false.
def self.valid_name_and_version?(name, version)
return false unless name =~ /\Ads\d+\z/
return false unless version =~ /\A[a-z0-9][\w\.\-]+\z/
query = '{snapshot(datasetId:"%s", tag:"%s"){id}}' % [name, version]

response = Typhoeus.post(OPENNEURO_API_URL,
:body => JSON.pretty_generate({ "query" => query }),
:headers => { 'Content-Type' => 'application/json',
'Accept' => 'application/json'
}
)

# Parse the response
body = response.response_body
json = JSON.parse(body)
return !json["errors"]
rescue => ex
Rails.logger.error "OpenNeuro API request failed: #{ex.class} #{ex.message}"
return nil
end

# Validation of a pair [ dataset, tag ] on github performed with:
#
# curl -H "Accept: application/vnd.github+json"
# -H "X-GitHub-Api-Version: 2022-11-28"
Expand All @@ -329,7 +363,7 @@ def self.data_provider_name_builder(name, version)
# }
#
# The method just returns true or false.
def self.valid_name_and_version?(name, version)
def self.valid_name_and_github_tag?(name, version)
return false unless name =~ /\Ads\d+\z/
return false unless version =~ /\A[a-z0-9][\w\.\-]+\z/

Expand Down