Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

4.0.0 rc1 #281

Merged
merged 99 commits into from
Dec 19, 2023
Merged
Changes from 8 commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
f445d9e
Update get_eurostat_json.R
ake123 Jun 9, 2023
1adff6f
Update get_eurostat_json.R
ake123 Jun 12, 2023
4f5a54e
Update get_eurostat_json.R
ake123 Jun 13, 2023
fee43b9
Update get_eurostat_json.R
ake123 Jun 13, 2023
a86d575
Update get_eurostat_json.R
ake123 Jun 14, 2023
6614b12
Update get_eurostat_json
ake123 Jun 16, 2023
b90e669
Update get_eurostat_json.R #262
ake123 Jun 19, 2023
dcad4e1
Merge pull request #262 from ake123/master
pitkant Jun 20, 2023
d74bcb7
Rewrite get_eurostat_json error message handling
pitkant Jun 21, 2023
4bf171d
Remove references to legacy bulk download from functions and tests
pitkant Jun 21, 2023
cfdaf37
Reduce RAM usage when handling big datasets with weekly data
pitkant Jun 28, 2023
cb8ec7e
Typo fix
pitkant Jun 28, 2023
51624ad
Migrate to use giscoR
dieghernan Jul 1, 2023
20b0c32
Finish R code and actions
dieghernan Jul 2, 2023
55134fe
Update action versions
dieghernan Jul 2, 2023
3d98b72
Delete docs
dieghernan Jul 2, 2023
ad7c8c9
Update NEWS.md
dieghernan Jul 2, 2023
756cf9f
Adjust messages and tests
dieghernan Jul 3, 2023
3a6e48f
Make columns on spatial output consistent
dieghernan Jul 31, 2023
37fce83
Clarify some lines, fix typos
pitkant Jul 31, 2023
46d8a78
Merge pull request #264 from rOpenGov/gisco
pitkant Jul 31, 2023
bec9635
Silence first error message in case of errors
pitkant Aug 2, 2023
49878f8
Rewrite cache functionalities
pitkant Aug 10, 2023
f03d738
Update NAMESPACE, NEWS, DESCRIPTION
pitkant Aug 10, 2023
e2b5999
Using cache filtering only if download method is the same
pitkant Aug 11, 2023
d0df373
Add info on download date to query
pitkant Aug 11, 2023
2f1318b
Update TOC URL
pitkant Aug 11, 2023
f533daa
Add test for new filter behavior
pitkant Aug 14, 2023
19bd764
Reorder slightly for better logic
pitkant Aug 14, 2023
8fb1013
Lint code
pitkant Aug 15, 2023
3ae8337
Update get_bibentry url and replace year field with date
pitkant Aug 15, 2023
7d86040
Revert "Update get_bibentry url and replace year field with date"
pitkant Aug 16, 2023
be2b98d
Merge pull request #267 from rOpenGov/cache
pitkant Aug 17, 2023
61db0b8
Make function usable without sf package
pitkant Aug 17, 2023
e03bdbb
Merge pull request #269 from rOpenGov/cache
pitkant Aug 18, 2023
16142a3
Fix title-code mismatch, add attribution
pitkant Aug 18, 2023
ce5a500
Use c() instead of personList()
pitkant Aug 21, 2023
6efac2f
Warn user for missing items in output
pitkant Aug 22, 2023
e45fa9c
Rephrase warning messages
pitkant Aug 22, 2023
77d0b7a
Edit example, remove EC from authors
pitkant Aug 24, 2023
524c4a5
Use inheriting documentation where possible
pitkant Aug 29, 2023
cbf5ce6
Include NEWS.md to package build
pitkant Aug 29, 2023
ae46605
Add additional argument to search_eurostat()
pitkant Aug 31, 2023
9b22591
Clean TOC names by replacing empty spaces with .
pitkant Aug 31, 2023
8f23a6d
Update documentation
pitkant Aug 31, 2023
4d1f5f9
Add hierarchy column to TOC
pitkant Aug 31, 2023
a5d11bc
Add function for counting number of children in TOC
pitkant Aug 31, 2023
79e0c3a
Add function get_eurostat_folder()
pitkant Sep 1, 2023
c72f1a7
Update documentation on filtering datasets
pitkant Sep 4, 2023
65e29d3
Use new URL for Eurostat dictionaries
pitkant Sep 4, 2023
11634d6
Clarify documentation on filtering
pitkant Sep 4, 2023
d3be394
Explicit deprecation and defunct, fix label_eurostat_tables and label…
pitkant Sep 7, 2023
1fb6f28
Merge pull request #271 from rOpenGov/search_eurostat
pitkant Sep 12, 2023
7597740
Update CITATION
pitkant Sep 12, 2023
a63c141
Fix function to mitigate for changes made in set_eurostat_toc()
pitkant Sep 13, 2023
5949580
Add experimental language support to get_bibentry, fix output
pitkant Sep 14, 2023
7581895
Use check_lang function to enforce correct lang usage
pitkant Sep 15, 2023
740d36d
Test for lang parameter messages
pitkant Sep 15, 2023
1fca371
Enable searching French and German TOC objects
pitkant Sep 15, 2023
5d3d9d1
Test for multilingual search
pitkant Sep 15, 2023
4e5cc17
Test for multilingual bibentry
pitkant Sep 15, 2023
53a6bd2
Update NEWS
pitkant Sep 15, 2023
cecfdf0
Improve tests
pitkant Sep 15, 2023
0ef26d9
Remove *_multilingual functions and use only regular functions
pitkant Sep 18, 2023
0d55890
Remove redundant arguments from set_eurostat_toc
pitkant Sep 20, 2023
38bb289
Add new function: list_cache_items
pitkant Sep 21, 2023
be2de77
Add possibility of returning a list in get_eurostat_folder
pitkant Sep 21, 2023
d81d2d9
Correct journal name in textVersion
pitkant Sep 26, 2023
899963a
Check for missing cache_list.json; export function
pitkant Sep 26, 2023
8d16a39
Fix mention of hard limit in get_eurostat_folder: 10 -> 20
pitkant Sep 26, 2023
c17e0ca
Merge pull request #270 from rOpenGov/get_bibentry
pitkant Sep 27, 2023
5e2e6f3
Re-render README
pitkant Sep 27, 2023
900e433
Switch from httr to httr2
pitkant Sep 27, 2023
fb6352f
Enable proxy use in requests
pitkant Sep 28, 2023
e2598fd
Merge pull request #276 from rOpenGov/httr2
pitkant Sep 29, 2023
1e323cd
Implement data.table functions for handling big objects in memory
pitkant Sep 29, 2023
668a2d8
Update documentation
pitkant Sep 29, 2023
febbea6
Use argument header=TRUE with data.table fread
pitkant Oct 2, 2023
6ad9abb
Include example of use.data.table = TRUE in tests
pitkant Oct 3, 2023
7116503
Set TOC values column as numeric
pitkant Oct 3, 2023
2675efa
Clarify documentation
pitkant Oct 3, 2023
9786a0e
Reduce deep copying, aim to mutate by reference
pitkant Oct 5, 2023
327964a
Fix get/set_eurostat_toc and tests to better handle certain anomalies
pitkant Oct 5, 2023
c2136a9
Require always sf for get_eurostat_geospatial
dieghernan Oct 14, 2023
f68da84
Merge pull request #280 from rOpenGov/use_sf
pitkant Oct 20, 2023
6ac2007
Bump minimum supported version of R
pitkant Oct 20, 2023
cf8a251
Merge pull request #278 from rOpenGov/httr2
pitkant Oct 20, 2023
d22a905
Update test_04_get and testthat snaps
pitkant Oct 31, 2023
2173e65
Add new functions: get_eurostat_interactive and fixity_checksum
pitkant Nov 3, 2023
07d92df
Combine all 4.0.0.900* NEWS items under 4.0.0
pitkant Nov 3, 2023
386a23a
Categorise fixity_checksum under helpers
pitkant Nov 3, 2023
3f412c2
4.0.0 rc1
pitkant Nov 3, 2023
3d42882
Reorder NEWS items, add issue and PR numbers, add attribution
pitkant Nov 3, 2023
2484225
First update to vignette (article) with 4.0.0
pitkant Nov 3, 2023
041afb6
Turn fixity_checksum into internal function
pitkant Nov 6, 2023
e696b91
Merge label_eurostat_vars2 to label_eurostat_vars
pitkant Dec 19, 2023
250ce74
Update news
pitkant Dec 19, 2023
b9ff2b7
Add article explaining difference between old and new dic methods
pitkant Dec 19, 2023
f16e0d1
Merge pull request #282 from rOpenGov/update-tutorial
pitkant Dec 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 54 additions & 20 deletions R/get_eurostat_json.R
Original file line number Diff line number Diff line change
@@ -106,7 +106,7 @@ get_eurostat_json <- function(id,
message("You have no access to ec.europe.eu.
Please check your connection and/or review your proxy settings")
} else {

msg <- " Some datasets are not accessible via the eurostat interface. You can try to search the data manually from the comext database at https://ec.europa.eu/eurostat/comext/newxtweb/ or explore data at https://ec.europa.eu/eurostat/web/main/data/database"
# get response
# url <- try(eurostat_json_url(id = id, filters = filters, lang = lang))
# if (class(url) == "try-error") { stop(paste("The requested data set cannot be found with the following specifications to get_eurostat_json function: ", "id: ", id, "/ filters: ", filters, "/ lang: ", lang)) }
@@ -115,40 +115,74 @@ get_eurostat_json <- function(id,
# resp <- try(httr::GET(url, ...))
# if (class(resp) == "try-error") { stop(paste("The requested url cannot be found within the get_eurostat_json function:", url)) }
resp <- httr::RETRY("GET", url, terminate_on = c(404))

#status <- httr::status_code(resp)
#id <- httr::content(resp)$error[[1]]$id
#label_des <-httr::content(resp)$error[[1]]$label
result <- httr::content(resp)$error
json_data_frame <- as.data.frame(result)
status <- json_data_frame$status
id <- json_data_frame$id
label <- json_data_frame$label
#print(json_data_frame)
if (httr::http_error(resp)) {
stop(paste("The requested url cannot be found within the get_eurostat_json function:", url))
# stop(paste("The requested url cannot be found within the get_eurostat_json function:
# Client Error - 100 No results found
# Description - The requested resource is not available.", url))
#
paste("Status : ",status,
", error id : ",id,"No results found",
", label : ", label,"\n",
msg)
}
}

status <- httr::status_code(resp)

#status <- httr::status_code(resp)
# check status and get json

msg <- ". Some datasets are not accessible via the eurostat
interface. You can try to search the data manually from the comext
database at http://epp.eurostat.ec.europa.eu/newxtweb/ or bulk
download facility at
http://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing
or annual Excel files
http://ec.europa.eu/eurostat/web/prodcom/data/excel-files-nace-rev.2"
# msg <- ". Some datasets are not accessible via the eurostat
# interface. You can try to search the data manually from the comext
# database at http://epp.eurostat.ec.europa.eu/newxtweb/ or bulk
# download facility at
# http://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing
# or annual Excel files
# http://ec.europa.eu/eurostat/web/prodcom/data/excel-files-nace-rev.2"

if (status == 200) {
jdat <- jsonlite::fromJSON(url)
} else if (status == 400) {
stop(
"Failure to get data. Probably invalid dataset id. Status code: ",
status, msg
)

# stop(paste("Status : ",status,"Bad request",
# if(id ==100){paste(", error id : ",id , "No result found")}
# else if(id ==140){paste(", error id : ",id , "Syntax error")}
# else if(id ==150){paste(", error id : ",id , "Semantic error")},
# ", label : ", label,"\n",
# msg))
stop(paste("Status :",status,"Bad request",
if(100 %in% id){paste(" errod id: ", id , "No result found")}
else if(140 %in% id){paste("error id: ",id , "Syntax error",)}
else if(150 %in% id){paste("errod id: ",id , "Semantic error")},label))
} else if (status == 500) {
stop("Failure to get data. Probably filters did not return any data
or data exceeded query size limitation. Status code: ", status, msg)

stop(paste("Status : ",status,"Internal Server error",
", error id : ",id," Internal Server error",
", label : ", label),"\n",
msg)
} else if (status == 416) {
stop(
"Too many categories have been requested. Maximum is 50.",
status, msg
status,
msg
)
} else {
stop("Failure to get data. Status code: ", status, msg)
stop(paste("Failure to get data. Status code: ",
"Status : ",status,
", error id : ",id,"No results found",
", label : ", label,"\n",
msg))
}


# get json data
# dims <- jdat[[1]]$dimension # Was called like this in API v1.1
@@ -188,7 +222,7 @@ get_eurostat_json <- function(id,

tibble::as_tibble(dat)
}
}