[🐞]function loadbigfile(fname)

[MartinHanssonSMHI]

Describe the bug
Hej!

We found a problem with this function as it load the wrong columns as it does not handle empy fields. Our suggestion is to add a \t to make it work. See below:

#Added '\t' otherwise any empty field will make the function load the wrong field. /MHAN o LVIK
for i = 1:nobs
rec = split(data[i], '\t')
lon[i] = parse(Float64, rec[1])
lat[i] = parse(Float64, rec[2])
value[i] = parse(Float64, rec[3])
depth[i] = parse(Float64, rec[4])

Environment

• Julia version: Version 1.6.6 (2022-03-28)
• Operating system: (ex: Windows)

Installed modules
julia> import Pkg; Pkg.status()
Status C:\Users\a001109\.julia\environments\v1.6\Project.toml
[336ed68f] CSV v0.10.14
[efc8151c] DIVAnd v2.7.11
[a93c6f00] DataFrames v1.6.1
[864edb3b] DataStructures v0.18.16
[7073ff75] IJulia v1.24.2
[4138dd39] JLD v0.13.4
[682c06a0] JSON v0.21.4
[e1d29d7a] Missings v1.1.0
[85f8d34a] NCDatasets v0.14.0
[438e738f] PyCall v1.96.4
[d330b81b] PyPlot v2.11.2

To Reproduce
function loadbigfile(fname)

Full screen output
Preferably obtained by setting ENV["JULIA_DEBUG"] = "DIVAnd"

** Full stack trace with error message**
this is wrong due to empy fields in data file: 1900-01-01T00:00:00.0

0.62659 60.62659 1900-01-01T00:00:00.0 11
18.86667 55.84 339.87021000000004 10.40435 10.40435 10.40435 10.40435 10.40435 10.40435 1900-01-01T00:00:00.0 12:40:00
19.76976 56.95182 326.0253 10.4529 10.4529 10.4529 10.4529 10.4529 10.4529 1900-01-01T00:00:00.0 11
19.83 57.0717 337.19055000000003 10.49242 10.49242 10.49242 10.49242 10.49242 10.49242 1900-01-01T00:00:00.0 11
20.05 57.32 335.40411 25.48155 25.48155 25.48155 25.48155 25.48155 25.48155 1900-01-01T00:00:00.0 11
19.9 58.0 286.27701 40.55664 40.55664 40.55664 40.55664 40.55664 40.55664 1900-01-01T00:00:00.0 11
19.9 58.0 207.67365 60.62395 60.62395 60.62395 60.62395 60.62395 60.62395 1900-01-01T00:00:00.0 17:47:00
17.66667 57.11667 243.84906 40.58936 40.58936 40.58936 40.58936 40.58936 40.58936 1900-01-01T00:00:00.0 11
15.98333 55.25 243.40245000000002 40.58592 40.58592 40.58592 40.58592 40.58592 40.58592 1900-01-01T00:00:00.0 11
15.98333 55.25 87.08895 60.62819 60.62819 60.62819 60.62819 60.62819 60.62819 1900-01-01T00:00:00.0 11
13.60667 55.025 322.00581 10.46453 10.46453 10.46453 10.46453 10.46453 10.46453 1900-01-01T00:00:00.0 11
13.5 54.925 318.43293 10.42499 10.42499 10.42499 10.42499 10.42499 10.42499 1900-01-01T00:00:00.0 11
12.78333 54.72333 326.47191 10.43508 10.43508 10.43508 10.43508 10.43508 10.43508 1900-01-01T00:00:00.0 11
12.21667 54.46667 294.7626 10.4254 10.4254 10.4254 10.4254 10.4254 10.4254 1900-01-01T00:00:00.0 11
11.55 54.315 307.71429 10.37599 10.37599 10.37599 10.37599 10.37599 10.37599 1900-01-01T00:00:00.0 11

[ctroupin]

Thanks Martin for reporting the issue.

If I remember well, the function loadbigfile was created to read some files in the notebooks of the project Diva-Workshops, so I will if your suggestion doesn't break anything there.

[ctroupin]

Hi Martin,
I'm testing the code modification but still get issues while trying to read the big file with an empty field.
Hence I would prefer not to modify the code source in DIVAnd.jl.

This format is quite specific to DIVAnd training, since we expect to have

• longitude as the 1st column,
• latitude as the 2nd.
• value as the 3rd
• depth as the 4th
• dates as the 10th and
• IDs as the 11th.

Maybe to be sure I get it properly: how did you get the empty field in this file?

[ctroupin]

Hi, I propose we close this issue since we cannot reproduce it and because the loadbigfile( ) function works when the file is correctly formated.

Feel free to re-open if needed.