- names
Parse scientific names.
- source
Describe a source dataset.
- sourceio
Read and write sources and source properties.
- taxamatch
Match scientific names.
- types
- LICENSES :
Object
Common data licenses.
Each property is a license identifier. The value is an object with the following properties:
- spdx: Whether the license is in the Software Package Data Exchange (SPDX) license list.
- name: License name.
- url: License URL.
- downloadFile(url, dir) ⇒
object
|null
Download file and compute MD5 hash of the stream.
- loadPage(url, page) ⇒
Promise.<puppeteer.HTTPResponse>
Load URL in browser page.
- readPageHtml(page) ⇒
Promise.<string>
Read browser page as HTML.
- readPageMhtml(page) ⇒
Promise.<string>
Read browser page as MHTML.
- readPagePng(page) ⇒
Promise.<Buffer>
Read browser page as PNG.
- readPagePdf(page) ⇒
Promise.<Buffer>
Read browser page as PDF.
- md5(x) ⇒
string
Compute MD5 hash.
- hashFile(file) ⇒
Promise.<string>
Compute MD5 hash of a file read as a stream.
Uses base64 encoding by default as it was found to be much faster for large binary files and same as UTF-8 for text.
- buildPath(url, checksum, date) ⇒
string
Build archive path.
- log(params) ⇒
Promise.<ArchiveEntry>
Add an entry to the archive log.
- guessFilename(headers, defaultBasename, url) ⇒
string
Guess filename from HTTP response headers.
- logData(params) ⇒
ArchiveEntry
Write data to file and add to log.
- search(params, options) ⇒
Array.<ArchiveEntry>
Search log for matching entries.
- getBrowser() ⇒
Promise.<puppeteer.Browser>
Get cached browser instance.
- downloadFile(params) ⇒
Promise.<ArchiveEntry>
Download file from URL and log result.
- downloadArcgisFeatureLayer(params) ⇒
Promise.<ArchiveEntry>
Download features from ArcGIS Feature Layer and log result.
- registerFile(params) ⇒
Promise.<ArchiveEntry>
Register existing file in archive log.
- buildWfsDownloadUrl(url) ⇒
object
Build WFS GetFeature URL.
- downloadPage(params) ⇒
Promise.<ArchiveEntry>
Download web page as MHTML and log result.
Page is rendered in a headless browser (puppeteer) and saved as MHTML.
- geocode(address) ⇒
Promise.<object>
Geocode address.
- geocodeCached(address) ⇒
Promise.<object>
Geocode address with caching.
- buildGetCapabilitiesUrl(url) ⇒
string
Build WFS GetCapabilities URL.
- parseCapabilities(xml) ⇒
object
Parse WFS GetCapabilities response.
- chooseOutputFormat(formats) ⇒
string
|null
Choose the output format.
- buildGetFeatureUrl(url, capabilities, paging) ⇒
string
Build WFS GetFeature URL.
Parse scientific names.
- names
- ~ScientificName
- new ScientificName(obj)
- instance
- .toString(options) ⇒
string
- .warnings() ⇒
Array.<string>
- .errors() ⇒
Array.<string>
- .report() ⇒
object
- .toString(options) ⇒
- static
- .fromString(str) ⇒
ScientificName
- .fromFields(fields) ⇒
ScientificName
- .compareStrings(options) ⇒
function
- .fromString(str) ⇒
- ~GENERIC
- ~SPECIFIC
- ~SUBG
- ~SP
- ~RANKS
- ~RANK
- ~HEAD
- ~UNINOMIAL
- ~GENUS
- ~HYBRID_GENUS
- ~SUBGENUS
- ~SPECIES
- ~INFRASPECIES
- ~RANK_EPITHET
- ~CULTIVAR
- ~FIRST
- ~HYBRID
- ~cleanName(s) ⇒
string
- ~parseInfraspecies(s) ⇒
Array.<Infraspecies>
- ~printInfraspecies(infraspecies, options) ⇒
string
- ~printScientificName(name, [options]) ⇒
string
- ~formatScientificName(name, defaultGenus) ⇒
ParsedScientificName
- ~parseScientificName(name) ⇒
ParsedScientificName
- ~ScientificName
Class representing a scientific name.
Kind: inner class of names
Properties
Name | Type | Description |
---|---|---|
parsed | ParsedScientificName |
Parsed scientific name. |
[input] | string | object |
Input from which parsed was derived. |
[matches] | Array.<object> |
Matches from a taxonomic database. |
- ~ScientificName
- new ScientificName(obj)
- instance
- .toString(options) ⇒
string
- .warnings() ⇒
Array.<string>
- .errors() ⇒
Array.<string>
- .report() ⇒
object
- .toString(options) ⇒
- static
- .fromString(str) ⇒
ScientificName
- .fromFields(fields) ⇒
ScientificName
- .compareStrings(options) ⇒
function
- .fromString(str) ⇒
Param | Type | Description |
---|---|---|
obj | ParsedScientificName |
Parsed scientific name. |
Print scientific name to string.
Kind: instance method of ScientificName
Param | Type | Description |
---|---|---|
options | object |
Print options (see printScientificName). |
Example
ScientificName.fromString(`Malus pumila var. asiatica 'Gala'`).toString()
// "Malus pumila var. asiatica 'Gala'"
Get warnings.
Kind: instance method of ScientificName
Example
ScientificName.fromString('... Malus x pumila ...').warnings()
// [ 'Unparsed head', 'Unparsed tail', 'Hybrid' ]
ScientificName.fromFields({genus: 'Malus', species: 'pumila', scientific: 'Pyrus communis'}).warnings()
// [ 'Inconsistent secondary fields: genus, species' ]
Get errors.
Kind: instance method of ScientificName
Example
(new ScientificName({species: 'pumila'})).errors()
// [ 'Missing genus' ]
Get full report.
Kind: instance method of ScientificName
Example
ScientificName.fromString('... Malus x pumila ...').report()
// {
// input: '... Malus x pumila ...',
// parsed: {
// head: '... ',
// genus: 'Malus',
// species: 'pumila',
// hybrid: true,
// tail: '...'
// },
// warnings: [ 'Unparsed head', 'Unparsed tail', 'Hybrid' ]
// }
Build scientific name from string.
Kind: static method of ScientificName
Param | Type |
---|---|
str | string |
Example
ScientificName.fromString('Malus pumila')
// ScientificName {
// parsed: { genus: 'Malus', species: 'pumila' },
// input: 'Malus pumila'
// }
Build scientific name from feature fields.
Kind: static method of ScientificName
Param | Type |
---|---|
fields | object |
Example
ScientificName.fromFields({ scientific: 'Malus pumila', other: 'Bloop' })
// ScientificName {
// parsed: { genus: 'Malus', species: 'pumila' },
// input: { scientific: 'Malus pumila' }
// }
ScientificName.fromFields({ genus: 'malus', species: 'PLATANOÏDES' })
// ScientificName {
// parsed: { genus: 'Malus', species: 'platanoides' },
// input: { genus: 'malus', species: 'PLATANOÏDES'}
// }
Generate compare function for sorting by string representation.
Kind: static method of ScientificName
Returns: function
- Compare function (a, b).
Param | Type | Description |
---|---|---|
options | object |
Print options (see printScientificName). |
Example
l = [new ScientificName({genus: 'Prunus'}), new ScientificName({genus: 'Malus'})]
l.sort(ScientificName.compareStrings())
// [
// ScientificName { parsed: { genus: 'Malus' } },
// ScientificName { parsed: { genus: 'Prunus' } }
// ]
Generic epithet.
Minimum two letters. Dash can be within three letters of end (e.g. 'Uva-ursi', 'Filix-mas').
Kind: inner constant of names
Specifc epithet.
Minimum two letters. Dashes can be within one letter of end (e.g. 's-stylata', 'laurel-y').
Kind: inner constant of names
Subgenus rank.
subg: subg(.) | subgen(.) | subgenus
Kind: inner constant of names
Species rank.
sp: sp(.), spp(.), species
Kind: inner constant of names
Infraspecific ranks.
subsp: subsp(.) | subspp(.) | ssp(.) | sspp(.) | subspecies var: var(.) | variety | varietas subvar: subvar(.), subvariety, subvarietas f: f(.) | form | forma subf: subf(.) | subform | subforma
Kind: inner constant of names
Any infraspecific rank.
Kind: inner constant of names
Everything before the first (latin) letter or hybrid symbol.
Kind: inner constant of names
Uninomial.
Kind: inner constant of names
Genus.
Identical to uninomial, but inferred to be a genus based on context.
Kind: inner constant of names
Secondary genus in hybrid formula.
May be abbreviated down to a single letter.
Kind: inner constant of names
Subgenus.
Kind: inner constant of names
Species.
Kind: inner constant of names
One or more infraspecific epithets, each preceded by an optional rank.
Kind: inner constant of names
Single infraspecific epithet preceded by an optional rank.
Kind: inner constant of names
Cultivar.
Must be wrapped in quotes and not include certain characters.
Kind: inner constant of names
Parse a scientific name (or the first name in a hybrid formula).
Each key is a regular expression with named capture groups. Try each in
order. As soon as a match is found, proceed to the children keys and repeat
until null
or no more children are found. Any tags
, if encountered, are
added to the result.
Kind: inner constant of names
Parse a secondary name in a hybrid formula.
Kind: inner constant of names
Clean name string.
- Latinizes characters.
- Replaces whitespace sequences with a single space.
- Removes leading and trailing whitespace.
Kind: inner method of names
Param | Type |
---|---|
s | string |
Example
cleanName(' Acer platanoïdes ') // 'Acer platanoides'
Parse infraspecific ranks and epithets.
Kind: inner method of names
Param | Type |
---|---|
s | string |
Example
parseInfraspecies('foo f bar') // [{epithet: 'foo'}, {rank: 'f.', epithet: 'bar'}]
Print infraspecific ranks and epithets.
Kind: inner method of names
Param | Type | Default | Description |
---|---|---|---|
infraspecies | Array.<Infraspecies> |
||
options | object |
||
[options.n] | object |
Infinity |
– Number of infraspecies. |
[options.rank] | object |
true |
– Print infraspecies rank. |
Example
printInfraspecies([ { rank: 'f.', epithet: 'mora' } ])
// 'f. mora'
printInfraspecies([ { rank: 'f.', epithet: 'mora' } ], { rank: false })
// 'mora'
Print scientific name.
Kind: inner method of names
Param | Type | Default | Description |
---|---|---|---|
name | ParsedScientificName |
Scientific name. | |
[options] | object |
Printing options. | |
[options.infraspecies] | number |
Infinity |
Number of infraspecies. |
[options.hybrid] | boolean |
true |
Print hybrid symbol and formulas. |
[options.rank] | boolean |
true |
Print infraspecies rank. |
[options.cultivar] | boolean |
true |
Print cultivar. |
Example
name = {
genus: 'Genus',
species: 'speciosa',
infraspecies: [{ rank: 'f.', epithet: 'formosa' }],
cultivar: 'Gala',
hybrid: true,
hybrids: [{ genus: 'Genus', species: 'pendula' }]
}
printScientificName(name)
// "Genus speciosa f. formosa 'Gala' × Genus pendula'"
printScientificName(name, {cultivar: false})
// "Genus speciosa f. formosa × Genus pendula'"
printScientificName(name, {infraspecies: 0, cultivar: false})
// 'Genus speciosa × Genus pendula'
printScientificName(name, {hybrid: false, infraspecies: 0, cultivar: false})
// 'Genus speciosa'
Format scientific name.
Kind: inner method of names
Param | Type | Default | Description |
---|---|---|---|
name | ParsedScientificName |
– Scientific name. | |
defaultGenus | string | boolean |
null |
– Genus to assume if hybrid genus is blank or an abbreviation of defaultGenus . Defaults to genus if null , or skipped if false . |
Example
name = {
genus: ' GENUS',
species: 'SPECIOSA ',
infraspecies: [ { rank: 'VAR', epithet: 'FORMOSA' } ],
cultivar: 'CULTI VAR',
hybrids: [ {genus: 'G', species: 'spéciosa' } ],
hybrid: true
}
formatScientificName(name)
// {
// genus: 'Genus',
// species: 'speciosa',
// infraspecies: [ { rank: 'var.', epithet: 'formosa' } ],
// cultivar: 'Culti Var',
// hybrids: [ { genus: 'Genus', species: 'speciosa' } ],
// hybrid: true
// }
Parse scientific name.
Kind: inner method of names
Param | Type | Description |
---|---|---|
name | string |
Name to parse as a scientific name. |
Example
parseScientificName(`Genus`)
// { uninomial: 'Genus' }
parseScientificName(`Genus speciosa var. segunda 'Cultivar' x Genus hybrida`)
// {
// genus: 'Genus',
// species: 'speciosa',
// infraspecies: [ { rank: 'var.', epithet: 'segunda' } ],
// cultivar: 'Cultivar',
// hybrids: [ { genus: 'Genus', species: 'hybrida' } ],
// hybrid: true
// }
Describe a source dataset.
- source
- ~Source
- new Source(props, [options])
- .findFiles([type], [options]) ⇒
Array.<ArchiveEntry>
- .fetchFiles([type], [options]) ⇒
Promise.<Array.<ArchiveEntry>>
- .process(file, [options]) ⇒
boolean
- .getFields() ⇒
object
- .getRows([n]) ⇒
Array.<object>
- .sample([options]) ⇒
object.<string, Array>
- .glimpse([options])
- .open() ⇒
Promise.<gdal.Layer>
- .close()
- .getSrs() ⇒
gdal.SpatialReference
- .getFeatureGeometry(feature, [options]) ⇒
gdal.Geometry
- .success(msg, ...objects)
- .log(msg, ...objects)
- .warn(msg, ...objects)
- .error(msg, ...objects)
- ~Source
Class representing a source dataset.
Kind: inner class of source
- ~Source
- new Source(props, [options])
- .findFiles([type], [options]) ⇒
Array.<ArchiveEntry>
- .fetchFiles([type], [options]) ⇒
Promise.<Array.<ArchiveEntry>>
- .process(file, [options]) ⇒
boolean
- .getFields() ⇒
object
- .getRows([n]) ⇒
Array.<object>
- .sample([options]) ⇒
object.<string, Array>
- .glimpse([options])
- .open() ⇒
Promise.<gdal.Layer>
- .close()
- .getSrs() ⇒
gdal.SpatialReference
- .getFeatureGeometry(feature, [options]) ⇒
gdal.Geometry
- .success(msg, ...objects)
- .log(msg, ...objects)
- .warn(msg, ...objects)
- .error(msg, ...objects)
Param | Type | Default | Description |
---|---|---|---|
props | SourceProperties |
Source properties. | |
[options] | object |
||
[options.exit] | boolean |
true |
Whether to throw errors or print them to the console. |
[options.srs] | string |
"EPSG:4326" |
Spatial reference system to assume if none is defined in props.srs and none can be read from the input files. |
Find files in archive (by type).
Retuns the most recent match for each file if multiple are found.
Kind: instance method of Source
Returns: Array.<ArchiveEntry>
- Archive entries.
Param | Type | Default | Description |
---|---|---|---|
[type] | FileType |
'data' |
– File type. |
[options] | object |
||
[options.format] | BrowserFormat |
Browser wepbage export format. | |
[options.maxDays] | number |
Maximum age of archive result in days. |
Find files in archive (and download if missing).
Kind: instance method of Source
Returns: Promise.<Array.<ArchiveEntry>>
- Archive entries.
Throws:
Error
If (checksum or manual-download) file not found in archive.Error
If download fails.
Param | Type | Default | Description |
---|---|---|---|
[type] | FileType |
'data' |
– File type. |
[options] | object |
||
[options.format] | BrowserFormat |
'mhtml' |
Browser webpage export format. |
[options.maxDays] | number |
Maximum age of search result in days. |
Process input and write to output.
Reading, writing, and coordinate transformations are performed by GDAL via the node-gdal-async bindings.
Processing steps include a schema crosswalk (this.props.crosswalk
),
skipping features by field values (this.props.deleteFunc
), reducing complex
geometries to centroid points (options.centroids
), and skipping features
outside a bounding box (options.bounds
).
Quirks
- GDAL date/time fields: Read as objects but written as ISO8601 strings
- GDAL list fields: Read as arrays and written as pipe-separated strings
Kind: instance method of Source
Returns: boolean
- Whether processed file (true) or skipped (false).
Param | Type | Default | Description |
---|---|---|---|
file | string |
Output file path. | |
[options] | object |
Output options. | |
[options.driver] | string |
Name of GDAL driver to use to write to the output file (see https://gdal.org/drivers/vector). Guessed from file extension if not provided. | |
[options.creation] | Array.<string> | object |
Driver-specific dataset creation options (see https://gdal.org/drivers/vector). | |
[options.overwrite] | boolean |
false |
Whether to proceed if file already exists. |
[options.srs] | string |
"EPSG:4326" |
Output spatial reference system in any format supported by OGRSpatialReference.SetFromUserInput(). Use 'EPSG:' for (latitude, longitude) and '+init=EPSG:' (PROJ<6 behavior) for (longitude, latitude). |
[options.centroids] | boolean |
false |
Whether to reduce non-point geometries to centroids. |
[options.keepInvalid] | boolean |
false |
Whether to keep features with empty or invalid geometries. |
[options.keepFields] | boolean |
false |
Whether to keep the input feature fields alongside the result of the schema crosswalk (this.props.crosswalk ). |
[options.prefix=] | string |
String to append to input field names to prevent collisions with output field names. Applies only if options.keepFields is true . |
|
[options.bounds] | Array.<number> |
Bounding box in output SRS (options.srs ) in the format [xmin, ymin, xmax, ymax]. If provided, features outside the bounds are skipped. |
Get layer field names and GDAL data types.
Kind: instance method of Source
Returns: object
- Field names (keys) and GDAL data types (values)
Get feature fields.
Kind: instance method of Source
Param | Type | Default | Description |
---|---|---|---|
[n] | integer |
Infinity |
Maximum number of features to read. |
Sample field values from input.
Kind: instance method of Source
Returns: object.<string, Array>
- Object of field values with field names as keys.
Param | Type | Default | Description |
---|---|---|---|
[options] | object |
||
[options.n] | number |
1000 |
Maximum number of features to sample. |
[options.max] | number |
100 |
Maximum number of values to collect for each field. |
[options.sort] | boolean |
true |
Whether to sort values. |
[options.unique] | boolean |
true |
Whether to only save unique values. |
Print table of input field names, types, and unique values.
Kind: instance method of Source
Param | Type | Default | Description |
---|---|---|---|
[options] | object |
Options to pass to Source#sample, plus: | |
[options.sample] | object.<string, Array> |
Result of Source#sample. | |
[options.truncate] | number |
1280 |
Maximum number of characters to print per field. |
[options.widths] | Array.<number> |
[20, 10, 130] |
Column widths for field names, types, and unique values, respectively. |
[options.sep] | string |
"·" |
Separator between unique values. |
Open data file(s) with GDAL.
If the dataset has a single layer, the layer is returned. Otherwise, and
props.layer
or props.openFunc
is not provided, an error is thrown.
Kind: instance method of Source
Returns: Promise.<gdal.Layer>
- See the documentation for
node-gdal-async.
Result is cached until closed with Source#close.
Close input file if open with GDAL.
Kind: instance method of Source
Get spatial reference system (SRS) of input.
Kind: instance method of Source
Returns: gdal.SpatialReference
- The provided SRS (this.props.srs
),
the SRS read from the input file, or the default SRS (this.options.srs
).
Get feature geometry.
Kind: instance method of Source
Returns: gdal.Geometry
- The feature geometry.
Param | Type | Description |
---|---|---|
feature | gdal.Feature |
Input feature. |
[options] | object |
Additional options. |
[options.fields] | object |
Field values of the feature. |
[options.srs] | object |
Spatial reference system. |
[options.isXY] | boolean |
Whether the input coordinates are in (longitude/east, latitude/north) order. |
Print success message to console (green).
Kind: instance method of Source
Param | Type | Description |
---|---|---|
msg | string |
Message prepended with green tag ([props.id] ). |
...objects | * |
Additional objects passed to console.log() . |
Print message to console (cyan).
Kind: instance method of Source
Param | Type | Description |
---|---|---|
msg | string |
Message prepended with cyan tag ([props.id] ). |
...objects | * |
Additional objects passed to console.log() . |
Print warning to console (yellow).
Kind: instance method of Source
Param | Type | Description |
---|---|---|
msg | string |
Message prepended with yellow tag ([props.id] ). |
...objects | * |
Additional objects passed to console.log() . |
Throw or print error to console (red).
Kind: instance method of Source
Param | Type | Description |
---|---|---|
msg | string |
Message prepended with red tag ([props.id] ). |
...objects | * |
Additional objects passed directly to console.error() or appended to error via util.inspect() . |
Read and write sources and source properties.
- sourceio
- ~readSourceProperties(file) ⇒
Array.<SourceProperties>
- ~writeSourceProperties(sourceProps, file, currentFile)
- ~loadSources(file, [filters]) ⇒
Array.<Source>
- ~readSourceProperties(file) ⇒
Read source properties from a file.
Kind: inner method of sourceio
Returns: Array.<SourceProperties>
- Source properties.
Param | Type | Description |
---|---|---|
file | string |
Path to source properties file. |
Write source properties to a file.
Kind: inner method of sourceio
Param | Type | Description |
---|---|---|
sourceProps | Array.<SourceProperties> |
Source properties. |
file | string |
Path to new source properties file. |
currentFile | string |
Path to current source properties file ( defaults to file ). Used to replicate the header (everything before module.exports ). |
Load sources from source properties.
Crosswalks are modified for unit conversions and range parsing.
Kind: inner method of sourceio
Param | Type | Default | Description |
---|---|---|---|
file | string |
Path to file containing source properties. | |
[filters] | object |
{} |
|
filters.id | Array.<string> |
Filter by id. | |
filters.country | Array.<string> |
Filter by country. | |
filters.state | Array.<string> |
Filter by state. | |
filters.city | Array.<string> |
Filter by city. | |
filters.designation | Array.<string> |
Filter by designation. | |
filters.scope | Array.<string> |
Filter by scope. | |
filters.omit | boolean |
Whether to include sources flagged as omit: true . |
Match scientific names.
- taxamatch
- ~Matcher
- new Matcher(taxa, [id])
- .match(name) ⇒
Array.<object>
- ~Matcher
Class for matching scientific names to a taxonomic dictionary.
Currently supports exact, fuzzy, and phonetic matching on:
- genus
- species
- first infraspecies epithet and rank
Kind: inner class of taxamatch
- ~Matcher
- new Matcher(taxa, [id])
- .match(name) ⇒
Array.<object>
Param | Type | Default | Description |
---|---|---|---|
taxa | Array.<object> |
Taxonomic dictionary. Each taxon must have a unique id and genus , and may have species and infraspecies [{ rank, epithet }, ...]. |
|
[id] | string |
"'id'" |
Key in taxa to use as unique object identifier. |
Example
taxa = [
{ id: 0, genus: 'Malus' },
{ id: 1, genus: 'Malus', species: 'pumila' },
{ id: 2, genus: 'Malus', species: 'pumila', infraspecies: [{ rank: 'var.', epithet: 'asiatica' }] }
]
matcher = new Matcher(taxa)
matcher.match({ genus: 'Malus' })
matcher.match({ genus: 'Malis' })
matcher.match({ genus: 'Malus', species: 'pumila' })
matcher.match({ genus: 'Malus', species: 'pimila' })
matcher.match({ genus: 'Mala', species: 'pimila' })
matcher.match({ genus: 'Malus', species: 'pumila', infraspecies: [{ epithet: 'asiatica'}] })
matcher.match({ genus: 'Malus', species: 'pumila', infraspecies: [{ rank: 'f.', epithet: 'asiatica'}] })
matcher.match({ genus: 'Malus', species: 'pumila', infraspecies: [{ rank: 'var.', epithet: 'asiatica'}] })
matcher.match({ genus: 'Malis', species: 'pimila', infraspecies: [{ rank: 'var.', epithet: 'asiatica'}] })
matcher.match({ genus: 'malus', species: 'pu-mila' })
Match scientific name to taxa.
Kind: instance method of Matcher
Returns: Array.<object>
- Taxon match(es) in the following order:
- exact and complete match, or
- complete phonetic match or fuzzy match(es)
- incomplete exact, phonetic, or fuzzy match(es) Each match is in the following format:
- {boolean} incomplete - Whether match is of a higher rank than the provided name.
- {number[]} fuzzy - Similarity score (0-1) for each matched name component (in the order genus, species, infraspecies), if fuzzy.
- {boolean} phonetic - Whether match is phonetic.
- {object} taxon - Matched taxon.
Param | Type | Description |
---|---|---|
name | names.ParsedScientificName |
Scientific name. |
- types
- ~DownloadMethod :
'manual'
|'file'
|'arcgis'
|'browser'
- ~BrowserFormat :
'mhtml'
|'html'
|'png'
|'pdf'
- ~FileType :
'data'
|'metadata'
|'license'
- ~ChecksumFile :
object
- ~UrlFile :
object
- ~ArchiveFile :
ChecksumFile
|UrlFile
- ~SourceFile :
string
|Object
|Object
|Object
|Object
- ~ArchiveEntry :
object
- ~SourceProperties :
object
- ~SourcePropertiesExtended :
SourceProperties
- ~Infraspecies :
object
- ~Hybrid :
object
- ~ParsedScientificName :
object
- ~DownloadMethod :
File download method.
- manual: Download manually (e.g. clicking on a button in the browser)
- file: Download directly
- arcgis: Download with the ArcGIS Feature Layer API with paginated requests
- browser: Render in a browser and save the resulting webpage
Kind: inner typedef of types
Browser webpage export format.
- mhtml: Webpage with resources saved in a single file
- html: Webpage HTML only
- png: Screenshot as PNG image
- pdf: Webpage as PDF document
Kind: inner typedef of types
File type.
- data: Data file (Source.props.data)
- metadata: Metadata file (Source.props.metadata)
- license: License file (Source.props.license)
Kind: inner typedef of types
Checksum file descriptor.
Kind: inner typedef of types
Properties
Name | Type | Description |
---|---|---|
checksum | string |
File checksum (base-64 md5 hash). |
URL file descriptor.
Kind: inner typedef of types
Properties
Name | Type | Description |
---|---|---|
url | string |
File URL. |
method | DownloadMethod |
|
[format] | BrowserFormat |
Archive file descriptor.
Kind: inner typedef of types
Source file descriptor.
Kind: inner typedef of types
Archive entry.
Kind: inner typedef of types
Properties
Name | Type | Description |
---|---|---|
path | string |
File path. |
checksum | string |
File checksum (base-64 md5 hash). |
date | Date |
Date of acquisition. |
[maxDate] | boolean |
Whether date is a maximum-possible date. |
[dateAdded] | Date |
Date added to archive, if not date (if file was registered manually rather than downloaded). |
[url] | string |
File URL. |
[method] | DownloadMethod |
File download method. |
[format] | BrowserFormat |
Browser webpage export format (if method is browser ). |
[existed] | boolean |
Whether file already existed in archive, in which case path is the path of the existing file. |
[props] | object |
Additional properties. |
Properties used by Source for data processing.
Kind: inner typedef of types
Properties
Name | Type | Description |
---|---|---|
id | string |
Identifier prepended to console output. |
data | SourceFile | Array.<SourceFile> |
Data file(s). |
vfs | string |
GDAL virtual file system type (/vsizip/ ). |
filename | string |
Relative path to the file to open with GDAL within an archive file. |
layer | string |
Layer name to open with GDAL within a file. Only relevant for files with multiple layers. |
openFunc | function |
Function that takes a file path (or array) and returns a GDAL dataset. If provided, takes precedence over vfs and filename . |
geometry | object |
Geometry field names for formats without explicit geometries (e.g. CSV). If not provided, will attempt to guess from field names. |
geometry.wkt | string |
Name of field with well-known-text (wkt) geometry. If provided, takes precedence over x and y . |
geometry.x | string |
Name of field with x coordinate (longitude, easting). |
geometry.y | string |
Name of field with y coordinate (latitude, northing). |
srs | string |
Spatial reference system in any format supported by OGRSpatialReference.SetFromUserInput(). |
crosswalk | Object.<string, (string|function())> |
Crosswalk mapping to a target schema. For each key: value pair, key is the new field name and value is either the old field name (e.g. height: 'HEIGHT' ) or a function that takes an object (of feature field values) and returns a value (e.g. height: x => x['HEIGHT'] / 100 ). |
coordsFunc | function |
Function that takes an object (of feature field values before the crosswalk) and returns a number array of point coordinates [x, y] . This is a useful alternative to geometry if the coordinates need to be extracted from field values (e.g. obj => obj.XY.split(';').map(Number) ). |
addressFunc | function |
Function that takes an object (of feature field values before the crosswalk) and returns an address string for geocoding. |
deleteFunc | function |
Function that takes an object (of feature field values before the crosswalk) and returns a value (e.g. x => x['HEALTH'] === 'dead' ). The feature is excluded from the output if the returned value evaluates to true . |
metadata | SourceFile | Array.<SourceFile> |
Metadata webpage(s) or file(s). |
license | object |
Data license. |
license.id | string |
License identifier (see ./lib/licenses.js ). |
license.name | string |
License name. Only provide if id is not. |
license.url | string |
License URL. Only provide if id is not. |
Additional properties not used by Source but used elsewhere.
Kind: inner typedef of types
Properties
Name | Type | Description |
---|---|---|
pending | string |
Pending issues preventing processing. |
omit | string |
– Reason for omitting from processing. |
country | string |
Country name in English (e.g. Spain ). |
state | string |
Local name of first-level administrative division (see https://en.wikipedia.org/wiki/List_of_administrative_divisions_by_country) with the exception of: - Ireland: NUTS 3 Region (https://en.wikipedia.org/wiki/NUTS_statistical_regions_of_Ireland) - Japan: Region (https://en.wikipedia.org/wiki/List_of_regions_of_Japan) - Netherlands: Province (https://en.wikipedia.org/wiki/Provinces_of_the_Netherlands) - New Zealand: Region (https://en.wikipedia.org/wiki/Regions_of_New_Zealand) - United Kingdom (England): Region (https://en.wikipedia.org/wiki/Regions_of_England) - United Kingdom (other): Country |
city | string |
Local name of city or municipality. |
designation | string |
Local name of city subset, administrative unit, university, or other institution if not country , state , or city . |
scope | string |
Scope or type of the inventory (e.g. tree , tree-street , tree-street-main , tree-park , tree-notable ). |
language | string |
Language of contents as an ISO 639-1 code (e.g. en ) and an optional ISO 3166-1 alpha-2 region code (e.g. en-AU ). |
primary | string |
id of the primary source (for grouping sources together). |
long | string |
Full name of the government body, university, or other institution (e.g. City of Melbourne ). |
short | string |
Short name (e.g. Melbourne ). |
centre | object |
Centre point (for map label placement). |
centre.lon | number |
Longitude in decimal degrees (EPSG:4326). |
centre.lat | number |
Latitude in decimal degrees (EPSG:4326). |
Infraspecies epithet.
Kind: inner typedef of types
Properties
Name | Type | Description |
---|---|---|
rank | string |
Rank (subsp. , var. , f. , subvar. , subf. ). |
epithet | string |
Epithet (lowercase: e.g. pontica ). |
Hybrid name.
Represents a secondary scientific name in a hybrid formula.
Kind: inner typedef of types
Properties
Name | Type | Description |
---|---|---|
genus | string |
Genus (capitalized: e.g. Malus ). |
subgenus | string |
Subgenus (capitalized: e.g. Malus ). |
species | string |
Specific epithet (lowercase: e.g. pumila ). |
infraspecies | Array.<Infraspecies> |
Infraspecific epithets. |
cultivar | string |
Cultivar (title case: e.g. Golden Delicious ). |
Scientific name.
Kind: inner typedef of types
Properties
Name | Type | Description |
---|---|---|
head | string |
Unparsed head. |
uninomial | string |
– Uninomial name (maybe genus ). |
genus | string |
Genus (capitalized: e.g. Malus ). |
subgenus | string |
Subgenus (capitalized: e.g. Malus ). |
species | string |
Specific epithet (lowercase: e.g. pumila ). |
infraspecies | Array.<Infraspecies> |
Infraspecific epithets. |
cultivar | string |
Cultivar (title case: e.g. Golden Delicious ). |
hybrid | boolean |
Whether this is a hybrid. |
hybridGenus | boolean |
– Whether genus is a nothogenus (e.g. × Sorbopyrus ). |
hybrids | Array.<Hybrid> |
– Secondary names in a hybrid formula. |
tail | string |
Unparsed tail. |
Common data licenses.
Each property is a license identifier. The value is an object with the following properties:
- spdx: Whether the license is in the Software Package Data Exchange (SPDX) license list.
- name: License name.
- url: License URL.
Kind: global constant
Download file and compute MD5 hash of the stream.
Kind: global function
Returns: object
| null
- File path (file) and md5-base64 checksum (checksum)
Param | Type | Default | Description |
---|---|---|---|
url | string |
– URL to download | |
dir | string |
"." |
– Directory to save file to |
Load URL in browser page.
Kind: global function
Param | Type |
---|---|
url | string |
page | puppeteer.Page |
Read browser page as HTML.
Kind: global function
Returns: Promise.<string>
- HTML
Param | Type |
---|---|
page | puppeteer.Page |
Read browser page as MHTML.
Kind: global function
Returns: Promise.<string>
- MHTML
Param | Type |
---|---|
page | puppeteer.Page |
Read browser page as PNG.
Kind: global function
Returns: Promise.<Buffer>
- PNG
Param | Type |
---|---|
page | puppeteer.Page |
Read browser page as PDF.
Kind: global function
Returns: Promise.<Buffer>
- PDF
Param | Type |
---|---|
page | puppeteer.Page |
Compute MD5 hash.
Kind: global function
Returns: string
- MD5 hash
Param | Type |
---|---|
x | string |
Compute MD5 hash of a file read as a stream.
Uses base64 encoding by default as it was found to be much faster for large binary files and same as UTF-8 for text.
Kind: global function
Returns: Promise.<string>
- MD5 hash
Param | Type |
---|---|
file | string |
Build archive path.
Kind: global function
Param | Type | Description |
---|---|---|
url | string |
– Represented with MD5 hash |
checksum | string |
File hash. Used as the hash if no url |
date | Date |
– Represented with ISO 8601 string with no colons (:) |
Add an entry to the archive log.
Kind: global function
Param | Type |
---|---|
params | object |
params.date | Date |
Guess filename from HTTP response headers.
Kind: global function
Param | Type | Description |
---|---|---|
headers | object |
– HTTP response headers |
defaultBasename | string |
Basename to use if none is found |
url | string |
HTTP request URL |
Write data to file and add to log.
Kind: global function
Param | Type |
---|---|
params | object |
params.data | string |
params.filename | string |
params.url | string |
params.date | Date |
Search log for matching entries.
Kind: global function
Returns: Array.<ArchiveEntry>
- Entries that match search criteria, sorted by date
descending.
Param | Type | Description |
---|---|---|
params | object |
Search criteria as key-value pairs that must match |
options | object |
|
[options.limit] | int |
Maximum number of results to return |
[options.maxDays] | int |
Maximum age of result in days |
Get cached browser instance.
Kind: global function
Download file from URL and log result.
Kind: global function
Returns: Promise.<ArchiveEntry>
- Log entry
Param | Type | Description |
---|---|---|
params | object |
Parameters |
params.url | string |
URL to download |
params.maxDays | number |
Maximum age of existing result in days that would prevent downloading again |
[params.props] | object |
Additional properties to log |
Download features from ArcGIS Feature Layer and log result.
Kind: global function
Returns: Promise.<ArchiveEntry>
- Log entry
Param | Type | Description |
---|---|---|
params | object |
Parameters |
params.url | string |
Feature layer URL |
params.maxDays | number |
Maximum age of existing result in days that would prevent downloading again |
[params.props] | object |
Additional properties to log |
Register existing file in archive log.
Kind: global function
Returns: Promise.<ArchiveEntry>
- Log entry
Param | Type | Description |
---|---|---|
params | object |
Parameters |
params.file | string |
Path to file |
params.date | string |
Date of file |
params.url | string |
URL of original file download |
params.method | string |
Method used to download file from URL |
params.maxDays | number |
Maximum age of existing result in days that would prevent downloading again |
[params.props] | object |
Additional properties to log |
Build WFS GetFeature URL.
Kind: global function
Returns: object
- URL (url) and server capabilities (capabilities)
Param | Type | Description |
---|---|---|
url | string |
WFS server URL (ideally with typeName parameter) |
Download web page as MHTML and log result.
Page is rendered in a headless browser (puppeteer) and saved as MHTML.
Kind: global function
Returns: Promise.<ArchiveEntry>
- Log entry
Param | Type | Description |
---|---|---|
params | object |
Parameters |
params.url | string |
URL to download |
params.format | BrowserFormat |
Format to save page as |
params.maxDays | number |
Maximum age of existing result in days that would prevent downloading again |
[params.props] | object |
Additional properties to log |
Geocode address.
Kind: global function
Returns: Promise.<object>
- Geocode results.
Param | Type | Description |
---|---|---|
address | string |
Address to geocode. |
Geocode address with caching.
Kind: global function
Returns: Promise.<object>
- Geocode results.
Param | Type | Description |
---|---|---|
address | string |
Address to geocode. |
Build WFS GetCapabilities URL.
Kind: global function
Param | Type | Description |
---|---|---|
url | string |
WFS server URL |
Parse WFS GetCapabilities response.
Kind: global function
Returns: object
- Parsed capabilities (version, outputFormats, typeNames,
resultTypes, resultPaging).
Param | Type | Description |
---|---|---|
xml | string |
– XML string |
Choose the output format.
Kind: global function
Param | Type | Description |
---|---|---|
formats | Array.<string> |
List of output formats |
Build WFS GetFeature URL.
Kind: global function
Param | Type | Default | Description |
---|---|---|---|
url | string |
WFS server URL (ideally with typeName parameter) | |
capabilities | object |
Server capabilities | |
paging | boolean |
false |
Whether to set a start index and max feature count |