Skip to content

Latest commit

 

History

History
1855 lines (1293 loc) · 62.6 KB

API.md

File metadata and controls

1855 lines (1293 loc) · 62.6 KB

Modules

names

Parse scientific names.

source

Describe a source dataset.

sourceio

Read and write sources and source properties.

taxamatch

Match scientific names.

types

Constants

LICENSES : Object

Common data licenses.

Each property is a license identifier. The value is an object with the following properties:

  • spdx: Whether the license is in the Software Package Data Exchange (SPDX) license list.
  • name: License name.
  • url: License URL.

Functions

downloadFile(url, dir)object | null

Download file and compute MD5 hash of the stream.

loadPage(url, page)Promise.<puppeteer.HTTPResponse>

Load URL in browser page.

readPageHtml(page)Promise.<string>

Read browser page as HTML.

readPageMhtml(page)Promise.<string>

Read browser page as MHTML.

readPagePng(page)Promise.<Buffer>

Read browser page as PNG.

readPagePdf(page)Promise.<Buffer>

Read browser page as PDF.

md5(x)string

Compute MD5 hash.

hashFile(file)Promise.<string>

Compute MD5 hash of a file read as a stream.

Uses base64 encoding by default as it was found to be much faster for large binary files and same as UTF-8 for text.

buildPath(url, checksum, date)string

Build archive path.

log(params)Promise.<ArchiveEntry>

Add an entry to the archive log.

guessFilename(headers, defaultBasename, url)string

Guess filename from HTTP response headers.

logData(params)ArchiveEntry

Write data to file and add to log.

search(params, options)Array.<ArchiveEntry>

Search log for matching entries.

getBrowser()Promise.<puppeteer.Browser>

Get cached browser instance.

downloadFile(params)Promise.<ArchiveEntry>

Download file from URL and log result.

downloadArcgisFeatureLayer(params)Promise.<ArchiveEntry>

Download features from ArcGIS Feature Layer and log result.

registerFile(params)Promise.<ArchiveEntry>

Register existing file in archive log.

buildWfsDownloadUrl(url)object

Build WFS GetFeature URL.

downloadPage(params)Promise.<ArchiveEntry>

Download web page as MHTML and log result.

Page is rendered in a headless browser (puppeteer) and saved as MHTML.

geocode(address)Promise.<object>

Geocode address.

geocodeCached(address)Promise.<object>

Geocode address with caching.

buildGetCapabilitiesUrl(url)string

Build WFS GetCapabilities URL.

parseCapabilities(xml)object

Parse WFS GetCapabilities response.

chooseOutputFormat(formats)string | null

Choose the output format.

buildGetFeatureUrl(url, capabilities, paging)string

Build WFS GetFeature URL.

names

Parse scientific names.


names~ScientificName

Class representing a scientific name.

Kind: inner class of names
Properties

Name Type Description
parsed ParsedScientificName Parsed scientific name.
[input] string | object Input from which parsed was derived.
[matches] Array.<object> Matches from a taxonomic database.

new ScientificName(obj)

Param Type Description
obj ParsedScientificName Parsed scientific name.

scientificName.toString(options) ⇒ string

Print scientific name to string.

Kind: instance method of ScientificName

Param Type Description
options object Print options (see printScientificName).

Example

ScientificName.fromString(`Malus pumila var. asiatica 'Gala'`).toString()
// "Malus pumila var. asiatica 'Gala'"

scientificName.warnings() ⇒ Array.<string>

Get warnings.

Kind: instance method of ScientificName
Example

ScientificName.fromString('... Malus x pumila ...').warnings()
// [ 'Unparsed head', 'Unparsed tail', 'Hybrid' ]
ScientificName.fromFields({genus: 'Malus', species: 'pumila', scientific: 'Pyrus communis'}).warnings()
// [ 'Inconsistent secondary fields: genus, species' ]

scientificName.errors() ⇒ Array.<string>

Get errors.

Kind: instance method of ScientificName
Example

(new ScientificName({species: 'pumila'})).errors()
// [ 'Missing genus' ]

scientificName.report() ⇒ object

Get full report.

Kind: instance method of ScientificName
Example

ScientificName.fromString('... Malus x pumila ...').report()
// {
//    input: '... Malus x pumila ...',
//    parsed: {
//      head: '... ',
//      genus: 'Malus',
//      species: 'pumila',
//      hybrid: true,
//      tail: '...'
//    },
//    warnings: [ 'Unparsed head', 'Unparsed tail', 'Hybrid' ]
// }

ScientificName.fromString(str) ⇒ ScientificName

Build scientific name from string.

Kind: static method of ScientificName

Param Type
str string

Example

ScientificName.fromString('Malus pumila')
// ScientificName {
//   parsed: { genus: 'Malus', species: 'pumila' },
//   input: 'Malus pumila'
// }

ScientificName.fromFields(fields) ⇒ ScientificName

Build scientific name from feature fields.

Kind: static method of ScientificName

Param Type
fields object

Example

ScientificName.fromFields({ scientific: 'Malus pumila', other: 'Bloop' })
// ScientificName {
//   parsed: { genus: 'Malus', species: 'pumila' },
//   input: { scientific: 'Malus pumila' }
// }
ScientificName.fromFields({ genus: 'malus', species: 'PLATANOÏDES' })
// ScientificName {
//   parsed: { genus: 'Malus', species: 'platanoides' },
//   input: { genus: 'malus', species: 'PLATANOÏDES'}
// }

ScientificName.compareStrings(options) ⇒ function

Generate compare function for sorting by string representation.

Kind: static method of ScientificName
Returns: function - Compare function (a, b).

Param Type Description
options object Print options (see printScientificName).

Example

l = [new ScientificName({genus: 'Prunus'}), new ScientificName({genus: 'Malus'})]
l.sort(ScientificName.compareStrings())
// [
//   ScientificName { parsed: { genus: 'Malus' } },
//   ScientificName { parsed: { genus: 'Prunus' } }
// ]

names~GENERIC

Generic epithet.

Minimum two letters. Dash can be within three letters of end (e.g. 'Uva-ursi', 'Filix-mas').

Kind: inner constant of names


names~SPECIFIC

Specifc epithet.

Minimum two letters. Dashes can be within one letter of end (e.g. 's-stylata', 'laurel-y').

Kind: inner constant of names


names~SUBG

Subgenus rank.

subg: subg(.) | subgen(.) | subgenus

Kind: inner constant of names


names~SP

Species rank.

sp: sp(.), spp(.), species

Kind: inner constant of names


names~RANKS

Infraspecific ranks.

subsp: subsp(.) | subspp(.) | ssp(.) | sspp(.) | subspecies var: var(.) | variety | varietas subvar: subvar(.), subvariety, subvarietas f: f(.) | form | forma subf: subf(.) | subform | subforma

Kind: inner constant of names


names~RANK

Any infraspecific rank.

Kind: inner constant of names


names~HEAD

Everything before the first (latin) letter or hybrid symbol.

Kind: inner constant of names


names~UNINOMIAL

Uninomial.

Kind: inner constant of names


names~GENUS

Genus.

Identical to uninomial, but inferred to be a genus based on context.

Kind: inner constant of names


names~HYBRID_GENUS

Secondary genus in hybrid formula.

May be abbreviated down to a single letter.

Kind: inner constant of names


names~SUBGENUS

Subgenus.

Kind: inner constant of names


names~SPECIES

Species.

Kind: inner constant of names


names~INFRASPECIES

One or more infraspecific epithets, each preceded by an optional rank.

Kind: inner constant of names


names~RANK_EPITHET

Single infraspecific epithet preceded by an optional rank.

Kind: inner constant of names


names~CULTIVAR

Cultivar.

Must be wrapped in quotes and not include certain characters.

Kind: inner constant of names


names~FIRST

Parse a scientific name (or the first name in a hybrid formula).

Each key is a regular expression with named capture groups. Try each in order. As soon as a match is found, proceed to the children keys and repeat until null or no more children are found. Any tags, if encountered, are added to the result.

Kind: inner constant of names


names~HYBRID

Parse a secondary name in a hybrid formula.

Kind: inner constant of names


names~cleanName(s) ⇒ string

Clean name string.

  • Latinizes characters.
  • Replaces whitespace sequences with a single space.
  • Removes leading and trailing whitespace.

Kind: inner method of names

Param Type
s string

Example

cleanName(' Acer  platanoïdes ') // 'Acer platanoides'

names~parseInfraspecies(s) ⇒ Array.<Infraspecies>

Parse infraspecific ranks and epithets.

Kind: inner method of names

Param Type
s string

Example

parseInfraspecies('foo f bar') // [{epithet: 'foo'}, {rank: 'f.', epithet: 'bar'}]

names~printInfraspecies(infraspecies, options) ⇒ string

Print infraspecific ranks and epithets.

Kind: inner method of names

Param Type Default Description
infraspecies Array.<Infraspecies>
options object
[options.n] object Infinity – Number of infraspecies.
[options.rank] object true – Print infraspecies rank.

Example

printInfraspecies([ { rank: 'f.', epithet: 'mora' } ])
// 'f. mora'
printInfraspecies([ { rank: 'f.', epithet: 'mora' } ], { rank: false })
// 'mora'

names~printScientificName(name, [options]) ⇒ string

Print scientific name.

Kind: inner method of names

Param Type Default Description
name ParsedScientificName Scientific name.
[options] object Printing options.
[options.infraspecies] number Infinity Number of infraspecies.
[options.hybrid] boolean true Print hybrid symbol and formulas.
[options.rank] boolean true Print infraspecies rank.
[options.cultivar] boolean true Print cultivar.

Example

name = {
  genus: 'Genus',
  species: 'speciosa',
  infraspecies: [{ rank: 'f.', epithet: 'formosa' }],
  cultivar: 'Gala',
  hybrid: true,
  hybrids: [{ genus: 'Genus', species: 'pendula' }]
}
printScientificName(name)
// "Genus speciosa f. formosa 'Gala' × Genus pendula'"
printScientificName(name, {cultivar: false})
// "Genus speciosa f. formosa × Genus pendula'"
printScientificName(name, {infraspecies: 0, cultivar: false})
// 'Genus speciosa × Genus pendula'
printScientificName(name, {hybrid: false, infraspecies: 0, cultivar: false})
// 'Genus speciosa'

names~formatScientificName(name, defaultGenus) ⇒ ParsedScientificName

Format scientific name.

Kind: inner method of names

Param Type Default Description
name ParsedScientificName – Scientific name.
defaultGenus string | boolean null – Genus to assume if hybrid genus is blank or an abbreviation of defaultGenus. Defaults to genus if null, or skipped if false.

Example

name = {
  genus: ' GENUS',
  species: 'SPECIOSA ',
  infraspecies: [ { rank: 'VAR', epithet: 'FORMOSA' } ],
  cultivar: 'CULTI VAR',
  hybrids: [ {genus: 'G', species: 'spéciosa' } ],
  hybrid: true
}
formatScientificName(name)
// {
//   genus: 'Genus',
//   species: 'speciosa',
//   infraspecies: [ { rank: 'var.', epithet: 'formosa' } ],
//   cultivar: 'Culti Var',
//   hybrids: [ { genus: 'Genus', species: 'speciosa' } ],
//   hybrid: true
// }

names~parseScientificName(name) ⇒ ParsedScientificName

Parse scientific name.

Kind: inner method of names

Param Type Description
name string Name to parse as a scientific name.

Example

parseScientificName(`Genus`)
// { uninomial: 'Genus' }
parseScientificName(`Genus speciosa var. segunda 'Cultivar' x Genus hybrida`)
// {
//   genus: 'Genus',
//   species: 'speciosa',
//   infraspecies: [ { rank: 'var.', epithet: 'segunda' } ],
//   cultivar: 'Cultivar',
//   hybrids: [ { genus: 'Genus', species: 'hybrida' } ],
//   hybrid: true
// }

source

Describe a source dataset.


source~Source

Class representing a source dataset.

Kind: inner class of source


new Source(props, [options])

Param Type Default Description
props SourceProperties Source properties.
[options] object
[options.exit] boolean true Whether to throw errors or print them to the console.
[options.srs] string "EPSG:4326" Spatial reference system to assume if none is defined in props.srs and none can be read from the input files.

source.findFiles([type], [options]) ⇒ Array.<ArchiveEntry>

Find files in archive (by type).

Retuns the most recent match for each file if multiple are found.

Kind: instance method of Source
Returns: Array.<ArchiveEntry> - Archive entries.

Param Type Default Description
[type] FileType 'data' – File type.
[options] object
[options.format] BrowserFormat Browser wepbage export format.
[options.maxDays] number Maximum age of archive result in days.

source.fetchFiles([type], [options]) ⇒ Promise.<Array.<ArchiveEntry>>

Find files in archive (and download if missing).

Kind: instance method of Source
Returns: Promise.<Array.<ArchiveEntry>> - Archive entries.
Throws:

  • Error If (checksum or manual-download) file not found in archive.
  • Error If download fails.
Param Type Default Description
[type] FileType 'data' – File type.
[options] object
[options.format] BrowserFormat 'mhtml' Browser webpage export format.
[options.maxDays] number Maximum age of search result in days.

source.process(file, [options]) ⇒ boolean

Process input and write to output.

Reading, writing, and coordinate transformations are performed by GDAL via the node-gdal-async bindings.

Processing steps include a schema crosswalk (this.props.crosswalk), skipping features by field values (this.props.deleteFunc), reducing complex geometries to centroid points (options.centroids), and skipping features outside a bounding box (options.bounds).

Quirks

  • GDAL date/time fields: Read as objects but written as ISO8601 strings
  • GDAL list fields: Read as arrays and written as pipe-separated strings

Kind: instance method of Source
Returns: boolean - Whether processed file (true) or skipped (false).

Param Type Default Description
file string Output file path.
[options] object Output options.
[options.driver] string Name of GDAL driver to use to write to the output file (see https://gdal.org/drivers/vector). Guessed from file extension if not provided.
[options.creation] Array.<string> | object Driver-specific dataset creation options (see https://gdal.org/drivers/vector).
[options.overwrite] boolean false Whether to proceed if file already exists.
[options.srs] string "EPSG:4326" Output spatial reference system in any format supported by OGRSpatialReference.SetFromUserInput(). Use 'EPSG:' for (latitude, longitude) and '+init=EPSG:' (PROJ<6 behavior) for (longitude, latitude).
[options.centroids] boolean false Whether to reduce non-point geometries to centroids.
[options.keepInvalid] boolean false Whether to keep features with empty or invalid geometries.
[options.keepFields] boolean false Whether to keep the input feature fields alongside the result of the schema crosswalk (this.props.crosswalk).
[options.prefix=] string String to append to input field names to prevent collisions with output field names. Applies only if options.keepFields is true.
[options.bounds] Array.<number> Bounding box in output SRS (options.srs) in the format [xmin, ymin, xmax, ymax]. If provided, features outside the bounds are skipped.

source.getFields() ⇒ object

Get layer field names and GDAL data types.

Kind: instance method of Source
Returns: object - Field names (keys) and GDAL data types (values)


source.getRows([n]) ⇒ Array.<object>

Get feature fields.

Kind: instance method of Source

Param Type Default Description
[n] integer Infinity Maximum number of features to read.

source.sample([options]) ⇒ object.<string, Array>

Sample field values from input.

Kind: instance method of Source
Returns: object.<string, Array> - Object of field values with field names as keys.

Param Type Default Description
[options] object
[options.n] number 1000 Maximum number of features to sample.
[options.max] number 100 Maximum number of values to collect for each field.
[options.sort] boolean true Whether to sort values.
[options.unique] boolean true Whether to only save unique values.

source.glimpse([options])

Print table of input field names, types, and unique values.

Kind: instance method of Source

Param Type Default Description
[options] object Options to pass to Source#sample, plus:
[options.sample] object.<string, Array> Result of Source#sample.
[options.truncate] number 1280 Maximum number of characters to print per field.
[options.widths] Array.<number> [20, 10, 130] Column widths for field names, types, and unique values, respectively.
[options.sep] string "·" Separator between unique values.

source.open() ⇒ Promise.<gdal.Layer>

Open data file(s) with GDAL.

If the dataset has a single layer, the layer is returned. Otherwise, and props.layer or props.openFunc is not provided, an error is thrown.

Kind: instance method of Source
Returns: Promise.<gdal.Layer> - See the documentation for node-gdal-async. Result is cached until closed with Source#close.


source.close()

Close input file if open with GDAL.

Kind: instance method of Source


source.getSrs() ⇒ gdal.SpatialReference

Get spatial reference system (SRS) of input.

Kind: instance method of Source
Returns: gdal.SpatialReference - The provided SRS (this.props.srs), the SRS read from the input file, or the default SRS (this.options.srs).


source.getFeatureGeometry(feature, [options]) ⇒ gdal.Geometry

Get feature geometry.

Kind: instance method of Source
Returns: gdal.Geometry - The feature geometry.

Param Type Description
feature gdal.Feature Input feature.
[options] object Additional options.
[options.fields] object Field values of the feature.
[options.srs] object Spatial reference system.
[options.isXY] boolean Whether the input coordinates are in (longitude/east, latitude/north) order.

source.success(msg, ...objects)

Print success message to console (green).

Kind: instance method of Source

Param Type Description
msg string Message prepended with green tag ([props.id]).
...objects * Additional objects passed to console.log().

source.log(msg, ...objects)

Print message to console (cyan).

Kind: instance method of Source

Param Type Description
msg string Message prepended with cyan tag ([props.id]).
...objects * Additional objects passed to console.log().

source.warn(msg, ...objects)

Print warning to console (yellow).

Kind: instance method of Source

Param Type Description
msg string Message prepended with yellow tag ([props.id]).
...objects * Additional objects passed to console.log().

source.error(msg, ...objects)

Throw or print error to console (red).

Kind: instance method of Source

Param Type Description
msg string Message prepended with red tag ([props.id]).
...objects * Additional objects passed directly to console.error() or appended to error via util.inspect().

sourceio

Read and write sources and source properties.


sourceio~readSourceProperties(file) ⇒ Array.<SourceProperties>

Read source properties from a file.

Kind: inner method of sourceio
Returns: Array.<SourceProperties> - Source properties.

Param Type Description
file string Path to source properties file.

sourceio~writeSourceProperties(sourceProps, file, currentFile)

Write source properties to a file.

Kind: inner method of sourceio

Param Type Description
sourceProps Array.<SourceProperties> Source properties.
file string Path to new source properties file.
currentFile string Path to current source properties file ( defaults to file). Used to replicate the header (everything before module.exports).

sourceio~loadSources(file, [filters]) ⇒ Array.<Source>

Load sources from source properties.

Crosswalks are modified for unit conversions and range parsing.

Kind: inner method of sourceio

Param Type Default Description
file string Path to file containing source properties.
[filters] object {}
filters.id Array.<string> Filter by id.
filters.country Array.<string> Filter by country.
filters.state Array.<string> Filter by state.
filters.city Array.<string> Filter by city.
filters.designation Array.<string> Filter by designation.
filters.scope Array.<string> Filter by scope.
filters.omit boolean Whether to include sources flagged as omit: true.

taxamatch

Match scientific names.


taxamatch~Matcher

Class for matching scientific names to a taxonomic dictionary.

Currently supports exact, fuzzy, and phonetic matching on:

  • genus
  • species
  • first infraspecies epithet and rank

Kind: inner class of taxamatch


new Matcher(taxa, [id])

Param Type Default Description
taxa Array.<object> Taxonomic dictionary. Each taxon must have a unique id and genus, and may have species and infraspecies [{ rank, epithet }, ...].
[id] string "'id'" Key in taxa to use as unique object identifier.

Example

taxa = [
 { id: 0, genus: 'Malus' },
 { id: 1, genus: 'Malus', species: 'pumila' },
 { id: 2, genus: 'Malus', species: 'pumila', infraspecies: [{ rank: 'var.', epithet: 'asiatica' }] }
]
matcher = new Matcher(taxa)
matcher.match({ genus: 'Malus' })
matcher.match({ genus: 'Malis' })
matcher.match({ genus: 'Malus', species: 'pumila' })
matcher.match({ genus: 'Malus', species: 'pimila' })
matcher.match({ genus: 'Mala', species: 'pimila' })
matcher.match({ genus: 'Malus', species: 'pumila', infraspecies: [{ epithet: 'asiatica'}] })
matcher.match({ genus: 'Malus', species: 'pumila', infraspecies: [{ rank: 'f.', epithet: 'asiatica'}] })
matcher.match({ genus: 'Malus', species: 'pumila', infraspecies: [{ rank: 'var.', epithet: 'asiatica'}] })
matcher.match({ genus: 'Malis', species: 'pimila', infraspecies: [{ rank: 'var.', epithet: 'asiatica'}] })
matcher.match({ genus: 'malus', species: 'pu-mila' })

matcher.match(name) ⇒ Array.<object>

Match scientific name to taxa.

Kind: instance method of Matcher
Returns: Array.<object> - Taxon match(es) in the following order:

  • exact and complete match, or
  • complete phonetic match or fuzzy match(es)
  • incomplete exact, phonetic, or fuzzy match(es) Each match is in the following format:
  • {boolean} incomplete - Whether match is of a higher rank than the provided name.
  • {number[]} fuzzy - Similarity score (0-1) for each matched name component (in the order genus, species, infraspecies), if fuzzy.
  • {boolean} phonetic - Whether match is phonetic.
  • {object} taxon - Matched taxon.
Param Type Description
name names.ParsedScientificName Scientific name.

types


types~DownloadMethod : 'manual' | 'file' | 'arcgis' | 'browser'

File download method.

  • manual: Download manually (e.g. clicking on a button in the browser)
  • file: Download directly
  • arcgis: Download with the ArcGIS Feature Layer API with paginated requests
  • browser: Render in a browser and save the resulting webpage

Kind: inner typedef of types


types~BrowserFormat : 'mhtml' | 'html' | 'png' | 'pdf'

Browser webpage export format.

  • mhtml: Webpage with resources saved in a single file
  • html: Webpage HTML only
  • png: Screenshot as PNG image
  • pdf: Webpage as PDF document

Kind: inner typedef of types


types~FileType : 'data' | 'metadata' | 'license'

File type.

  • data: Data file (Source.props.data)
  • metadata: Metadata file (Source.props.metadata)
  • license: License file (Source.props.license)

Kind: inner typedef of types


types~ChecksumFile : object

Checksum file descriptor.

Kind: inner typedef of types
Properties

Name Type Description
checksum string File checksum (base-64 md5 hash).

types~UrlFile : object

URL file descriptor.

Kind: inner typedef of types
Properties

Name Type Description
url string File URL.
method DownloadMethod
[format] BrowserFormat

types~ArchiveFile : ChecksumFile | UrlFile

Archive file descriptor.

Kind: inner typedef of types


types~SourceFile : string | Object | Object | Object | Object

Source file descriptor.

Kind: inner typedef of types


types~ArchiveEntry : object

Archive entry.

Kind: inner typedef of types
Properties

Name Type Description
path string File path.
checksum string File checksum (base-64 md5 hash).
date Date Date of acquisition.
[maxDate] boolean Whether date is a maximum-possible date.
[dateAdded] Date Date added to archive, if not date (if file was registered manually rather than downloaded).
[url] string File URL.
[method] DownloadMethod File download method.
[format] BrowserFormat Browser webpage export format (if method is browser).
[existed] boolean Whether file already existed in archive, in which case path is the path of the existing file.
[props] object Additional properties.

types~SourceProperties : object

Properties used by Source for data processing.

Kind: inner typedef of types
Properties

Name Type Description
id string Identifier prepended to console output.
data SourceFile | Array.<SourceFile> Data file(s).
vfs string GDAL virtual file system type (/vsizip/).
filename string Relative path to the file to open with GDAL within an archive file.
layer string Layer name to open with GDAL within a file. Only relevant for files with multiple layers.
openFunc function Function that takes a file path (or array) and returns a GDAL dataset. If provided, takes precedence over vfs and filename.
geometry object Geometry field names for formats without explicit geometries (e.g. CSV). If not provided, will attempt to guess from field names.
geometry.wkt string Name of field with well-known-text (wkt) geometry. If provided, takes precedence over x and y.
geometry.x string Name of field with x coordinate (longitude, easting).
geometry.y string Name of field with y coordinate (latitude, northing).
srs string Spatial reference system in any format supported by OGRSpatialReference.SetFromUserInput().
crosswalk Object.<string, (string|function())> Crosswalk mapping to a target schema. For each key: value pair, key is the new field name and value is either the old field name (e.g. height: 'HEIGHT') or a function that takes an object (of feature field values) and returns a value (e.g. height: x => x['HEIGHT'] / 100).
coordsFunc function Function that takes an object (of feature field values before the crosswalk) and returns a number array of point coordinates [x, y]. This is a useful alternative to geometry if the coordinates need to be extracted from field values (e.g. obj => obj.XY.split(';').map(Number)).
addressFunc function Function that takes an object (of feature field values before the crosswalk) and returns an address string for geocoding.
deleteFunc function Function that takes an object (of feature field values before the crosswalk) and returns a value (e.g. x => x['HEALTH'] === 'dead'). The feature is excluded from the output if the returned value evaluates to true.
metadata SourceFile | Array.<SourceFile> Metadata webpage(s) or file(s).
license object Data license.
license.id string License identifier (see ./lib/licenses.js).
license.name string License name. Only provide if id is not.
license.url string License URL. Only provide if id is not.

types~SourcePropertiesExtended : SourceProperties

Additional properties not used by Source but used elsewhere.

Kind: inner typedef of types
Properties

Name Type Description
pending string Pending issues preventing processing.
omit string – Reason for omitting from processing.
country string Country name in English (e.g. Spain).
state string Local name of first-level administrative division (see https://en.wikipedia.org/wiki/List_of_administrative_divisions_by_country) with the exception of: - Ireland: NUTS 3 Region (https://en.wikipedia.org/wiki/NUTS_statistical_regions_of_Ireland) - Japan: Region (https://en.wikipedia.org/wiki/List_of_regions_of_Japan) - Netherlands: Province (https://en.wikipedia.org/wiki/Provinces_of_the_Netherlands) - New Zealand: Region (https://en.wikipedia.org/wiki/Regions_of_New_Zealand) - United Kingdom (England): Region (https://en.wikipedia.org/wiki/Regions_of_England) - United Kingdom (other): Country
city string Local name of city or municipality.
designation string Local name of city subset, administrative unit, university, or other institution if not country, state, or city.
scope string Scope or type of the inventory (e.g. tree, tree-street, tree-street-main, tree-park, tree-notable).
language string Language of contents as an ISO 639-1 code (e.g. en) and an optional ISO 3166-1 alpha-2 region code (e.g. en-AU).
primary string id of the primary source (for grouping sources together).
long string Full name of the government body, university, or other institution (e.g. City of Melbourne).
short string Short name (e.g. Melbourne).
centre object Centre point (for map label placement).
centre.lon number Longitude in decimal degrees (EPSG:4326).
centre.lat number Latitude in decimal degrees (EPSG:4326).

types~Infraspecies : object

Infraspecies epithet.

Kind: inner typedef of types
Properties

Name Type Description
rank string Rank (subsp., var., f., subvar., subf.).
epithet string Epithet (lowercase: e.g. pontica).

types~Hybrid : object

Hybrid name.

Represents a secondary scientific name in a hybrid formula.

Kind: inner typedef of types
Properties

Name Type Description
genus string Genus (capitalized: e.g. Malus).
subgenus string Subgenus (capitalized: e.g. Malus).
species string Specific epithet (lowercase: e.g. pumila).
infraspecies Array.<Infraspecies> Infraspecific epithets.
cultivar string Cultivar (title case: e.g. Golden Delicious).

types~ParsedScientificName : object

Scientific name.

Kind: inner typedef of types
Properties

Name Type Description
head string Unparsed head.
uninomial string – Uninomial name (maybe genus).
genus string Genus (capitalized: e.g. Malus).
subgenus string Subgenus (capitalized: e.g. Malus).
species string Specific epithet (lowercase: e.g. pumila).
infraspecies Array.<Infraspecies> Infraspecific epithets.
cultivar string Cultivar (title case: e.g. Golden Delicious).
hybrid boolean Whether this is a hybrid.
hybridGenus boolean – Whether genus is a nothogenus (e.g. × Sorbopyrus).
hybrids Array.<Hybrid> – Secondary names in a hybrid formula.
tail string Unparsed tail.

LICENSES : Object

Common data licenses.

Each property is a license identifier. The value is an object with the following properties:

  • spdx: Whether the license is in the Software Package Data Exchange (SPDX) license list.
  • name: License name.
  • url: License URL.

Kind: global constant


downloadFile(url, dir) ⇒ object | null

Download file and compute MD5 hash of the stream.

Kind: global function
Returns: object | null - File path (file) and md5-base64 checksum (checksum)

Param Type Default Description
url string – URL to download
dir string "." – Directory to save file to

loadPage(url, page) ⇒ Promise.<puppeteer.HTTPResponse>

Load URL in browser page.

Kind: global function

Param Type
url string
page puppeteer.Page

readPageHtml(page) ⇒ Promise.<string>

Read browser page as HTML.

Kind: global function
Returns: Promise.<string> - HTML

Param Type
page puppeteer.Page

readPageMhtml(page) ⇒ Promise.<string>

Read browser page as MHTML.

Kind: global function
Returns: Promise.<string> - MHTML

Param Type
page puppeteer.Page

readPagePng(page) ⇒ Promise.<Buffer>

Read browser page as PNG.

Kind: global function
Returns: Promise.<Buffer> - PNG

Param Type
page puppeteer.Page

readPagePdf(page) ⇒ Promise.<Buffer>

Read browser page as PDF.

Kind: global function
Returns: Promise.<Buffer> - PDF

Param Type
page puppeteer.Page

md5(x) ⇒ string

Compute MD5 hash.

Kind: global function
Returns: string - MD5 hash

Param Type
x string

hashFile(file) ⇒ Promise.<string>

Compute MD5 hash of a file read as a stream.

Uses base64 encoding by default as it was found to be much faster for large binary files and same as UTF-8 for text.

Kind: global function
Returns: Promise.<string> - MD5 hash

Param Type
file string

buildPath(url, checksum, date) ⇒ string

Build archive path.

Kind: global function

Param Type Description
url string – Represented with MD5 hash
checksum string File hash. Used as the hash if no url
date Date – Represented with ISO 8601 string with no colons (:)

log(params) ⇒ Promise.<ArchiveEntry>

Add an entry to the archive log.

Kind: global function

Param Type
params object
params.date Date

guessFilename(headers, defaultBasename, url) ⇒ string

Guess filename from HTTP response headers.

Kind: global function

Param Type Description
headers object – HTTP response headers
defaultBasename string Basename to use if none is found
url string HTTP request URL

logData(params) ⇒ ArchiveEntry

Write data to file and add to log.

Kind: global function

Param Type
params object
params.data string
params.filename string
params.url string
params.date Date

search(params, options) ⇒ Array.<ArchiveEntry>

Search log for matching entries.

Kind: global function
Returns: Array.<ArchiveEntry> - Entries that match search criteria, sorted by date descending.

Param Type Description
params object Search criteria as key-value pairs that must match
options object
[options.limit] int Maximum number of results to return
[options.maxDays] int Maximum age of result in days

getBrowser() ⇒ Promise.<puppeteer.Browser>

Get cached browser instance.

Kind: global function


downloadFile(params) ⇒ Promise.<ArchiveEntry>

Download file from URL and log result.

Kind: global function
Returns: Promise.<ArchiveEntry> - Log entry

Param Type Description
params object Parameters
params.url string URL to download
params.maxDays number Maximum age of existing result in days that would prevent downloading again
[params.props] object Additional properties to log

downloadArcgisFeatureLayer(params) ⇒ Promise.<ArchiveEntry>

Download features from ArcGIS Feature Layer and log result.

Kind: global function
Returns: Promise.<ArchiveEntry> - Log entry

Param Type Description
params object Parameters
params.url string Feature layer URL
params.maxDays number Maximum age of existing result in days that would prevent downloading again
[params.props] object Additional properties to log

registerFile(params) ⇒ Promise.<ArchiveEntry>

Register existing file in archive log.

Kind: global function
Returns: Promise.<ArchiveEntry> - Log entry

Param Type Description
params object Parameters
params.file string Path to file
params.date string Date of file
params.url string URL of original file download
params.method string Method used to download file from URL
params.maxDays number Maximum age of existing result in days that would prevent downloading again
[params.props] object Additional properties to log

buildWfsDownloadUrl(url) ⇒ object

Build WFS GetFeature URL.

Kind: global function
Returns: object - URL (url) and server capabilities (capabilities)

Param Type Description
url string WFS server URL (ideally with typeName parameter)

downloadPage(params) ⇒ Promise.<ArchiveEntry>

Download web page as MHTML and log result.

Page is rendered in a headless browser (puppeteer) and saved as MHTML.

Kind: global function
Returns: Promise.<ArchiveEntry> - Log entry

Param Type Description
params object Parameters
params.url string URL to download
params.format BrowserFormat Format to save page as
params.maxDays number Maximum age of existing result in days that would prevent downloading again
[params.props] object Additional properties to log

geocode(address) ⇒ Promise.<object>

Geocode address.

Kind: global function
Returns: Promise.<object> - Geocode results.

Param Type Description
address string Address to geocode.

geocodeCached(address) ⇒ Promise.<object>

Geocode address with caching.

Kind: global function
Returns: Promise.<object> - Geocode results.

Param Type Description
address string Address to geocode.

buildGetCapabilitiesUrl(url) ⇒ string

Build WFS GetCapabilities URL.

Kind: global function

Param Type Description
url string WFS server URL

parseCapabilities(xml) ⇒ object

Parse WFS GetCapabilities response.

Kind: global function
Returns: object - Parsed capabilities (version, outputFormats, typeNames, resultTypes, resultPaging).

Param Type Description
xml string – XML string

chooseOutputFormat(formats) ⇒ string | null

Choose the output format.

Kind: global function

Param Type Description
formats Array.<string> List of output formats

buildGetFeatureUrl(url, capabilities, paging) ⇒ string

Build WFS GetFeature URL.

Kind: global function

Param Type Default Description
url string WFS server URL (ideally with typeName parameter)
capabilities object Server capabilities
paging boolean false Whether to set a start index and max feature count