Skip to content

Latest commit

 

History

History
104 lines (77 loc) · 6.54 KB

elasticsearch.md

File metadata and controls

104 lines (77 loc) · 6.54 KB
Raw

Elasticsearch

Sync overview

Output schema

Elasticsearch is a Lucene based search engine that's a type of NoSql storage.
Documents are created in an index, similar to a tablein a relation database.

The output schema matches the input schema of a source. Each source stream becomes a destination index.
For example, in with a relational database source -
The DB table name is mapped to the destination index. The DB table columns become fields in the destination document.
Each row becomes a document in the destination index.

Data type mapping

See Elastic documentation for detailed information about the field types This section should contain a table mapping each of the connector's data types to Airbyte types. At the moment, Airbyte uses the same types used by JSONSchema. string, date-time, object, array, boolean, integer, and number are the most commonly used data types.

Integration Type Airbyte Type Notes
text string more info
date date-time more info
object object more info
array array more info
boolean boolean more info
numeric integer more info
numeric number more info

Features

This section should contain a table with the following format:

Feature Supported?(Yes/No) Notes
Full Refresh Sync yes
Incremental Sync yes
Replicate Incremental Deletes no
SSL connection yes
SSH Tunnel Support yes

Performance considerations

Batch/bulk writes are performed. Large records may impact performance.
The connector should be enhanced to support variable batch sizes.

Getting started

Requirements

  • Elasticsearch >= 7.x
  • Configuration
    • Endpoint URL [ex. https://elasticsearch.savantly.net:9423]
    • Username [optional] (basic auth)
    • Password [optional] (basic auth)
    • CA certificate [optional]
    • Api key ID [optional]
    • Api key secret [optional]
  • If authentication is used, the user should have permission to create an index if it doesn't exist, and/or be able to create documents

CA certificate

Ca certificate may be fetched from the Elasticsearch server from /usr/share/elasticsearch/config/certs/http_ca.crt Fetching example from dockerized Elasticsearch: docker cp es01:/usr/share/elasticsearch/config/certs/http_ca.crt . where es01 is a container's name. For more details please visit https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

Setup guide

Enter the endpoint URL, select authentication method, and whether to use 'upsert' method when indexing new documents.

Connection via SSH Tunnel

Airbyte has the ability to connect to an Elastic instance via an SSH Tunnel. The reason you might want to do this because it is not possible (or against security policy) to connect to your Elastic instance directly (e.g. it does not have a public IP address).

When using an SSH tunnel, you are configuring Airbyte to connect to an intermediate server (a.k.a. a bastion sever) that does have direct access to the Elastic instance. Airbyte connects to the bastion and then asks the bastion to connect directly to the server.

Using this feature requires additional configuration, when creating the source. We will talk through what each piece of configuration means.

  1. Configure all fields for the source as you normally would, except SSH Tunnel Method.
  2. SSH Tunnel Method defaults to No Tunnel (meaning a direct connection). If you want to use an SSH Tunnel choose SSH Key Authentication or Password Authentication.
    1. Choose Key Authentication if you will be using an RSA private key as your secret for establishing the SSH Tunnel (see below for more information on generating this key).
    2. Choose Password Authentication if you will be using a password as your secret for establishing the SSH Tunnel.
  3. SSH Tunnel Jump Server Host refers to the intermediate (bastion) server that Airbyte will connect to. This should be a hostname or an IP Address.
  4. SSH Connection Port is the port on the bastion server with which to make the SSH connection. The default port for SSH connections is 22, so unless you have explicitly changed something, go with the default.
  5. SSH Login Username is the username that Airbyte should use when connection to the bastion server. This is NOT the TiDB username.
  6. If you are using Password Authentication, then SSH Login Username should be set to the password of the User from the previous step. If you are using SSH Key Authentication TiDB password, but the password for the OS-user that Airbyte is using to perform commands on the bastion.
  7. If you are using SSH Key Authentication, then SSH Private Key should be set to the RSA Private Key that you are using to create the SSH connection. This should be the full contents of the key file starting with -----BEGIN RSA PRIVATE KEY----- and ending with -----END RSA PRIVATE KEY-----.

CHANGELOG

Version Date Pull Request Subject
0.1.6 2022-10-26 18341 enforce ssl connection on cloud
0.1.5 2022-10-24 18177 add custom CA certificate processing
0.1.4 2022-10-14 17805 add SSH Tunneling
0.1.3 2022-05-30 14640 Include lifecycle management
0.1.2 2022-04-19 11752 Reduce batch size to 32Mb
0.1.1 2022-02-10 10256 Add ExitOnOutOfMemoryError connectors
0.1.0 2021-10-13 7005 Initial release.