Skip to content

Commit

Permalink
Update README
Browse files Browse the repository at this point in the history
  • Loading branch information
rowanseymour committed Jul 8, 2022
1 parent 75fc168 commit 4af922e
Showing 1 changed file with 47 additions and 52 deletions.
99 changes: 47 additions & 52 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,43 +1,43 @@
# RapidPro Indexer
# Indexer

[![Build Status](https://github.com/nyaruka/rp-indexer/workflows/CI/badge.svg)](https://github.com/nyaruka/rp-indexer/actions?query=workflow%3ACI)
[![codecov](https://codecov.io/gh/nyaruka/rp-indexer/branch/main/graph/badge.svg)](https://codecov.io/gh/nyaruka/rp-indexer)
[![Go Report Card](https://goreportcard.com/badge/github.com/nyaruka/rp-indexer)](https://goreportcard.com/report/github.com/nyaruka/rp-indexer)

Simple service for indexing RapidPro contacts into ElasticSearch.
Service for indexing RapidPro/TextIt contacts into Elasticsearch.

# Deploying
## Deploying

As Indexer is a Go application, it compiles to a binary and that binary along with the config file is all
As it is a Go application, it compiles to a binary and that binary along with the config file is all
you need to run it on your server. You can find bundles for each platform in the
[releases directory](https://github.com/nyaruka/rp-indexer/releases). You should only run a single indexer
[releases directory](https://github.com/nyaruka/rp-indexer/releases). You should only run a single
instance for a deployment.

Indexer can run in two modes:
It can run in two modes:

1) the default mode, which simply queries the ElasticSearch database, finds the most recently
modified contact, then on a schedule queries the `contacts_contact` table on the RapidPro
modified contact, then on a schedule queries the `contacts_contact` table in the
database for contacts to add or delete. You should run this as a long running service which
constantly keeps ElasticSearch in sync with your RapidPro contacts.
constantly keeps ElasticSearch in sync with your contacts.

2) a rebuild mode, started with `--rebuild`. This builds a brand new index from nothing, querying
all contacts on RapidPro. Once complete, this switches out the alias for the contact index
with the newly build index. This can be run on a cron (in parallel with the mode above) to rebuild
your index occasionally to get rid of bloat.

# Configuration
## Configuration

Indexer uses a tiered configuration system, each option takes precendence over the ones above it:
The service uses a tiered configuration system, each option takes precendence over the ones above it:

1. The configuration file
2. Environment variables starting with `INDEXER_`
3. Command line parameters

We recommend running Indexer with no changes to the configuration and no parameters, using only
We recommend running it with no changes to the configuration and no parameters, using only
environment variables to configure it. You can use `% rp-indexer --help` to see a list of the
environment variables and parameters and for more details on each option.

## RapidPro Configuration
### RapidPro

For use with RapidPro, you will want to configure these settings:

Expand All @@ -48,9 +48,42 @@ Recommended settings for error reporting:

* `INDEXER_SENTRY_DSN`: The DSN to use when logging errors to Sentry

# Development
### Reference

Once you've checked out the code, you can build Indexer with:
These are the configuration options that can be provided as parameters or environment variables. If using environment
varibles, convert to uppercase, replace dashes with underscores and prefix the name with `INDEXER_`, e.g. `-log-level`
becomes `INDEXER_LOG_LEVEL`.

```
-cleanup
whether to remove old indexes after a rebuild
-db string
the connection string for our database (default "postgres://localhost/rapidpro?sslmode=disable")
-debug-conf
print where config values are coming from
-elastic-url string
the url for our elastic search instance (default "http://localhost:9200")
-help
print usage information
-index string
the alias for our contact index (default "contacts")
-librato-username
the Librato username for metrics reporting
-librato-token
the Librato token for metrics reporting
-log-level string
the log level, one of error, warn, info, debug (default "info")
-poll int
the number of seconds to wait between checking for updated contacts (default 5)
-rebuild
whether to rebuild the index, swapping it when complete, then exiting (default false)
-sentry-dsn string
the sentry configuration to log errors to, if any
```

## Development

Once you've checked out the code, you can build the service with:

```
go build github.com/nyaruka/rp-indexer/cmd/rp-indexer
Expand All @@ -69,41 +102,3 @@ To run all of the tests:
```
go test ./... -p=1
```

# Usage

```
Indexes RapidPro contacts to ElasticSearch
Usage of indexer:
-cleanup
whether to remove old indexes after a rebuild
-db string
the connection string for our database (default "postgres://localhost/rapidpro?sslmode=disable")
-debug-conf
print where config values are coming from
-elastic-url string
the url for our elastic search instance (default "http://localhost:9200")
-help
print usage information
-index string
the alias for our contact index (default "contacts")
-log-level string
the log level, one of error, warn, info, debug (default "info")
-poll int
the number of seconds to wait between checking for updated contacts (default 5)
-rebuild
whether to rebuild the index, swapping it when complete, then exiting (default false)
-sentry-dsn string
the sentry configuration to log errors to, if any
Environment variables:
INDEXER_CLEANUP - bool
INDEXER_DB - string
INDEXER_ELASTIC_URL - string
INDEXER_INDEX - string
INDEXER_LOG_LEVEL - string
INDEXER_POLL - int
INDEXER_REBUILD - bool
INDEXER_SENTRY_DSN - string
```

0 comments on commit 4af922e

Please sign in to comment.