OpenFn/Lightning

Lightning ⚡ (aka "OpenFn v2") is a workflow automation platform that's used to automate critical business processes and integrate information systems. From last-mile services to national-level reporting, it boosts efficiency & effectiveness while enabling secure, stable, scalable interoperability and data integration at all levels.

Use it online at app.openfn.org

Explore in a sandbox on demo.openfn.org

Or learn more at docs.openfn.org

OpenFn Lightning is:

• the latest version of OpenFn: first launched in 2014, it's been tried and tested by NGOs and governments in 40+ countries
• fully open source: there's no "community edition" and "premium edition", you get the same product whether you are self-hosting or using the OpenFn.org software-as-a-service
• the leading DPGA certified Digital Public Good for workflow automation
• a Digital Square certified Global Good for Health

Contents

• Demo
• Features
  • Build
  • Monitor
  • Manage
  • Roles and permissions
  • Roadmap
• Getting Started
  • Run via Docker
  • Deploy on external infrastructure
  • Dev on Lightning locally
• Security and Standards
• Contribute to this project
  • Pick up an issue
  • Open a pull request
  • Generate the docs pages
• Server Specs for Self-Hosting
• Benchmarking
• Troubleshooting
  • Problems with environment variables
  • Problems with Postgres
  • Problems with Debian
  • Problems with Docker
  • Problems with Rambo
• Support

Demo

Watch a short demo video or explore a public sandbox with the login details below, but please note that this deployment is reset every night at 12:00:00 UTC and is 100% publicly accessible. Don't build anything you want to keep, or keep private!

username: [email protected]
password: welcome12345

Features

Build

Plan and build workflows using Lightning's visual interface to quickly define when, where and what you want your automation to do.

Use our CLI to quickly build, edit and deploy projects from the comfort of your own code editor.

Monitor

Monitor all workflow activity in one place.

• Filter and search runs to identify issues that need addressing and follow how a specific request has been processed
• Configure alerts to be notified on run failures
• Receive a project digest for a daily/weekly/monthly summary of your project activity

Manage

Manage users and access by project.

Roles and permissions

Authorization is a central part of Lightning. As such, users are given different roles which determine what level of access they have for resources in the application. For more details about roles and permissions in Lightning, please refer to our documentation.

Roadmap

View our public GitHub project to see what we're working on now and what's coming next.

Getting Started

• If you only want to RUN Lightning on your own server, we recommend using Docker.
• If you want to DEPLOY Lightning, we recommend Docker builds and Kubernetes.
• If you want to CONTRIBUTE to the project, we recommend running Lightning on your local machine.

Run via Docker

1. Install the latest version of Docker
2. Clone this repo using git
3. Setup PostgreSQL database with: docker compose build && docker compose run --rm web mix ecto.migrate
4. Run Lightning and PostgresSQL with: docker compose up

By default the application will be running at localhost:4000.

See "Problems with Docker" for additional troubleshooting help. Note that you can also create your own docker-compose.yml file, configuring a postgres database and using a pre-built image from Dockerhub.

Deploy on external infrastructure

Head to the Deploy section of our docs site to get started.

For technical guidelines, see deployment considerations for more detailed information.

Dev on Lightning locally

Clone the repo and optionally set ENVs

git clone [email protected]:OpenFn/Lightning.git # or from YOUR fork!
cd Lightning
cp .env.example .env # and adjust as necessary!

Take note of database names and ports in particular—they've got to match across your Postgres setup and your ENVs. You can run lightning without any ENVs assuming a vanilla postgres setup (see below), but you may want to make adjustments.

Database Setup

If you're already using Postgres locally, create a new database called lightning_dev, for example.

If you'd rather use Docker to set up a Postgres DB, create a new volume and image:

docker volume create lightning-postgres-data

docker create \
  --name lightning-postgres \
  --mount source=lightning-postgres-data,target=/var/lib/postgresql/data \
  --publish 5432:5432 \
  -e POSTGRES_PASSWORD=postgres \
  postgres:15.3-alpine

docker start lightning-postgres

Elixir & Ecto Setup

We use asdf to configure our local environments. Included in the repo is a .tool-versions file that is read by asdf in order to dynamically make the specified versions of Elixir and Erlang available. You'll need asdf plugins for Erlang, NodeJs Elixir and k6.

We use libsodium for encoding values as required by the Github API. You'll need to install libsodium in order for the application to compile.

For Mac Users:

brew install libsodium

For Debian Users:

sudo apt-get install libsodium-dev

You can find more on how to install libsodium here

asdf install  # Install language versions
mix local.hex
mix deps.get
mix local.rebar --force
mix ecto.create # Create a development database in Postgres
mix ecto.migrate
[[ $(uname -m) == 'arm64' ]] && CPATH=/opt/homebrew/include LIBRARY_PATH=/opt/homebrew/lib mix deps.compile enacl # Force compile enacl if on M1
[[ $(uname -m) == 'arm64' ]] && mix compile.rambo # Force compile rambo if on M1
mix lightning.install_runtime
mix lightning.install_schemas
mix lightning.install_adaptor_icons
npm install --prefix assets

Run the app

Lightning is a web app. To run it in interactive Elixir mode, start the development server by running with your environment variables by running:

iex -S mix phx.server

or if you have set up custom environment variables, run:

env $(cat .env | grep -v "#" | xargs ) iex -S mix phx.server

Once the server has started, head to localhost:4000 in your browser.

Run the tests

Before the first time running the tests, you need a test database setup.

MIX_ENV=test mix ecto.create

And then after that run the tests using:

MIX_ENV=test mix test

We also have test.watch installed which can be used to rerun the tests on file changes.

Security and Standards

We use a host of common Elixir static analysis tools to help us avoid common pitfalls and make sure we keep everything clean and consistent.

In addition to our test suite, you can run the following commands:

• mix format --check-formatted Code formatting checker, run again without the --check-formatted flag to have your code automatically changed.
• mix dialyzer Static analysis for type mismatches and other common warnings. See dialyxir.
• mix credo --strict --all Static analysis for consistency, and coding standards. See Credo.
• mix sobelow Check for commonly known security exploits. See Sobelow.
• MIX_ENV=test mix coveralls Test coverage reporter. This command also runs the test suite, and can be used in place of mix test when checking everything before pushing your code. See excoveralls.

For convenience there is a verify mix task that runs all of the above and defaults the MIX_ENV to test.

For more guidance on security best practices for workflow automation implementations, check out OpenFn Docs: docs.openfn.org/documentation/getting-started/security

Contribute to this project

First, thanks for being here! You're contributing to a digital public good that will always be free and open source and aimed at serving innovative NGOs, governments, and social impact organizations the world over! You rock. ❤️

FYI, Lightning is built in Elixir, harnessing the Phoenix Framework. Currently, the only unbundled dependency is a PostgreSQL database.

If you'd like to contribute to this projects, follow the steps below:

Pick up an issue

Read through the existing issues, assign yourself to the issue you have chosen. Leave a comment on the issue to let us know you'll be working on it, and if you have any questions of clarifications that would help you get started ask them there - we will get back to you as soon as possible.

If there isn't already an issue for the feature you would like to contribute, please start a discussion in our community forum.

Open a pull request

1. Clone the Lightning repository, then fork it.

2. Run through setting up your environment and make your changes.

3. Make sure you have written your tests and updated /CHANGELOG.md (in the 'Unreleased' section, add a short description of the changes you are making, along with a link to your issue).

4. Open a draft pull request by clicking "Contribute > Open Pull Request" from your forked repository. Fill out the pull request template (this will be added automatically for you), then make sure to self-review your code and go through the 'Review checklist'. Don't worry about the QA checkbox, our product manager Amber will tick that once she has reviewed your PR. You can leave any notes for the reviewer in a comment.

5. Once you're ready to submit a pull request, you can mark your draft PR as 'Ready for review' and assign @stuartc or @taylordowns2000.

Generate the docs pages

You can generate the HTML and EPUB documentation locally using:

mix docs and opening doc/index.html in your browser.

Server Specs for Self-Hosting

For recommend server specifications for self-hosting of Lightning, head to the deployment planning section of the documentation or check out this self-hosting thread on our community forum.

Benchmarking

We are using k6 to benchmark Lightning. Under benchmarking folder you can find a script for benchmarking Webhook Workflows.

See Benchmarking for more detailed information.

Troubleshooting

Problems with environment variables

For troubleshooting custom environment variable configuration it's important to know how an Elixir app loads and modifies configuration. The order is as follows:

1. Stuff in config.exs is loaded.
2. That is then modified (think: overwritten) by stuff your ENV-specific config: dev.exs, prod.exs or test.exs.
3. That is then modified by runtime.exs which is where you are allowed to use System.env()
4. Finally init/2 (if present in a child application) gets called (which takes the config which has been set in steps 1-3) when that child application is started during the parent app startup defined in application.ex.

Problems with Postgres

If you're having connecting issues with Postgres, check the database section of your .env to ensure the DB url is correctly set for your environment — note that composing a DB url out of other, earlier declared variables, does not work while using xargs.

Problems with Debian

If you're getting this error on debian

==> earmark_parser
Compiling 1 file (.yrl)
/usr/lib/erlang/lib/parsetools-2.3.1/include/yeccpre.hrl: no such file or directory
could not compile dependency :earmark_parser, "mix compile" failed. You can recompile this dependency with "mix deps.compile earmark_parser", update it with "mix deps.update earmark_parser" or clean it with "mix deps.clean earmark_parser"

You need to install erlang development environment sudo apt install erlang-dev refer to this issue

Problems with Docker

Versions

The build may not work on old versions of Docker and Docker compose. It has been tested against:

Docker version 20.10.17, build 100c701
Docker Compose version v2.6.0

Starting from scratch

If you're actively working with docker, you start experiencing issues, and you would like to start from scratch you can clean up everything and start over like this:

# To remove any ignored files and reset your .env to it's example
git clean -fdx && cp .env.example .env
# You can skip the line below if you want to keep your database
docker compose down --rmi all --volumes

docker compose build --no-cache web && \
  docker compose create --force-recreate

docker compose run --rm web mix ecto.migrate
docker compose up

Apple Silicon

When running docker compose up on Apple Silicon (M1, M2, M3), you might encounter the following error:

> [web 20/20] RUN npm install --prefix assets:
0.091 rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
0.091  Trace/breakpoint trap
------
failed to solve: process "/bin/sh -c npm install --prefix assets" did not complete successfully: exit code: 133

You can solve this by setting the default docker platform to linux/amd64, i.e. DOCKER_DEFAULT_PLATFORM=linux/amd64. You can read more on this here: https://stackoverflow.com/questions/71040681/qemu-x86-64-could-not-open-lib64-ld-linux-x86-64-so-2-no-such-file-or-direc

You might also run into:

[notice] Application ssl exited: exited in: :ssl_app.start(:normal, [])
 ** (EXIT) an exception was raised:
     ** (ArgumentError) could not call Module.put_attribute/3 because the module Lightning.MixProject is already compiled
         (elixir 1.16.2) lib/module.ex:2360: Module.assert_not_readonly!/2
         (elixir 1.16.2) lib/module.ex:2041: Module.__put_attribute__/5
         (ssl 11.1.4) ssl_app.erl:35: :ssl_app.stop/1
         (kernel 9.2.4) application_master.erl:293: :application_master.start_it_old/4

** (MatchError) no match of right hand side value: {:error, {:ssl, {:bad_return, {{:ssl_app, :start, [:normal, []]}, {:EXIT, {%ArgumentError{message: "could not call Module.put_attribute/3 because the module Lightning.MixProject is already compiled"}, [{Module, :assert_not_readonly!, 2, [file: ~c"lib/module.ex", line: 2360]}, {Module, :__put_attribute__, 5, [file: ~c"lib/module.ex", line: 2041]}, {:ssl_app, :stop, 1, [file: ~c"ssl_app.erl", line: 35]}, {:application_master, :start_it_old, 4, [file: ~c"application_master.erl", line: 293]}]}}}}}}

You can resolve this by setting ERL_FLAGS="+JPperf true" env to the failing stage. You can follow this thread on our community forum for more info: https://community.openfn.org/t/lightning-prebuilt-images-throw-no-matching-manifest-for-linux-arm64-v8-in-the-manifest-list-entries/465/15

Problems with Rambo

When running mix compile.rambo on Apple Silicon (an Apple M1/M2, macarm, aarch64-apple-darwin) and encountering the following error:

** (RuntimeError) Rambo does not ship with binaries for your environment.

    aarch64-apple-darwin22.3.0 detected

Install the Rust compiler so a binary can be prepared for you.

    lib/mix/tasks/compile.rambo.ex:89: Mix.Tasks.Compile.Rambo.compile!/0
    lib/mix/tasks/compile.rambo.ex:51: Mix.Tasks.Compile.Rambo.run/1
    (mix 1.14.2) lib/mix/task.ex:421: anonymous fn/3 in Mix.Task.run_task/4
    (mix 1.14.2) lib/mix/cli.ex:84: Mix.CLI.run_task/2

You can resolve this error by installing the Rust compiler using Homebrew. Run the following command in your terminal: brew install rust

If you have already compiled Rambo explicitly via mix compile.rambo, and you are still seeing the following error:

sh: /path_to_directory/Lightning/_build/dev/lib/rambo/priv/rambo: No such file or directory
sh: line 0: exec: /path_to_directory/Lightning/_build/dev/lib/rambo/priv/rambo: cannot execute: No such file or directory

You can try renaming deps/rambo/priv/rambo-mac to deps/rambo/priv/rambo.

If neither of the approaches above work, please raise an issue.

Support

If you have any questions, feedback, or issues, please:

• Post on the OpenFn Community at community.openfn.org
• Open an issue directly on this GitHub Repo: github.com/OpenFn/Lightning/issues