Skip to content

Commit

Permalink
Add readme hint regarding split requirements.txt
Browse files Browse the repository at this point in the history
  • Loading branch information
hbruch committed Nov 30, 2023
1 parent 85ef7e2 commit 599f800
Showing 1 changed file with 20 additions and 2 deletions.
22 changes: 20 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,24 @@ It uses [Dagster](https://dagster.io) to retrieve and transform several datasour

*Note:* This repo to is designed to be run as a part of the entire IPL platform, [as defined in the `ipl-orchestration` repo](https://github.com/mobidata-bw/ipl-orchestration). But you can also run it in a standalone fashion.

### Prerequisites

If you intend to run this project locally via `dagster dev`, you need to have python 3.10 or 3.11 installed. Python 3.12 is not yet supported by dagster.

To install all required libraries, use

`pip install -r requirements.txt`

Note: `requirements.txt` imports `requirements-pipeline.txt` and `requirements-dagster.txt`, which include the
dependencies for the different dagster services. `pipeline.Dockerfile` and `dagster.Dockerfile` just import these respective requirements.

In addition, you need a postgres database into which the datasets are loaded.
This database can be started via

`docker compose up -f docker-compose.dev.yml`

### Running

To start this dagster project in interactive develepment mode, you should use a DAGSTER_HOME other than
this project directory, as a) the dagster.yml defines a postgres storage for dagster run information
and is usually intended for prod use, and b) a number of files is generated in the temporary dagster
Expand All @@ -18,10 +36,10 @@ directories which would impact your IDE responsiveness if it's indexing new file
$ DAGSTER_HOME=/tmp/DAGSTER_HOME dagster dev
```

or via docker-compose, which is the way it is itended to be deployed with:
or via `docker compose`, which is the way it is itended to be deployed with:

```sh
$ docker-compose up --build
$ docker compose up --build
```

Note that the config differs in that for docker-compose, `workspace.docker.yaml` and `dagster.docker.yaml` will be used, which configure a postgres db as dagster storage, whild `dagster dev` will use sqlite and temporary folders.

0 comments on commit 599f800

Please sign in to comment.