PaSh: Light-touch Data-Parallel Shell Processing

A system for parallelizing POSIX shell scripts.

Quick Jump: Repo Structure | Running PaSh | Installation | Testing | Community & More

Repo Structure

This repo hosts the core pash development. The structure is as follows:

• annotations: DSL characterizing commands, parallelizability study, and associated annotations.
• compiler: Shell-Dataflow translations and associated parallelization transformations.
• docs: Design documents, tutorials, installation instructions, etc.
• evaluation: Shell pipelines and example scripts used for the evaluation.
• runtime: Runtime component — e.g., eager, split, and assocaited combiners.
• scripts: Scripts related to continuous integration, deployment, and testing.

Running PaSh

To parallelize, say, ./evaluation/hello-world.sh with parallelization width of 2, from the top-level directory of the repository run:

./pa.sh ./evaluation/hello-world.sh

Run ./pa.sh --help to get more information about the available commands. Read a longer tutorial, see docs/tutorial.

Installation

Docker: The easiest way to play with pash today is using Docker:

curl img.pash.ndr.md | docker load; docker run --name pash-playground -it pash/18.04

PaSh can be found in the container's /pash directory, so run cd pash; git pull to fetch the latest updates; more information in the pash-on-docker guide.

Alternatively, you can built the Docker container from scratch by running

git clone [email protected]:andromeda/pash.git
cd pash/scripts
docker build -t "pash/18.04" .
# Then you launch the container as above
docker run --name pash-playground -it pash/18.04

This should build a fresh docker image running pash.

Linux: Alternatively, if you're on an Ubuntu 18.04, run:

curl up.pash.ndr.md | bash

This runs the install script in scritps/install.sh. We have only tested this script on Ubuntu 18.04 (like the one used for the docker container and on AWS).

Tests

To execute the current tests, one-liner shell scripts, simply run:

cd compiler
./test_evaluation_scripts.sh

Community & More

Mailing Lists:

• Discussion: Join this mailing list for discussing all things pash
• Commits: Join this mailing list for commit notifications

Development/contributions:

• Contribution guide: docs/contrib
• Continuous Integration Server: ci.pash.ndr.md