Skip to content

Tracer-Cloud/nextflow-test-pipelines

Repository files navigation

Tracer
Reliable Nextflow Pipelines with Tracer Observability

Try tracer.cloud instantly in the sandbox environment.

Linux aarch64 Ubuntu Linux x86_64 Ubuntu Linux aarch64 Amazon Linux Linux x86_64 Amazon Linux macOS ARM64 macOS Intel x86 Codespaces

Docker Image CI Status Latest Binary Release

Open in GitHub Codespaces


Tracer is a system-level observability platform purpose-built for scientific computing. With a one-line Linux client install and instant dashboards, Tracer gives you deep insights into pipeline performance and cost optimization.


Why Tracer?

  • Reliable Nextflow pipelines: Fully-tested examples that run seamlessly across all environments
  • Automated CI/CD: Regularly validated pipelines for consistent functionality
  • Easy Installation: Automated scripts for quick setup
  • Fast Community Support: Open an issue and get a response within 24 hours
  • GitHub Codespaces Compatible: Run examples directly in Codespaces

Quickstart (Sandbox)

Get started instantly:

  1. Visit the Tracer Sandbox
  2. Follow the onboarding instructions to launch your first monitored pipeline

Getting Started

1. Install Tracer

Install Tracer on your system (one-time setup):

Note: Root privileges required

curl -sSL https://install.tracer.cloud | sh && source ~/.bashrc && source ~/.zshrc

2. Initialize the Tracer Client

Start the Tracer client to initialize a pipeline and enable monitoring:

Note: Root privileges required

tracer init

3. Run a Pipeline

You can use your own pipelines or try our ready-to-run examples. For the smoothest onboarding, follow the sandbox instructions or pick a pipeline for your OS below:

We provide a variety of examples for different compute environments, written in Bash, Python, Nextflow, WDL, and CWL.

Recommended: Start with a simple Nextflow fastquorum pipeline.


Monitor Your Pipeline

Track your pipeline's progress in real time via the Tracer dashboard. Access it through the 'Open Grafana Dashboard' button during onboarding.

The dashboard provides:

  • Execution metrics
  • Pipeline stages
  • Status updates

What Makes Tracer Different?

Tracer is a cutting-edge observability platform designed for scientific computing. Unlike general-purpose monitoring tools, Tracer is purpose-built for scientific pipelines, offering:

  • Intelligent organization and labeling of pipelines, runs, and steps
  • Zero code changes: runs directly on Linux
  • Support for any programming language
  • Seamless integration with diverse environments (AlphaFold, Slurm, Airflow, Nextflow, Bash, and more)
  • Enterprise-grade security: your data never leaves your infrastructure

This is especially valuable for environments like AWS Batch, where tracking processes across containers is challenging and logs can be lost.


Key Features

  • Time and cost per dataset processed
  • Step-level execution duration and bottleneck identification
  • Cost attribution across pipelines, teams, and environments (dev, CI/CD, prod)

These insights help you optimize complex scientific toolchains that traditionally lack proper observability.


Mission

"The goal of Tracer's Rust client is to equip scientists and engineers with DevOps intelligence to efficiently harness massive computational power for humanity's most critical challenges."


For more details, see the tracer client and our website.