Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for environment.yml/lockfiles #26

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
*.py[co]

rpmbuild/
_version.py

# Packages
*.egg
*.eggs
*.egg-info
dist
build
eggs
parts
Expand Down
20 changes: 20 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
FROM continuumio/miniconda3

SHELL ["/bin/bash", "-c"]

ENV TINI_VERSION v0.19.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini /tini
RUN chmod +x /tini


COPY ./dev-env.yml environment.yml
RUN conda env create --quiet -n conda-rpms-dev --file environment.yml

RUN echo "source activate conda-rpms-dev" > ~/.bashrc
ENV PATH /opt/conda/envs/conda-rpms-dev/bin:$PATH

COPY . /conda-rpms
WORKDIR /conda-rpms
RUN pip install -e .[test]

ENTRYPOINT ["/tini", "--"]
56 changes: 25 additions & 31 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,57 +1,51 @@
Create RPMs from a conda-gitenv
===============================
Create RPMs from a conda environment
===================================

conda-rpms is designed to convert a resolved [conda-gitenv](https://github.com/SciTools/conda-gitenv) environment into a collection of RPM specs suitable for deployment on compatible platforms (Red Hat Enterprise Linux, Fedora, etc.).

A high level description of the motivation for the creation of RPMs from a [conda-gitenv](https://github.com/SciTools/conda-gitenv) is described in the [centrally managed scientific software workflow](https://rawgit.com/pelson/conda-rpms/master/tmp_overview_docs/scitools-env-workflow.html) document.

To transform the conda distribution to RPM specs, conda-rpms uses tools including git-python, Jinja and conda.
conda-rpms is designed to convert a conda environment into a collection of RPM specs suitable for deployment on compatible platforms (Red Hat Enterprise Linux, Fedora, etc.).

Because a built RPM knows its destined installation location, a number of RPM abstractions have been made that enable us to retain conda's hard-linking and relocatability benefits.


Usage
=====

There are two conda-rpms command entrypoints.
There is a single conda-rpms command entrypoint:

`python -m conda_rpms.build_rpm_structure` creates the RPM specs and sources.

`python -m conda_rpms.build` is a general purpose rpmbuild wrapper that inspects the RPM build directory for RPMs that have already been built, and then builds those that haven't. This is a general purpose tool that has nothing to do with conda - if you are aware of such a tool already existing, please raise an issue let us know! `;)`
`python -m conda_rpms.generate` creates the RPM specs and sources.

`conda-rpms` no longer builds specs to RPMs, this is left to the user as may
require context-specific congfiguration. However, typically it can be achieved
with something like the example below.
Assuming you have a [conda-lock](https://github.com/conda-incubator/conda-lock)
named `~/data_science.lock`:
```
# create the specs and copy source files to ./dist/SPECS and ./dist/SOURCES
python -m conda_rpms.generate ~/data_science.lock data_science.yml --output dist --modulefile_template modulefile.tmpl
# build the RPMs to ./dist/RPMS
rpmbuild -bb --define "_topdir $(pwd)/dist" dist/SPECS/*.spec
```

RPM Types
=========

Package RPM
-----------

RPM name format: ``<RPM prefix>-pkg-<pkg name>-<pkg version>-<pkg build id>``
RPM name format: ``<RPM namespace>-pkg-<pkg name>-<pkg version>-<pkg build id>``

A package RPM represents the conda "package cache" (the thing that normally lives in `<conda root prefix>/pkgs/<pkg name>-<pkg version>-<pkg build id>`).
A package RPM *does not* express its dependencies and can not be usefully installed as a standalone entity.

Tagged environment RPM
----------------------

RPM name format: ``<RPM prefix>-env-<env name>-tag-<env tag>``

A tagged environment RPM represents a resolved conda environment.
It depends on all Package RPMs that should be installed in order to produce a working environment. The tagged environment RPM knows its target installation prefix, and uses conda functionality at install time to link the Package RPMs to the desired installation prefix.

Labelled environment RPM
------------------------

RPM name format: `<RPM prefix>-env-<env name>-label-<label>`
Environment RPM
---------------

A labelled environment is simply a meta package which depends on one, and only one, Tagged Environment RPM.
Unlike the immutable tagged environment, a labelled environment will tend to change to reference newer tags as time moves on.
This change is reflected in the labelled environment's `version`, meaning that it is only possible to ever have one labelled environment RPM installed per environment name.
RPM name format: ``<RPM namespace>-env-<env name>``

An environment RPM represents a resolved conda environment.
It depends on all Package RPMs that should be installed in order to produce a working environment. The environment RPM knows its target installation prefix, and uses conda functionality at install time to link the Package RPMs to the desired installation prefix.

conda-rpms without conda-gitenv
===============================

In some situations, it would be valuable to be able to produce RPMs from a standard conda yaml environment definition, rather than through conda-gitenv.
Whilst we haven't yet implemented this obvious feature, it shouldn't be much of an extension to the existing codebase, and PRs would be happily received.
Testing
=======

```pytest cov=conda_rpms tests/```
14 changes: 0 additions & 14 deletions circle.yml

This file was deleted.

1 change: 0 additions & 1 deletion conda_rpms/__init__.py

This file was deleted.

52 changes: 0 additions & 52 deletions conda_rpms/build.py

This file was deleted.

Loading