Skip to content

Commit

Permalink
Clarifies installation/runtime options for CI/PROD images. (#15320)
Browse files Browse the repository at this point in the history 
After PROD images were added, some of the flags had two meanings

These behaved differently in PROD image and CI image and were the
source of confusion especially when start-airflow command was used.

For PROD image, the image can be customized during image building,
and packages could be installed from .whl or .sdist packages
available in `docker-context-files`. This
is used at CI and dockerhub building time to produce image built
packages that were prepared using local sources.

The CI image is always built from local sources but airflow can
be removed and re-installed at runtime from pypi.
Both airflow and provider packages can be installed
from .whl or .sdist packages available in dist folder. This is
used in CI to test current provider packages with older
Airflow released (2.0.0) and to test provider packages locally.

After the change we have two sets of flags/variables:

PROD image (building image):

* install-airflow-version, install-airflow-reference,
  install-from-docker-context-files

CI image (runtime):

* use-airflow-version, use-packages-from-dist

That should avoid confusion and failures of commands such as
`start-airflow` that is used to test provider packages and
airflow itself.
  • Loading branch information
@potiuk
potiuk authored Apr 19, 2021
1 parent 3584455 commit 36ba5b6
Show file tree
Hide file tree
Showing 22 changed files with 289 additions and 319 deletions.
6 changes: 3 additions & 3 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -551,12 +551,12 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- name: "Install and test provider packages and airflow via wheel files"
run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
env:
INSTALL_AIRFLOW_VERSION: "wheel"
USE_AIRFLOW_VERSION: "wheel"
PACKAGE_FORMAT: "wheel"
- name: "Install and test provider packages and airflow on Airflow 2.0 files"
run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
env:
INSTALL_AIRFLOW_VERSION: "2.0.0"
USE_AIRFLOW_VERSION: "2.0.0"
PACKAGE_FORMAT: "wheel"

prepare-test-provider-packages-sdist:
Expand Down Expand Up @@ -597,7 +597,7 @@ ${{ hashFiles('.pre-commit-config.yaml') }}"
- name: "Install and test provider packages and airflow via sdist files"
run: ./scripts/ci/provider_packages/ci_install_and_test_provider_packages.sh
env:
INSTALL_AIRFLOW_VERSION: "sdist"
USE_AIRFLOW_VERSION: "sdist"
PACKAGE_FORMAT: "sdist"

tests-helm:
Expand Down
113 changes: 55 additions & 58 deletions BREEZE.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1250,6 +1250,8 @@ This is the current syntax for `./breeze <./breeze>`_:
'breeze \
--github-image-id 209845560' - pull/use image with RUN_ID
Most flags are applicable to the shell command as it will run build when needed.
####################################################################################################
Expand Down Expand Up @@ -1310,41 +1312,21 @@ This is the current syntax for `./breeze <./breeze>`_:
2.7 3.5 3.6 3.7 3.8
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
In CI image, installs Airflow (in entrypoint) from PIP released version or using
the installation method specified (sdist, wheel, none).
In PROD image the installation of selected method or version happens during image building.
For PROD image, the 'none' options is not valid.
One of:
Uses differen version of Airflow when building PROD image.
2.0.1 2.0.0 1.10.15 1.10.14 1.10.12 1.10.11 1.10.10 1.10.9 none wheel sdist
When 'none' is used, you can install airflow from local packages. When building image,
airflow package should be added to 'docker-context-files' and
--install-from-docker-context-files flag should be used. When running an image, airflow
package should be added to dist folder and --install-packages-from-dist flag should be used.
2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
If specified, installs Airflow directly from reference in GitHub. This happens at
image building time in production image and at container entering time for CI image.
Installs Airflow directly from reference in GitHub when building PROD image.
This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
--installation-method INSTALLATION_METHOD
Method of installing airflow for production image - either from the sources ('.')
Method of installing Airflow in PROD image - either from the sources ('.')
or from package 'apache-airflow' to install from PyPI.
Default in Breeze is to install from sources. One of:
. apache-airflow
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
--install-packages-from-dist
If specified it will look for packages placed in dist folder and it will install the
packages after installing Airflow. This is useful for testing provider
packages.
--upgrade-to-newer-dependencies
Upgrades PIP packages to latest versions available without looking at the constraints.
Expand Down Expand Up @@ -1392,12 +1374,12 @@ This is the current syntax for `./breeze <./breeze>`_:
are needed by the extras. When you build image during the development (which is
default in Breeze) all providers are installed by default from sources.
You can disable it by adding this flag but then you have to install providers from
wheel packages via --install-packages-from-dist flag.
wheel packages via --use-packages-from-dist flag.
--disable-pypi-when-building
Disable installing Airflow from pypi when building. If you use this flag and want
to install Airflow, you have to install it from packages placed in
'docker-context-files' and use --install-from-local-files-when-building flag.
'docker-context-files' and use --install-from-docker-context-files flag.
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
Expand Down Expand Up @@ -1452,7 +1434,7 @@ This is the current syntax for `./breeze <./breeze>`_:
--disable-pip-cache
Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
--install-from-local-files-when-building
--install-from-docker-context-files
This flag is used during image building. If it is used additionally to installing
Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
in the 'docker-context-files' folder. The same flag can be used during entering the image in
Expand Down Expand Up @@ -1839,6 +1821,22 @@ This is the current syntax for `./breeze <./breeze>`_:
Flags:
--use-airflow-version AIRFLOW_SPECIFICATION
In CI image, installs Airflow at runtime from PIP released version or using
the installation method specified (sdist, wheel, none). When 'none' is used,
airflow is just removed. In this case airflow package should be added to dist folder
and --use-packages-from-dist flag should be used.
2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
--use-packages-from-dist
In CI image, if specified it will look for packages placed in dist folder and
it will install the packages after entering the image.
This is useful for testing provider packages.
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
--load-example-dags
Include Airflow example dags.
Expand Down Expand Up @@ -2043,12 +2041,12 @@ This is the current syntax for `./breeze <./breeze>`_:
are needed by the extras. When you build image during the development (which is
default in Breeze) all providers are installed by default from sources.
You can disable it by adding this flag but then you have to install providers from
wheel packages via --install-packages-from-dist flag.
wheel packages via --use-packages-from-dist flag.
--disable-pypi-when-building
Disable installing Airflow from pypi when building. If you use this flag and want
to install Airflow, you have to install it from packages placed in
'docker-context-files' and use --install-from-local-files-when-building flag.
'docker-context-files' and use --install-from-docker-context-files flag.
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
Expand Down Expand Up @@ -2103,7 +2101,7 @@ This is the current syntax for `./breeze <./breeze>`_:
--disable-pip-cache
Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
--install-from-local-files-when-building
--install-from-docker-context-files
This flag is used during image building. If it is used additionally to installing
Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
in the 'docker-context-files' folder. The same flag can be used during entering the image in
Expand Down Expand Up @@ -2526,50 +2524,49 @@ This is the current syntax for `./breeze <./breeze>`_:
Assume 'quit' answer to all questions.
****************************************************************************************************
Choose different Airflow version to install or run
Install different Airflow version during PROD image build
-a, --install-airflow-version INSTALL_AIRFLOW_VERSION
In CI image, installs Airflow (in entrypoint) from PIP released version or using
the installation method specified (sdist, wheel, none).
Uses differen version of Airflow when building PROD image.
In PROD image the installation of selected method or version happens during image building.
For PROD image, the 'none' options is not valid.
One of:
2.0.1 2.0.0 1.10.15 1.10.14 1.10.12 1.10.11 1.10.10 1.10.9 none wheel sdist
When 'none' is used, you can install airflow from local packages. When building image,
airflow package should be added to 'docker-context-files' and
--install-from-docker-context-files flag should be used. When running an image, airflow
package should be added to dist folder and --install-packages-from-dist flag should be used.
2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist
-t, --install-airflow-reference INSTALL_AIRFLOW_REFERENCE
If specified, installs Airflow directly from reference in GitHub. This happens at
image building time in production image and at container entering time for CI image.
Installs Airflow directly from reference in GitHub when building PROD image.
This can be a GitHub branch like master or v1-10-test, or a tag like 2.0.0a1.
--installation-method INSTALLATION_METHOD
Method of installing airflow for production image - either from the sources ('.')
Method of installing Airflow in PROD image - either from the sources ('.')
or from package 'apache-airflow' to install from PyPI.
Default in Breeze is to install from sources. One of:
. apache-airflow
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
--install-packages-from-dist
If specified it will look for packages placed in dist folder and it will install the
packages after installing Airflow. This is useful for testing provider
packages.
--upgrade-to-newer-dependencies
Upgrades PIP packages to latest versions available without looking at the constraints.
--continue-on-pip-check-failure
Continue even if 'pip check' fails.
****************************************************************************************************
Use different Airflow version at runtime in CI image
--use-airflow-version AIRFLOW_SPECIFICATION
In CI image, installs Airflow at runtime from PIP released version or using
the installation method specified (sdist, wheel, none). When 'none' is used,
airflow is just removed. In this case airflow package should be added to dist folder
and --use-packages-from-dist flag should be used.
2.0.1 2.0.0 1.10.15 1.10.14 wheel sdist none
--use-packages-from-dist
In CI image, if specified it will look for packages placed in dist folder and
it will install the packages after entering the image.
This is useful for testing provider packages.
--no-rbac-ui
Disables RBAC UI when Airflow 1.10.* is installed.
****************************************************************************************************
Credentials
Expand Down Expand Up @@ -2618,12 +2615,12 @@ This is the current syntax for `./breeze <./breeze>`_:
are needed by the extras. When you build image during the development (which is
default in Breeze) all providers are installed by default from sources.
You can disable it by adding this flag but then you have to install providers from
wheel packages via --install-packages-from-dist flag.
wheel packages via --use-packages-from-dist flag.
--disable-pypi-when-building
Disable installing Airflow from pypi when building. If you use this flag and want
to install Airflow, you have to install it from packages placed in
'docker-context-files' and use --install-from-local-files-when-building flag.
'docker-context-files' and use --install-from-docker-context-files flag.
--additional-extras ADDITIONAL_EXTRAS
Additional extras to pass to build images The default is no additional extras.
Expand Down Expand Up @@ -2678,7 +2675,7 @@ This is the current syntax for `./breeze <./breeze>`_:
--disable-pip-cache
Disables GitHub PIP cache during the build. Useful if GitHub is not reachable during build.
--install-from-local-files-when-building
--install-from-docker-context-files
This flag is used during image building. If it is used additionally to installing
Airflow from PyPI, the packages are installed from the .whl and .tar.gz packages placed
in the 'docker-context-files' folder. The same flag can be used during entering the image in
Expand Down
10 changes: 0 additions & 10 deletions CI.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -253,16 +253,6 @@ You can use those variables when you try to reproduce the build locally.
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| ``HOST_HOME`` | | | | Home directory on the host. |
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| Image variables |
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| ``INSTALL_AIRFLOW_VERSION`` | | | | Installs Airflow version from PyPI when |
| | | | | building image. Can be "none" to skip airflow |
| | | | | installation so that it can be installed from |
| | | | | locally prepared packages. |
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| ``INSTALL_AIRFLOW_REFERENCE`` | | | | Installs Airflow version from GitHub |
| | | | | branch or tag. |
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| Version suffix variables |
+-----------------------------------------+-------------+-------------+------------+-------------------------------------------------+
| ``VERSION_SUFFIX_FOR_PYPI`` | | | | Version suffix used during provider |
Expand Down
14 changes: 0 additions & 14 deletions Dockerfile.ci
View file
Original file line number Diff line number Diff line change
Expand Up @@ -260,9 +260,6 @@ ENV AIRFLOW_PRE_CACHED_PIP_PACKAGES=${AIRFLOW_PRE_CACHED_PIP_PACKAGES}
ARG INSTALL_PROVIDERS_FROM_SOURCES="true"
ENV INSTALL_PROVIDERS_FROM_SOURCES=${INSTALL_PROVIDERS_FROM_SOURCES}

ARG INSTALL_FROM_DOCKER_CONTEXT_FILES=""
ENV INSTALL_FROM_DOCKER_CONTEXT_FILES=${INSTALL_FROM_DOCKER_CONTEXT_FILES}

ARG INSTALL_FROM_PYPI="true"
ENV INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI}

Expand Down Expand Up @@ -367,17 +364,6 @@ RUN if [[ ${INSTALL_FROM_PYPI} == "true" ]]; then \
bash /scripts/docker/install_airflow.sh; \
fi

# Only copy install_from_docker_context_files.sh to not invalidate cache on other script changes
COPY scripts/docker/install_from_docker_context_files.sh /scripts/docker/install_from_docker_context_files.sh

# If wheel files are found in /docker-context-files during installation
# they are also installed additionally to whatever is installed from Airflow.
COPY docker-context-files/ /docker-context-files/

RUN if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then \
bash /scripts/docker/install_from_docker_context_files.sh; \
fi

# Copy all the www/ files we need to compile assets. Done as two separate COPY
# commands so as otherwise it copies the _contents_ of static/ in to www/
COPY airflow/www/webpack.config.js ${AIRFLOW_SOURCES}/airflow/www/
Expand Down
Loading

0 comments on commit 36ba5b6

Please sign in to comment.