Skip to content

Commit

Permalink
create beamline tutorial complete
Browse files Browse the repository at this point in the history
  • Loading branch information
gilesknap committed Oct 25, 2023
1 parent d7d50a3 commit c961741
Show file tree
Hide file tree
Showing 9 changed files with 490 additions and 184 deletions.
39 changes: 0 additions & 39 deletions docs/user/explanations/cli-tools.rst

This file was deleted.

14 changes: 7 additions & 7 deletions docs/user/explanations/repositories.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Source and Registry Locations
- Beamline Source repos: ``https://gitlab.diamond.ac.uk/controls/containers/beamline/<beamline>``
- Accelerator Source repos: ``https://gitlab.diamond.ac.uk/controls/containers/accelerator/<domain>``
- Generic IOC Container Images: ``ghcr.io/epics-containers/<Generic ioc>``
- IOC Instance Helm Charts: ``helm-test.diamond.ac.uk/iocs/<domain>/<ioc instance>``
- epics-containers Helm Charts: ``https://github.com/orgs/epics-containers/packages?repo_name=ec-helm-charts``

Where to Keep Source Code
-------------------------
Expand Down Expand Up @@ -64,13 +64,13 @@ kept in a public repository as it is a live example of a domain repo.
Where to put Registries
-----------------------

Generic IOC Container Images
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Generic IOC Container Images and Source Repos
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

TODO: usually GHCR but internal supported for license e.g. Nexus Repository Manager
Usually GHCR but internal supported for license e.g. Nexus Repository Manager

IOC Instance Helm Charts
~~~~~~~~~~~~~~~~~~~~~~~~
IOC Instance Domain Repos
~~~~~~~~~~~~~~~~~~~~~~~~~

TODO: Can be GHCR but internal supported supported e.g. Nexus Repository Manager
Internal git registry or private GitHub registry.

3 changes: 2 additions & 1 deletion docs/user/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ side-bar.
tutorials/create_beamline
tutorials/deploy_example
tutorials/ioc_changes
tutorials/release_beamline
tutorials/generic_ioc
tutorials/debug_generic_ioc
tutorials/test_generic_ioc
Expand Down Expand Up @@ -58,7 +59,6 @@ side-bar.
explanations/kubernetes_cluster
explanations/docs-structure
explanations/repositories
explanations/cli-tools
explanations/repos

+++
Expand All @@ -72,6 +72,7 @@ side-bar.
:maxdepth: 1

reference/faq
reference/environment
reference/cli
reference/ioc_helm_chart
reference/k8s_resources
Expand Down
79 changes: 67 additions & 12 deletions docs/user/reference/cli.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,74 @@
Command Line Interface for IOC Management
=========================================

Built in to the devcontainer is epics-containers-cli, a command line interface
for assisting with building and managing IOCs in the cluster.
The python project ``epics-containers-cli`` is installed as part of the
Python section of the initial tutorial `../tutorials/setup_workstation`.
It provides a command line function ``ec`` with support for:

The CLI is just a thin wrapper around these tools:-
- building and debugging Generic IOC Containers
- deploying, managing and debugging IOC Instances

- kubectl
- helm
- podman (or docker)
The CLI is just a thin wrapper around the underlying tools that do the real
work:

To help with learning the above tools. The CLI will print out the commands it
executes before running them. This can be suppressed with the ``--quiet`` flag,
or by setting the ``K8S_QUIET=true`` in your environment.
:kubectl: the command line interface to the Kubernetes APIs
:helm: the command line interface to the Kubernetes Helm package manager
:podman (or docker): CLIs for container engines
:git: the git version control system client

The CLI entrypoint is ``ec``. To see the available commands, run ``ec --help``.
Much of the functionality is available through subcommands dev and ioc.
Below is pasted the current version of help for the CLI.
``ec`` is useful because it saves typing and provides a consistent interface
when working on multiple beamlines. This is because it uses the environment
setup by the beamline repo's ``environment.sh`` script. See `environment`.

To see the available commands, run ``ec --help``.
Much of the functionality is available through subcommands dev and ioc. To get
full details of a given command type the full command plus ``--help``. For
example:

.. raw:: html

<pre>$ ec dev launch --help
<b> </b>
<b> </b><font color="#A2734C"><b>Usage: </b></font><b>ec dev launch [OPTIONS] IOC_INSTANCE </b>
<b> </b>
Launch an IOC instance using configuration from a domain repo. Or by passing a
generic IOC image ID. Can be used for local testing of IOC instances. You may
find the devcontainer a more convenient way to do this.

<font color="#AAAAAA">╭─ Arguments ───────────────────────────────────────────────────────────────────╮</font>
<font color="#AAAAAA">│ </font><font color="#C01C28">*</font> ioc_instance <font color="#A2734C"><b>DIRECTORY</b></font> local IOC definition folder from domain │
<font color="#AAAAAA">│ repo │</font>
<font color="#AAAAAA">│ [default: None] │</font>
<font color="#AAAAAA">│ </font><font color="#80121A">[required] </font> │
<font color="#AAAAAA">╰───────────────────────────────────────────────────────────────────────────────╯</font>
<font color="#AAAAAA">╭─ Options ─────────────────────────────────────────────────────────────────────╮</font>
<font color="#AAAAAA">│ </font><font color="#2AA1B3"><b>--execute</b></font> <font color="#A2734C"><b>TEXT </b></font> command to execute in the container. │
<font color="#AAAAAA">│ Defaults to executing the IOC │</font>
<font color="#AAAAAA">│ [default: /epics/ioc/start.sh; bash] │</font>
<font color="#AAAAAA">│ </font><font color="#2AA1B3"><b>--target</b></font> <font color="#6C4C32"><b>[developer|runtime]</b></font> choose runtime or developer target │
<font color="#AAAAAA">│ [default: developer] │</font>
<font color="#AAAAAA">│ </font><font color="#2AA1B3"><b>--image</b></font> <font color="#A2734C"><b>TEXT </b></font> override container image to use │
<font color="#AAAAAA">│ </font><font color="#2AA1B3"><b>--tag</b></font> <font color="#A2734C"><b>TEXT </b></font> override image tag to use. │
<font color="#AAAAAA">│ [default: None] │</font>
<font color="#AAAAAA">│ </font><font color="#2AA1B3"><b>--args</b></font> <font color="#A2734C"><b>TEXT </b></font> Additional args for podman/docker, │
<font color="#AAAAAA">│ &apos;must be quoted&apos; │</font>
<font color="#AAAAAA">│ </font><font color="#2AA1B3"><b>--ioc-name</b></font> <font color="#A2734C"><b>TEXT </b></font> container name override. Use to run │
<font color="#AAAAAA">│ multiple instances │</font>
<font color="#AAAAAA">│ [default: test-ioc] │</font>
<font color="#AAAAAA">│ </font><font color="#2AA1B3"><b>--help</b></font> <font color="#A2734C"><b> </b></font> Show this message and exit. │
<font color="#AAAAAA">╰───────────────────────────────────────────────────────────────────────────────╯</font>
</pre>

It may be instructive to understand the underlying tools and how they are
are being called. For this reason ``ec`` supports a ``-v`` option to show
the underlying commands being executed. e.g.

.. raw:: html

<pre>$ ec -v ps
<font color="#5F8787">kubectl get namespace p38-iocs -o name</font>
<font color="#5F8787">kubectl -n p38-iocs get pod -l is_ioc==True -o custom-columns=IOC_NAME:metadata.labels.app,VERSION:metadata.labels.ioc_version,STATE:status.phase,RESTARTS:status.containerStatuses[0].restartCount,STARTED:metadata.managedFields[0].time</font>
IOC_NAME VERSION STATE RESTARTS STARTED
bl38p-ea-ioc-03 2023.10.6 Running 0 2023-10-25T14:07:44Z
bl38p-ea-panda-02 2023.10.25-b16.24 Running 0 2023-10-25T15:24:41Z
</pre>
128 changes: 128 additions & 0 deletions docs/user/reference/environment.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,128 @@
The Environment Configuration File
==================================

``environment.sh`` is a configuration file that is provided in each domain
(beamline or accelerator) repository. It is used to set up the environment
such that the ``epics-containers-cli`` will interact with the correct
cluster (or local server if you are not using Kubernetes).

An important part of creating a new domain repository is to edit the
``environment.sh`` so that it suits the domain you are targetting.

There are 3 sections to the file as follows:

Environment Variables Setup
---------------------------

The first section defines a number of environment variables. These should
be adjusted to suit your domain. The variables are as follows:


Required Variables
~~~~~~~~~~~~~~~~~~

- **EC_REGISTRY_MAPPING**: defines a mapping between git source repositories and
container repositories. This is used to determine where a Generic IOC
container image is pushed to (or pulled from). If you are using GitHub then
you could use the following value:

- ``EC_REGISTRY_MAPPING='github.com=ghcr.io'``

Which means containers whose source repository is at:

- ``GitHub/organisation/my_generic_ioc``

will be pushed to:

- ``ghcr.io/organisation/my_generic_ioc:VERSION``.

You can have multiple mappings if needed by separating them with a space.

- **EC_NAMESPACE**: defines the namespace in a Kubernetes Cluster that your IOC
Instances will be deployed to. When you come to set up a cluster you will
need to create a namespace for your domain. This is the name you should
use here. If you are not using Kubernetes then you can leave this as
``EC_NAMESPACE=`` and this will deploy IOC Instances to the local server's
docker or podman instance.

- **EC_DOMAIN_REPO**: this is a link back to the repository that defines this
domain. For example the bl38p reference beamline repository uses
``[email protected]:epics-containers/bl38p.git``. This variable
is used by ``ec`` to fetch versioned IOC Instance from the repo and deploy
time.


Optional Variables
~~~~~~~~~~~~~~~~~~

- **EC_LOG_URL**: if you have a centralized logging service with a web UI then
you can set this variable to the URL of the web UI. This will then be
displayed when the command ``ec ioc log-history <ioc-name>`` is run. The
ioc name is added to the URL using ``{ioc-name}`` as a placeholder e.g.

- ``EC_LOG_URL='https://graylog2.diamond.ac.uk/search?rangetype=relative&fields``
``=message%2Csource&width=1489&highlightMessage=&relative=172800&q=pod_name%3A``
``{ioc_name}*'``

- **EC_CONTAINER_CLI**: this sets the name of the container CLI to use. supported
options are ``podman`` or ``docker``. If not set then ``ec`` will try to
determine which one to use. You would only need this variable if you have
both podman and docker installed and you want to use one over the other, or
if you want to use a different container CLI such as ``singularity``.
IMPORTANT: the application you reference must have docker compatible CLI
(at least for common functions).

- **EC_DEBUG**: causes the ``ec`` command to output debug information for all
commands. For more targetted debugging you can use ``ec -d ...``.

Installation of ``ec``
----------------------

The second section of the ``environment.sh`` file is used to install the
``ec`` command by pip installing the ``epics-containers-cli`` package. The
``blxxi-template`` project comes with a suggested way of doing this, but
it would probably be best to have ``ec`` installed globally on your
workstation and then omit this section from your ``environment.sh`` files.

Perhaps the simplest way to achieve this is to install ``ec`` into your user
space using the following command:

.. code:: bash
pip install --user epics-containers-cli
Then add the following to your ``.bashrc`` file:

.. code:: bash
export PATH=$PATH:$HOME/.local/bin
Connecting to a Namespace on your Kubernetes Cluster
----------------------------------------------------

The third section of the ``environment.sh`` sets up how the ``kubectl`` command
will connect to a namespace on your Kubernetes cluster. This usually involves
setting the ``KUBECONFIG`` environment variable to point to a file that contains
the cluster configuration.

When we set up a cluster in the tutorials we will create a namespace for your
and discuss how to update ``environment.sh`` to connect to it.

If you are connecting to your own facility's cluster then you will need to
ask your admins for the correct configuration.

If you are not using Kubernetes then you can leave this section out.

.. note::

DLS users: the module system is used to connect us to each beamline/accelerator
cluster. The example ``environment.sh`` file in ``bl38p`` shows how to do this
see https://github.com/epics-containers/bl38p/blob/main/environment.sh

One other thing that is useful is to set up command line completion for the
Kubernetes tools ``kubectl`` and ``helm``. See the end of the bl38p
``environment.sh`` file for an example of how to do this at
https://github.com/epics-containers/bl38p/blob/main/environment.sh.



Loading

0 comments on commit c961741

Please sign in to comment.