diff --git a/docs/user/tutorials/dev_container.rst b/docs/user/tutorials/dev_container.rst new file mode 100644 index 00000000..9ff812a9 --- /dev/null +++ b/docs/user/tutorials/dev_container.rst @@ -0,0 +1,268 @@ +Developer Containers +==================== + +.. _ioc_change_types: + +Types of Changes +---------------- + +Containerized IOCs can be modified in 3 distinct places (in order of decreasing +frequency of change but increasing complexity): + +#. The IOC instance: this means making changes to the IOC instance folders + which appear in the ``iocs`` folder of a domain repository. e.g.: + + - changing the EPICS DB (or the ``ibek`` files that generate it) + - altering the IOC boot script (or the ``ibek`` files that generate it) + - changing the version of the Generic IOC used in values.yaml + - for Kubernetes: the values.yaml can override any settings used by helm + so these can also be adjusted on a per IOC instance basis. + - for Kubernetes: changes to the global values.yaml + file found in ``beamline-chart``, these affect all IOCs in the domain. + +#. The Generic IOC: i.e. altering how the Generic IOC container image + is built. This means making changes to an ``ioc-XXX`` + source repo and publishing a new version of the container image. + Types of changes include: + + - changing the EPICS base version + - changing the versions of EPICS support modules compiled into the IOC binary + - adding new support modules + - altering the system dependencies installed into the container image + +#. The dependencies - Support modules used by the Generic IOC. Changes to support + module repos. To make use of these changes would require: + + - publishing a new release of the support module, + - updating and publishing the Generic IOC + - updating and publishing the IOC instance + +For all of the above, the epics-containers approach allows +local testing of the changes before going through the publishing cycle. +This allows us to have a fast 'inner loop' of development and testing. + +Also, epics-containers provides a mechanism for creating a separate workspace for +working on all of the above elements in one place. + +Need for a Developer Container +------------------------------ + +The earlier tutorials were firmly in the realm of ``1`` above. +It was adequate for us to install a container platform, IDE and python +and that is all we needed. + +Once you get to level ``2`` changes you need to have compilers and build tools +installed. You might also require system level dependencies. AreaDetector, +that we used earlier has a long list of system dependencies that need to be +installed in order to compile it. Traditionally we have installed all of these +onto developer workstations or separately compiled the dependencies as part of +the build. + +These tools and dependencies will differ from one Generic IOC to the next. + +When using epics-containers we don't need to install any of these tools or +dependencies on our local machine. Instead we can use a developer container, +and in fact our Generic IOC *is* our developer container. + +When the CI builds a Generic IOC it creates +`two targets `_: + +:developer: this target installs all the build tools and build time dependencies + into the container image. It then compiles the support modules and IOC. + +:runtime: this target installs only the runtime dependencies into the container. + It also extracts the built runtime assets from the developer target. + +The developer stage of the build is a necessary step in order to get a +working runtime container. However, we choose to keep this stage as an additional +build target and it then becomes a perfect candidate for a developer container. + +VSCode has excellent support for using a container as a development environment. +The next section will show you how to use this feature. Note that you can use +any IDE that supports remote development in a container, you could also +simply launch the developer container in a shell and use it via CLI only. + +Starting a Developer Container +------------------------------ + +For this section we will work with the ADSimDetector Generic IOC that we +used in previous tutorials. Let's go and fetch a version of the Generic IOC +source and build it locally. + +For the purposes of this tutorial we will place the source in a folder right +next to your test beamline ``bl01t`` folder. We will also be getting a +specific version of the Generic IOC source so that future changes don't break +this tutorial: + +.. code-block:: bash + + # starting from folder bl01t so that the clone is next to bl01t + cd .. + git clone --recursive git@github.com:epics-containers/ioc-adsimdetector.git -b 2023.10.6b1 + cd ioc-adsimdetector + ec dev build + +The last step uses one of the ``ec dev`` sub commands to build the developer +target of the container to your local container cache. This will take a few +minutes to complete. A philosophy of epics-containers is that Generic IOCs +build all of their own support. This is to avoid problematic dependency trees. +For this reason building something as complex as AreaDetector will take a +few minutes when you first build it. + +A nice thing about containers is that the build is +cached so that a second build will be almost instant unless you have changed +something that requires some steps to be rebuilt. + +The ``ec dev`` commands are a set of convenience commands +for working on Generic IOCs from *outside* of the container. These commands +are useful for debugging container builds: although most work is done inside +the container, you will need these commands if it fails to build. + +Once built, open the project in VSCode: + +.. code-block:: bash + + code . + +When it opens, VSCode may prompt you to open in a devcontainer. If not then click +the green icon in the bottom left of the VSCode window and select +``Reopen in Container``. + +You should now be *inside* the container. All terminals started in VSCode will +be inside the container. Every file that you open with the VSCode editor +will be inside the container. + +There are some caveats because some folders are mounted from the host file +system. For example, the ``ioc-adsimdetector`` project folder +is mounted into the container as a volume. It is mounted under +``/epics/ioc-adsimdetector``. This means that you can edit the source code +from your local machine and the changes will be visible inside the container and +outside the container. This is a good thing as you should consider the container +filesystem to be a temporary filesystem that will be destroyed when the container +deleted. + +Now that you are *inside* the container you have access to the tools built into +it, this includes ``ibek``. The first command you should run is: + +.. code-block:: bash + + ibek ioc build + +This generates an IOC source tree in the ``ioc`` folder under your +``ioc-adsimdetector`` folder and compiles it. Note that the IOC code is +boilerplate, but that the ``src/Makefile`` is generated according to the +support modules this Generic IOC contains. You can go and take a look at +the Makefile and see that it contains ``dbd`` and ``lib`` references for each +of the support modules in the container. +See ``/epics/ioc-adsimdetector/ioc/iocApp/src/Makefile`` + +You will note that the ``ioc`` folder is greyed out in the VSCode explorer. This +is because it is in ``.gitignore`` and it is purely generated code. If you +particularly needed to customize the contents of the ioc source tree then +you can remove it from ``.gitignore`` and commit your changes to the repo. These +changes would then always get loaded for every instance of the Generic IOC. + +The Generic IOC should now be ready to run inside of the container. To do this: + +.. code-block:: bash + + cd ioc + ./start.sh + +You will just see the default output of a Generic IOC that has no Instance +configuration. Next we will add some instance configuration from one of the +IOC instances in the ``bl01t`` beamline. + +Let's now add some other folders to our VSCode workspace to make it easier to +work with ``bl01t`` and to investigate the container. + +Adding the Beamline to the Workspace +------------------------------------ + +To meaningfully test the Generic IOC we will need an instance to test it +against. We will use the ``bl01t`` beamline that you already made. + +In VSCode click the ``File`` menu and select ``Add Folder to Workspace``. +Navigate to ``/repos`` and you will see all the peers of your ``ioc-adsimdetector`` +folder (see `container-layout` below) . Choose the ``bl01t`` folder and add it to the +workspace - you may see an error but if so clicking "reload window" will +clear it. + +Also take this opportunity to add the folder ``/epics`` to the workspace. + +You can now easily browse around the ``/epics`` folder and see all the +support modules and epics-base. This will give you a feel for the layout of +files in the container. Here is a summary (where WS is the root +of your host folders): + +.. _container-layout: + +.. list-table:: Developer Container Layout + :widths: 25 35 45 + :header-rows: 1 + + * - Path Inside Container + - Host Mount Path + - Description + + * - /epics/support + - N/A + - root of compiled support modules + + * - /epics/epics-base + - N/A + - compiled epics-base + + * - /epics/ioc-adsimdetector + - WS/ioc-adsimdetector + - Source repository for the Generic IOC + + * - /epics/ioc + - WS/ioc-adsimdetector/ioc + - soft link to IOC source tree + + * - /epics/ibek + - N/A + - All ibek *Support yaml* files + + * - /epics/pvi + - N/A + - all PVI definitions from support modules + + * - /epics/opi + - N/A + - all OPI files (generated or copied from support) + + * - /repos + - WS + - all peers to Generic IOC source repo + + +Now that we have the beamline repo visible in our container we can +easily supply some instance configuration to the Generic IOC. +Try the following: + +.. code:: + + cd /epics/ioc-adsimdetector + rm -r ioc/config + ln -s /repos/bl01t/iocs/bl01t-ea-ioc-02/config ioc + ioc/start.sh + +This removed the boilerplate config and replaced it with the config from +the IOC instance bl01t-ea-ioc-02. Note that we used a soft link, this +means we can edit the config, restart the IOC to test it and the changes +will already be in place in the beamline repo. You can even open a shell +onto the beamline repo and commit and push the changes. + +Wrapping Up +----------- + +We now have a tidy development environment for working on the Generic IOC, +IOC Instances and even the support modules inside the Generic IOC, all in one +place. We can easily test our changes in place too. In particular note that +we are able to test changes without having to go through a container build +cycle. + +In the following tutorials we will look at how to make changes at each of the +3 levels listed in `ioc_change_types`. \ No newline at end of file diff --git a/docs/user/tutorials/ioc_changes1.rst b/docs/user/tutorials/ioc_changes1.rst new file mode 100644 index 00000000..32199c37 --- /dev/null +++ b/docs/user/tutorials/ioc_changes1.rst @@ -0,0 +1,151 @@ +Changing the IOC Instance +========================= + + +.. Warning:: + + This tutorial is out of date and will be updated in November 2023. + + +This tutorial will make a very simple change to the example IOC ``bl01t-ea-ioc-01``. +This is a type 1. change from the above list, types 2, 3 will be covered in the +following 2 tutorials. + +We are going to add a hand crafted EPICS DB file to the IOC instance. This will +be a simple record that we will be able to query to verify that the change +is working. + +Make the following changes in your test IOC config folder +(``bl01t/iocs/bl01t-ea-ioc-01/config``): + +1. Add a file called ``extra.db`` with the following contents. + IMPORTANT replace [$USER] with your username: + + .. code-block:: text + + record(ai, "[$USER]-EA-IOC-01:TEST") { + field(DESC, "Test record") + field(DTYP, "Soft Channel") + field(SCAN, "Passive") + field(VAL, "1") + } + +2. Add the following line to the ``st.cmd`` file after the last ``dbLoadRecords`` + line: + + .. code-block:: text + + dbLoadRecords(config/extra.db) + +Locally Testing Your changes +---------------------------- + +You can immediately test your changes by running the IOC locally. The following +command will run the IOC locally using the config files in your test IOC config +folder: + +.. code-block:: bash + + ec dev ioc-launch iocs/bl01t-ea-ioc-01 + +This will launch Generic IOC container specified in the ``bl01t-ea-ioc-01`` +helm chart and mount into it the local config specified in +``/iocs/bl01t-ea-ioc-01/config``. + +If all is well you should see your iocShell prompt and you can test your change +from another terminal (VSCode menus -> Terminal -> New Terminal) like so: + +.. code-block:: bash + + caget $USER-EA-IOC-01:TEST + +If you see the value 1 then your change is working. + +.. note:: + + If you also wanted to make local changes + to the Generic IOC itself you could clone the Generic IOC source repo, + locally build the container image and then use ``ec dev ioc-launch`` as + follows: + + .. code-block:: bash + + # advanced example - not part of this tutorial + cd + git clone git@github.com:epics-containers/ioc-adsimdetector.git + cd ioc-adsimdetector + # this makes a local image with tag :local + ec dev build + cd ../bl01t + ec dev ioc-launch iocs/bl01t-ea-ioc-01 ../ioc-adsimdetector + + +Note you can see your running IOC in podman using this command: + +.. code-block:: bash + + podman ps + +You should see a container named bl01t-ea-ioc-01 and also a another one with a +random name and an image called ``localhost/vsc-work...``. The latter is the +container that is running your developer environment. + +If you would like to take a look inside the container you can run a bash shell +in the container like this: + +.. code-block:: bash + + podman exec -it bl01t-ea-ioc-01 bash + +When you type exit on the iocShell the container will stop and and be removed. + +.. _local_deploy_ioc: + +Deploying a Beta IOC Instance to The Cluster +============================================ + +In ``05_deploy_example`` we deployed a tagged version of the IOC instance to +the cluster. This the correct way to deploy a production IOC instance as it +means there is a record of version of the IOC instance in the Helm Chart +OCI registry and you can always roll back to that version if needed. + +However, it is also possible to directly deploy a version of the IOC instance +from your local machine to the cluster. +This is useful for testing changes to the IOC instance +before publishing a new version. In this case +your IOC will be given a beta tag in the cluster, indicating that it has +not yet been released. + +To deploy your changes direct to the cluster use the following command: + +.. code-block:: bash + + ec ioc deploy-local iocs/bl01t-ea-ioc-01 + +You will get a warning that this is a temporary deployment and you will see that +the version number will look something like ``2023.3.29-b14.29`` this +indicates that this is a beta deployment made at 14:29 on 29th March 2023. + +Now when you ask for the IOCs running in your domain you should see your IOC +with beta version listed: + +.. code-block:: bash + + $ ec ps -w + POD VERSION STATE RESTARTS STARTED IP GENERIC_IOC_IMAGE + bl01t-ea-ioc-01-7d7c5bc759-5bjsr 2023.3.29-b14.29 Running 0 2023-03-29T14:29:18Z 192.168.0.32 ghcr.io/epics-containers/ioc-adsimdetector-linux-runtime:23.3.4 + +You can check it is working as before (replace the IP with yours +from the above command): + +.. code-block:: bash + + export EPICS_CA_ADDR_LIST=192.168.0.32 + caget $USER-EA-IOC-01:TEST + +Once you are happy with your changes you can push and tag your beamline repo. +This will publish a new version of the IOC instance helm chart to the OCI helm +registry. You can then deploy the versioned IOC instance to the cluster. + + +