Merge pull request #75 from epics-containers/2024-rework

rewrite change a generic ioc tutorial

docs/user/explanations/ioc-source.rst

@@ -3,6 +3,69 @@
 Dev Container vs Runtime Container
 ==================================
 
-TODO: explain how and why ioc-xxxx is mounted in dev container and what happens
-to the ioc folder.
+Introduction
+------------
+
+The dev container is where all development of IOCs and support modules will
+take place. The runtime container is where the IOC will run when deployed
+to a target system.
+
+The dev container mounts several host folders into the container to achieve
+the following goals:
+
+- make the developer container look as similar as possible to the runtime
+  container
+- allow the developer to make changes and recompile things without having
+  to rebuild the container
+- make sure that all useful changes occur in the host filesystem so that
+  they are not lost when the container is rebuilt or deleted
+
+The details of which folders are mounted where in the container are
+shown here: `container-layout`.
+
+The ioc-XXX project folder is found in the container at ``/workspaces/ioc-XXX``,
+along with all of it's peers (because the parent folder is mounted
+at ``/workspaces``).
+
+
+The ioc Folder
+--------------
+
+The ioc folder contains the Generic IOC source code. It is typically the same
+for all Generic IOCs but is included in the ioc-XXX repo in /ioc so that it can be
+modified if necessary.
+
+At container build time this folder is copied into the container at
+``/epics/generic-source/ioc`` and it is compiled so that the binaries are
+available at runtime.
+
+In the dev container the ``/epics/generic-source`` folder has the project
+folder ioc-XXX mounted over the top of it. This means:
+
+- the project folder ioc-XXX is mounted in two locations in the container
+  - ``/workspaces/ioc-XXX``
+  - ``/epics/generic-source``
+- the ioc source folder ``/epics/generic-source/ioc`` is also mounted over
+  and now contains the source only. The compiled binaries are no longer
+  visible inside the dev container.
+
+It is for this reason that a newly created dev container needs to have the IOC
+binaries re-compiled. But this is a good thing, because now any changes you
+make to the IOC source code can be compiled and tested, but also those
+changes are now visible on the host filesystem inside the project folder
+``ioc-XXX/ioc``. This avoids loss of work.
+
+Finally the ``ioc`` folder is always soft linked from ``/epics/ioc`` so that
+the source and binaries are always in a known location.
+
+Summing Up
+----------
+
+The above description makes things sound rather complicated. However,
+you can for the most part ignore the details and just remember:
+
+- use ``/epics/ioc`` to compile and run the IOC.
+- you are free to make changes to the above folder and recompile
+- the changes you make will be visible on the host filesystem in the
+  original project folder.
 

docs/user/index.rst

@@ -75,6 +75,7 @@ side-bar.
             :maxdepth: 1
 
             reference/faq
+            reference/troubleshooting
             reference/configuration
             reference/environment
             reference/cli

docs/user/reference/troubleshooting.rst

@@ -0,0 +1,23 @@
+Troubleshooting
+===============
+
+Permissions issues with GitHub
+-------------------------------
+
+Problem: in the devcontainer you see the following error:
+
+.. code-block:: none
+
+    [email protected]: Permission denied (publickey).
+    fatal: Could not read from remote repository.
+
+Solution: you may need to add your github ssh key to the ssh-agent as
+follows:
+
+.. code-block:: none
+
+    eval "$(ssh-agent -s)"
+    ssh-add ~/.ssh/id_rsa
+
+Where ``id_rsa`` is the name of your private key file you use for connecting
+to GitHub.

docs/user/tutorials/ioc_changes2.rst

@@ -1,10 +1,6 @@
 Changing a Generic IOC
 ======================
 
-.. warning ::
-
-    TODO: This tutorial is a work in progress. It is not yet complete.
-
 This is a type 2 change from `ioc_change_types`.
 
 The changes that you can make in an IOC instance are limited to what
@@ -24,123 +20,228 @@ Some of the reasons for doing this are:
     want to add support for a second device, this is allowed but you should
     consider the alternative of creating a new Generic IOC.
     If you keep your Generic IOCs simple and focused on a single device, they
-    will be smaller and there will be less of them. IOCs can still be
-    linked via CA and this is preferable to recompiling a Generic IOC
+    will be smaller and there will be less of them. IOCs' records can still be
+    linked via CA links and this is preferable to recompiling a Generic IOC
     for every possible combination of devices. Using Kubernetes to
     manage multiple small services is cleaner than having a handful of
     monolithic services.
 
-This tutorial will make some changes to the generic IOC ``ioc-adsample``.
-This Generic IOC is a simplified copy of ``ioc-adsimdetector`` tailored for
-use in these tutorials.
 
-For this exercise we will initially work locally inside the ``ioc-adsample``
-developer container.
+This tutorial will make some changes to the generic IOC ``ioc-adsimdetector``
+that you already used in earlier tutorials.
 
-At the end we will push the changes and see the CI build a new version of the
-generic IOC container image. This allows for the demonstration of:
-
-- Deploying an IOC instance using a new image published by the CI
-- Showing how to do a Pull Request back to the original repository.
+For this exercise we will work locally inside the ``ioc-adsimdetector``
+developer container. Following tutorials will show how to fork repositories
+and push changes back to GitHub
 
 For this exercise we will be using an example IOC Instance to test our changes.
 Instead of working with a beamline repository, we will use the example ioc instance
-that comes with ``ioc-adsample``. It is a good idea for Generic IOC authors to
+inside ``ioc-adsimdetector``. It is a good idea for Generic IOC authors to
 include an example IOC Instance in their repository for testing changes in
 isolation.
 
-
 Preparation
 -----------
 
-Because we want to push our changes we will first make a fork of the
-``ioc-adsample`` repository. We will then clone our fork locally and
-make the changes there.
-
-To make a fork go to
-`ioc-adsample <https://github.com/epics-containers/ioc-adsample>`_
-and click the ``Fork`` button in the top right corner. This will create a fork
-of the repository under your own GitHub account.
-
-Now, clone the fork, build the container image locally and open the
-developer container:
+First, clone the ``ioc-adsimdetector`` repository and make sure the container
+build is working:
 
 .. code-block:: console
 
-    git clone [email protected]:<YOUR GITHUB ACCOUNT NAME>/ioc-adsample.git
-    cd ioc-adsample
+    git clone [email protected]:epics-containers/ioc-adsimdetector.git
+    cd ioc-adsimdetector
     ./build
     code .
-    # click the green button in the bottom left corner of vscode and select
-    # "Reopen in Container"
+    # Choose "Reopen in Container"
+
+Note that if you do not see the prompt to reopen in container, you can open
+the ``Remote`` menu with ``Ctrl+Alt+O`` and select ``Reopen in Container``.
 
 The ``build`` script does two things.
 
 - it fetches the git submodule called ``ibek-support``. This submodule is shared
   between all the EPICS IOC container images and contains the support YAML files
   that tell ``ibek`` how to build support modules inside the container
-  environment.
+  environment and how to use them at runtime.
 - it builds the Generic IOC container image locally.
 
 .. note::
 
     The ``build`` script is a convenience script that is provided in the
     Generic IOC Template project. It is exactly equivalent to cloning
-    with ``--recursive`` flag and then running ``ec dev build``.
+    with ``--recursive`` flag and then running ``ec dev build``. Equally,
+    opening a vscode dev container will also build the container for you, but it
+    does require the ``ibek-support`` submodule to be present - using
+    ``--recursive`` flag to git clone ensures this.
 
 Verify the Example IOC Instance is working
 ------------------------------------------
 
 When a new Generic IOC developer container is opened, there are two things
 that need to be done before you can run an IOC instance inside of it.
 
-- Build the IOC source code
+- Build the IOC binary
 - Select an IOC instance definition to run
 
-The folder ``ioc`` inside of the ``ioc-adsample`` is where the IOC source code
-is created and built. When you open the developer container, this folder does
-not yet exist. The following command will create it and build the IOC:
+The folder ``ioc`` inside of the ``ioc-adsimdetector`` is where the IOC source code
+resided. However our containers always make a symlink to this folder at
+``/epics/ioc``. This is so that it is always in the same place and can easily be
+found by ibek (and the developer!). Therefore you can build the binary with the
+following command:
 
 .. code-block:: console
 
-    ec ioc build
+    cd /epics/ioc
+    make
+
+.. note::
+
+    Note that we are required to build the IOC.
+    This is even though the container you are using already had the IOC
+    source code built by its Dockerfile (``ioc-adsimdetector/Dockerfile``
+    contains the same command).
+
+    For a detailed explanation of why this is the case see `ioc-source`
 
 The IOC instance definition is a YAML file that tells ``ibek`` what the runtime
 assets (ie. EPICS DB and startup script) should look like. Previous tutorials
 selected the IOC instance definition from a beamline repository. In this case
-we will use the example IOC instance that comes with ``ioc-adsample``. The
+we will use the example IOC instance that comes with ``ioc-adsimdetector``. The
 following command will select the example IOC instance:
 
 .. code-block:: console
 
-    ibek dev instance /epics/ioc-adsample/ioc_examples/bl01t-ea-ioc-02
+    ibek dev instance /workspaces/ioc-adsimdetector/ioc_examples/bl01t-ea-ioc-02
 
-In an earlier tutorial when learning about the dev container, we manually
-performed this step, see `choose-ioc-instance`. The above command does
-exactly the same thing: removes the existing config folder in ``/epics/ioc``
+The above command removes the existing config folder ``/epics/ioc/config`` and
 and symlinks in the chosen IOC instance definition's ``config`` folder.
 
 Now  run the IOC:
 
 .. code-block:: console
 
-    ibek dev run
+    cd /epics/ioc
+    ./start.sh
 
-You should see a iocShell prompt and no error messages above.
+You should see an iocShell prompt and no error messages above.
 
-.. note::
+Let us also make sure we can see the simulation images that the IOC is
+producing. For this we need the ``cd2v`` tool that we used earlier. You
+can use the same virtual environment that you created earlier, or create
+a new one and install again. Note that these commands are to be run
+in a terminal outside of the developer container.
 
-    The ``ec ioc build`` command required to re-create the IOC source code.
-    This is even though the container you are using already had the IOC
-    source code built by its Dockerfile (``ioc-adsample/Dockerfile``
-    contains the same command).
+.. code-block:: console
+
+    python3 -m venv cd2v
+    source ~/cd2v/bin/activate
+    pip install c2dataviewer
+
+Run the ``cd2v`` tool and connect it to our IOCs PVA output:
+
+.. code-block:: console
+
+    cd2v --pv BL01T-EA-TST-02:PVA:OUTPUT &
+
+
+Back inside the developer container, you can now start the detector and
+the PVA plugin, by opening a new terminal and running the following:
+
+.. code-block:: console
+
+    caput BL01T-EA-TST-02:PVA:EnableCallbacks 1
+    caput BL01T-EA-TST-02:CAM:Acquire 1
+
+You should see the moving image in the ``cd2v`` window. We now have a working
+IOC instance that we can use to test our changes.
+
+Making a change to the Generic IOC
+----------------------------------
+
+One interesting way of changing a Generic IOC is to modify the support YAML
+for one of the support modules. The support YAML describes the ``entities`` that
+an IOC instance can make use of in its instance YAML file. This will be
+covered in much more detail in `generic_ioc`.
+
+For this exercise we will make a change to the ``ioc-adsimdetector`` support
+YAML file. We will change the startup script that it generates so that the
+simulation detector is automatically started when the IOC starts.
+
+To make this change we just need to have the startup script set the values
+of the records ``BL01T-EA-TST-02:CAM:Acquire`` and
+``BL01T-EA-TST-02:PVA:EnableCallbacks`` to 1.
+
+To make this change, open the file
+``ibek-support/ADSimDetector/ADSimDetector.ibek.support.yaml``
+and add a ``post_init`` section just after the ``pre_init`` section:
+
+.. code-block:: yaml
+
+    post_init:
+      - type: text
+        value: |
+          dbpf {{P}}{{R}}Acquire 1
+
+Next make a change to the file ``ibek-support/ADCore/ADCore.ibek.support.yaml``.
+Find the NDPvaPlugin section and also add a ``post_init`` section:
+
+.. code-block:: yaml
+
+    post_init:
+      - type: text
+        value: |
+          dbpf {{P}}{{R}}EnableCallbacks 1
+
+
+If you now go to the terminal where you ran your IOC, you can stop it with
+``Ctrl+C`` and then start it again with ``./start.sh``. You should see the
+following output at the end of the startup log:
+
+.. code-block:: console
+
+    dbpf BL01T-EA-TST-02:CAM:Acquire 1
+    DBF_STRING:         "Acquire"
+    dbpf BL01T-EA-TST-02:PVA:EnableCallbacks 1
+    DBF_STRING:         "Enable"
+    epics>
+
+You should also see the ``cd2v`` window update with the moving image again.
+
+If you wanted to publish these changes you would have to commit both the
+``ibek-support`` submodule and the ``ioc-adsimdetector`` repository and push
+them in that order because of the sub-module dependency. But we won't be
+pushing these changes as they are just for demonstration purposes. In later
+tutorials we will cover making forks and doing pull requests for when you have
+changes to share back with the community.
+
+Note: this is a slightly artificial example, as it would change the behaviour
+for all instances of a PVA plugin and a simDetector. In a real IOC you would
+do this on a per instance basis.
+
+Let us quickly do the instance YAML change to demonstrate the correct approach
+to this auto-starting detector.
+
+Undo the support yaml changes:
+
+.. code-block:: console
+
+    cd /workspaces/ioc-adsimdetector/ibek-support
+    git reset --hard
+
+Add the following to
+``/workspaces/ioc-adsimdetector/ioc_examples/bl01t-ea-ioc-02/config/ioc.yaml``:
+
+.. code-block:: yaml
+
+  - type: epics.dbpf
+    pv: BL01T-EA-TST-02:CAM:Acquire
+    value: "1"
 
-    For a detailed explanation of why this is the case see
-    `ioc-source`
+  - type: epics.dbpf
+    pv: BL01T-EA-TST-02:PVA:EnableCallbacks
+    value: "1"
 
+Now restart the IOC and you should see the same behaviour as before. Here
+we have made the change on a per instance basis, and used the ``dbpf`` entity
+declared globally in ``ibek-support/_global/epics.ibek.support.yaml``.
 
-TODO: complete by adding iocStats and using it in the ioc instance, then
-pushing and verifying CI runs and publishes a new image.
-TODO: now that cacheing is working, consider using ioc-adsimdetector instead
-of ioc-adsample. This is simpler - the change could be the addition of
-auto start of the sim detector IOC just like the presentation.