Updates for 24.02

- Test Framework removed / needs update first
- Added Hello World csolution for simple integration test
- Signal library added to pack
- devcontainer using Arm-MLOps docker environment
- tasks for VSC

.devcontainer/Dockerfile

@@ -6,8 +6,8 @@ WORKDIR /tensorflow-pack
 
 RUN apt-get update -y && apt-get upgrade -y
 
-RUN apt-get install rsync wget git zip curl -y --fix-missing
+RUN apt-get install rsync wget git zip curl ninja-build -y --fix-missing
 
 RUN pip3 install pillow requests six semantic_version pyyml
 
-#RUN su -l arm_mlops_docker -c "cpackget index --force https://www.keil.com/pack/index.pidx"
+

tensorflow-build/README.md

@@ -1,20 +1,42 @@
+
 # tensorflow-build
 
 Directory contains scripts and assets to compile the CMSIS-pack containing the latest TensorFlow Lite Micro. 
 
-## build_r.sh
-Entry point to run the full workflow to download, sort, generate source trees and build the pack.
 
-### The workflow is:
+## build_r.sh (build_d.sh for development releases)
+The build_r.sh script serves as the entry point for running a full workflow that involves downloading, sorting, generating source trees, and building a pack. Steps involved in this workflow:
+
+1. **Get the latest root repository**: The script starts by retrieving the latest root repository from mlplatform.org. This repository contains a list of JSON files that index release SHAs for a specific release. This step ensures that the script is working with the most up-to-date information.
+
+2. **Retrieve repositories for a specified release**: The script then proceeds to retrieve the repositories associated with a specified release. The release version is passed to the script as a parameter. For example, if the release version is "20.02", the script will fetch the repositories related to that specific release.
+
+3. **Create source trees**: The create_tflm_tree.py script is used to generate source trees for different variants, namely "Reference", "CMSIS-NN", and "Ethos-U". These source trees serve as the foundation for building the pack.
+
+4. **Write source file lists**: For each source tree generated in the previous step, the script writes a list of files used in that particular source tree into a text file with a name like srcs.*.lst. These lists of source files provide a comprehensive overview of the files included in each source tree.
+
+5. **Merge determined sources**: The script then merges the determined sources into a build directory. This step brings together the necessary files from the different source trees to create a unified set of sources for the pack.
+
+6. **Apply patches**: The script applies a series of patches to the source files. These patches are necessary to ensure compatibility and correct functionality of the TensorFlow Lite Micro code within the CMSIS-Pack environment. The patches are applied using the 'patch' command, and the specific patches to be applied are determined by the script based on the release version and the specific source tree variant.
+
+7. **Run generate_cmsis_pack.py**: Finally, the script executes the generate_cmsis_pack.py script and passes the lists of sources as arguments. This script is responsible for generating the CMSIS pack, which is a collection of software components and device support files used in embedded systems development.
+
+## generate_cmsis_pack.py
+This script is responsible for generating the CMSIS pack. It takes several parameters, including paths to source files, headers, and the release version. Here are the steps involved in this script:
+
+1. **Read Parameters**: The script starts by reading the parameters passed to it. These parameters include paths to source files, headers, and the release version.
+
+2. **Pull PackChk**: The script then pulls PackChk for the platform the script is executed on. PackChk is a tool used to validate the CMSIS pack.
+
+3. **Generate PDSC File**: Using the input parameters, the script generates a PDSC (Pack Description) file. This file describes the pack contents, including software components, device support files, and examples.
 
-1. Get the latest root repository from mlplatform.org. It contains a list of json files with an index of release SHAs for a specific release.
-2. Retrieve the repositories for a specified release (e.g. "20.02"). This release is passed to the github workflow as a parameter.
-3. create_tflm_tree.py is used to create the source trees for variants "Reference", "CMSIS-NN", "Ethos-U".
-4. For each source tree a list of files used it written into a srcs.*.lst text file.
-5. Merge determined sources into build directory.
-6. Run generate_cmsis_pack.py and pass the lists of sources.
+4. **Validate PDSC File**: The generated PDSC file is then validated using PackChk. If the validation fails, the script will stop execution.
 
-## generate_cmsis_pack.py 
+5. **Generate CMSIS Pack**: If the PDSC file passes validation, the script proceeds to generate the CMSIS pack. This pack is a collection of software components and device support files used in embedded systems development.
+
+6. **Output**: The output of this script is a tensorflow-lite-micro.*.pack file, which is only generated when the PDSC validation passes.
+
+## generate_cmsis_pack.py Usage
 ```
 Parameters are:
   --input_template        Path to template file pdsc.
@@ -33,4 +55,39 @@ Parameters are:
 ```
 
 Internally generate_cmsis_pack.py will pull PackChk for the platform the script is executed on. 
-The generated pdsc will be validated and the tensorflow-lite-micro.*.pack will only be generated when validation passes.
+The generated pdsc will be validated and the tensorflow-lite-micro.*.pack will only be generated when validation passes.
+
+## Additional Scripts in the Repository
+
+### get_releases.py
+
+"""
+This script retrieves the latest releases from a GitHub repository.
+
+Usage:
+  python get_releases.py [repository]
+
+Arguments:
+  repository (str): The name of the GitHub repository to retrieve releases from.
+
+Returns:
+  list: A list of dictionaries, where each dictionary represents a release and contains the following keys:
+    - name (str): The name of the release.
+    - tag_name (str): The tag name of the release.
+    - published_at (str): The date and time when the release was published.
+    - assets (list): A list of dictionaries representing the assets associated with the release, where each dictionary contains the following keys:
+      - name (str): The name of the asset.
+      - download_url (str): The URL to download the asset.
+
+Example:
+  python get_releases.py myrepo
+"""
+
+### clean_file_list.py
+"""
+This script provides functions to clean up a list of file names.
+
+The `clean_file_list` module contains functions that can be used to remove unwanted characters and formats from a list of file names. It provides a simple and efficient way to sanitize file names before further processing.
+
+Usage:
+   python clean_file_list.py

tensorflow-build/build_d.sh

@@ -1,6 +1,7 @@
 #!/bin/sh
 
-echo Building from latest main branch - nightly build $1
+
+echo Building from latest main branch - nightly build" $1
 
 mkdir ./tensorflow-pack/tensorflow-build/rel
 mkdir ./tensorflow-pack/tensorflow-build/rel/mlplatform
@@ -12,8 +13,11 @@ wget -O ./tensorflow-pack/tensorflow-build/rel/master.tar.gz https://review.mlpl
 tar -xzf ./tensorflow-pack/tensorflow-build/rel/master.tar.gz -C ./tensorflow-pack/tensorflow-build/rel/mlplatform
 # Get ml-platforms srcs
 cd ./tensorflow-pack/tensorflow-build/rel/mlplatform/
+
+echo Fetching ml-platforms sources
 python3 fetch_externals.py fetch
 
+
 cd ./core_software/tflite_micro
 
 python3 ./tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py  \
@@ -23,6 +27,7 @@ python3 ./tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py  \
    > ../../../../../tensorflow-build/srcs.raw
 rsync -a ../../../../../tensorflow-build/src/tensorflow ../../../../../tensorflow-build/gen/build
 
+
 python3 ./tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py  \
    --makefile_options="TARGET=cortex_m_generic OPTIMIZED_KERNEL_DIR=cmsis_nn TARGET_ARCH=cortex-m55" \
    --print_src_files --rename_cc_to_cpp\
@@ -50,6 +55,7 @@ python3 ./tensorflow-pack/tensorflow-build/clean_file_list.py \
 python3 ./tensorflow-pack/tensorflow-build/clean_file_list.py \
         ./tensorflow-pack/tensorflow-build/srcs.ethos_u.raw > ./tensorflow-pack/tensorflow-build/srcs.ethos_u.lst
 
+
 python3 ./tensorflow-pack/tensorflow-build/generate_cmsis_pack.py  \
    --release=1.99 \
    --candidate_rev=$1 \

tensorflow-build/build_r.sh

@@ -1,19 +1,45 @@
 #!/bin/sh
 
-echo $1 $2
-
+echo "\033[0;32m"
+echo "Building from mlplatforms release: " $1
+echo "Release tag: " $2
+
+# Check if working directories exist and delete if so. Then create new.
+if [ -d "./tensorflow-pack/tensorflow-build/rel" ]; then
+   rm -rf ./tensorflow-pack/tensorflow-build/rel
+fi
 mkdir ./tensorflow-pack/tensorflow-build/rel
+
+if [ -d "./tensorflow-pack/tensorflow-build/rel/mlplatform" ]; then
+   rm -rf ./tensorflow-pack/tensorflow-build/rel/mlplatform
+fi
 mkdir ./tensorflow-pack/tensorflow-build/rel/mlplatform
+
+if [ -d "./tensorflow-pack/tensorflow-build/gen" ]; then
+   rm -rf ./tensorflow-pack/tensorflow-build/gen
+fi
 mkdir ./tensorflow-pack/tensorflow-build/gen
+
+if [ -d "./tensorflow-pack/tensorflow-build/gen/build" ]; then
+   rm -rf ./tensorflow-pack/tensorflow-build/gen/build
+fi
 mkdir ./tensorflow-pack/tensorflow-build/gen/build
+
 # Get ml-platforms root
 wget -O ./tensorflow-pack/tensorflow-build/rel/master.tar.gz https://review.mlplatform.org/plugins/gitiles/ml/ethos-u/ethos-u/+archive/refs/heads/master.tar.gz
+
+
 # Extract tar.gz
 tar -xzf ./tensorflow-pack/tensorflow-build/rel/master.tar.gz -C ./tensorflow-pack/tensorflow-build/rel/mlplatform
 # Get ml-platforms srcs
 cd ./tensorflow-pack/tensorflow-build/rel/mlplatform/
+
+echo "\033[1;33m"
+
 python3 fetch_externals.py -c $1.json fetch
 
+echo "\033[0;32m"
+
 cd ./core_software/tflite_micro
 
 python3 ./tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py  \
@@ -22,20 +48,23 @@ python3 ./tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py  \
    ../../../../../tensorflow-build/src \
    > ../../../../../tensorflow-build/srcs.raw
 rsync -a ../../../../../tensorflow-build/src/tensorflow ../../../../../tensorflow-build/gen/build
+rsync -a ../../../../../tensorflow-build/src/signal ../../../../../tensorflow-build/gen/build
 
 python3 ./tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py  \
    --makefile_options="TARGET=cortex_m_generic OPTIMIZED_KERNEL_DIR=cmsis_nn TARGET_ARCH=cortex-m55" \
    --print_src_files --rename_cc_to_cpp\
    ../../../../../tensorflow-build/src \
    > ../../../../../tensorflow-build/srcs.cmsis_nn.raw
 rsync -a ../../../../../tensorflow-build/src/tensorflow ../../../../../tensorflow-build/gen/build
+rsync -a ../../../../../tensorflow-build/src/signal ../../../../../tensorflow-build/gen/build
 
 python3 ./tensorflow/lite/micro/tools/project_generation/create_tflm_tree.py  \
    --makefile_options="TARGET=cortex_m_generic OPTIMIZED_KERNEL_DIR=cmsis_nn CO_PROCESSOR=ethos_u ETHOSU_ARCH=u55 TARGET_ARCH=cortex-m55" \
    --print_src_files \
-   ../../../../../tensorflow-build/src --rename_cc_to_cpp\
+   ../../../../../tensorflow-build/src --rename_cc_to_cpp \
    > ../../../../../tensorflow-build/srcs.ethos_u.raw
 rsync -a ../../../../../tensorflow-build/src/tensorflow ../../../../../tensorflow-build/gen/build
+rsync -a ../../../../../tensorflow-build/src/signal ../../../../../tensorflow-build/gen/build
 
 rsync -a ./tensorflow/lite/micro/testing/*.h ../../../../../tensorflow-build/gen/build/tensorflow/lite/micro/testing/ 
 
@@ -50,6 +79,16 @@ python3 ./tensorflow-pack/tensorflow-build/clean_file_list.py \
 python3 ./tensorflow-pack/tensorflow-build/clean_file_list.py \
         ./tensorflow-pack/tensorflow-build/srcs.ethos_u.raw > ./tensorflow-pack/tensorflow-build/srcs.ethos_u.lst
 
+echo "\033[0;33m"
+
+# If a folder ./patches/$1 exists, call the patch.sh in this folder
+if [ -d "./tensorflow-pack/tensorflow-build/patches/$1" ]; then
+  echo "Patching"
+  ./tensorflow-pack/tensorflow-build/patches/$1/patch.sh
+fi
+
+echo "\033[1;34m"
+
 python3 ./tensorflow-pack/tensorflow-build/generate_cmsis_pack.py  \
    --release=1.$1 \
    --candidate_rev=$2 \
@@ -64,7 +103,7 @@ python3 ./tensorflow-pack/tensorflow-build/generate_cmsis_pack.py  \
    --srcs-ethos=./tensorflow-pack/tensorflow-build/srcs.ethos_u.lst \
    --testhdrs=./tensorflow-pack/tensorflow-build/empty.lst \
    --testsrcs=./tensorflow-pack/tensorflow-build/testsrcs.lst \
-   --util_src=./tensorflow-pack/tensorflow-build/empty.lst  
+   --util_src=./tensorflow-pack/tensorflow-build/kernelutil.lst  
 
 
 

tensorflow-build/clean_file_list.py

@@ -8,7 +8,18 @@
 txt_file.close()
 
 no_dupes_list = list(dict.fromkeys(content_list))
+signal_files_list = []
 
 for filename in no_dupes_list:
-   if not filename.startswith("signal") and if not "downloads" in filename and not "LICENSE" in filename:
+   if filename.startswith("signal/"):
+      signal_files_list.append(filename)
+      no_dupes_list.remove(filename)
+   elif not "downloads" in filename and not "LICENSE" in filename:
       print(filename)
+
+
+signal_filename = in_filename.replace("srcs.", "signal.").replace(".raw", ".lst")
+with open(signal_filename, 'w') as f:
+   for item in signal_files_list:
+      f.write("%s\n" % item)
+

tensorflow-build/generate_cmsis_pack.py

@@ -156,7 +156,6 @@ def make_component_file_list(srcs_list):
                 and src != "tensorflow/lite/c/common.c" \
                 and src != "tensorflow/lite/micro/cortex_m_generic/micro_time.cpp" \
                 and src != "tensorflow/lite/micro/cortex_m_generic/debug_log.cpp" \
-                and src != "tensorflow/lite/micro/recording_micro_allocator.cpp" \
                 and src.endswith("_test.cc") == False:
             replace_srcs += '        <file category=\"' + \
                 category + '\" name=\"' + src + '\"/> \n'
@@ -236,6 +235,14 @@ def main(unparsed_args, flags):
     all_test_srcs_list = test_srcs_list + test_hdrs_list
     all_test_srcs_list.sort()
 
+    # get signal library if exists ("signal.lst" alongside flags.srcs)
+    replace_signal_srcs = ""
+    if os.path.exists(flags.srcs.replace("srcs.", "signal.")):
+      signal_srcs_list = load_list_from_file(flags.srcs.replace("srcs.", "signal."))
+    replace_signal_srcs = make_component_file_list(signal_srcs_list)
+
+    print(replace_signal_srcs)
+
     replace_core_srcs = make_component_file_list(all_core_srcs_list)
     replace_util_srcs = make_component_file_list(all_util_srcs_list)
     replace_test_srcs = make_component_file_list(all_test_srcs_list)
@@ -280,6 +287,8 @@ def main(unparsed_args, flags):
         r'%{KERNEL_UTIL_FILES}%', replace_util_srcs, six.ensure_str(template_file_text))
     template_file_text = re.sub(
         r'%{TEST_FILES}%', replace_test_srcs, six.ensure_str(template_file_text))
+    template_file_text = re.sub(
+        r'%{SIGNAL_FILES_REFERENCE}%', replace_signal_srcs, six.ensure_str(template_file_text))
     template_file_text = re.sub(
         r'%{RELEASE_VERSION}%', pack_version, template_file_text)
     template_file_text = re.sub(

tensorflow-build/template/cmsis_pdsc.tpl

@@ -95,7 +95,7 @@
     </component>
      <!-- this component can be merged into the Kernel components, when duplicate module name issue is solved elsewhere -->
     <component Cclass="Machine Learning" Cgroup="TensorFlow" Csub="Kernel Utils" Cversion="%{RELEASE_VERSION}%">
-      <description>TensorFlow Lite Micro Library</description>
+      <description>TensorFlow Lite Micro Library - Kernel Utilities</description>
       <files>
         <file category="sourceCpp" name="tensorflow/lite/kernels/kernel_util.cpp"/> 
         <file category="sourceCpp" name="tensorflow/lite/micro/system_setup.cpp" version="%{RELEASE_VERSION}%" attr="config"/>
@@ -105,7 +105,7 @@
       </files>
     </component>
     <component Cclass="Machine Learning" Cgroup="TensorFlow" Csub="Testing" Cversion="%{RELEASE_VERSION}%">
-      <description>TensorFlow Lite Micro Library</description>
+      <description>TensorFlow Lite Micro Test Utilities</description>
       <RTE_Components_h>
         <!-- the following content goes into file 'RTE_Components.h' -->
         #define RTE_ML_TF_LITE     /* TF */
@@ -115,6 +115,16 @@
         <file category="include" name="./"/>
       </files>
     </component>
-
+    <component Cclass="Machine Learning" Cgroup="TensorFlow" Csub="Signal Library" Cvariant="Reference" Cversion="%{RELEASE_VERSION}%">
+      <description>TensorFlow Lite Micro Signal Library</description>
+      <RTE_Components_h>
+        <!-- the following content goes into file 'RTE_Components.h' -->
+        #define RTE_ML_TF_LITE     /* TF */
+      </RTE_Components_h>
+      <files>
+        %{SIGNAL_FILES_REFERENCE}%
+        <file category="include" name="./"/>
+      </files>
+    </component>
   </components>
 </package>