Skip to content

Development

Jump to bottom
Andrew Beltrano edited this page Jun 30, 2024 · 8 revisions

NetRemote Development

The project is structured to allow primary development on multiple platforms/OSes. Hence, CMake is used as the build system generator. The source tree is organized into OS-dependent and OS-independent sub-trees. See the project structure and composition section below for complete details.

Project dependencies are managed by vcpkg using manifest mode and listed in the project vcpkg manifest file vcpkg.json. All dependencies must be resolved using vcpkg such that the project can be (easily) used in the Windows build system.

Project Structure and Composition

To support maximal testability and code re-use across all supported platforms, each major software component is divided into two (2) main parts, the core, os independent/common part, and the detail, os dependent/specific part. The core part includes the interface, logic, and flow-control and must not be aware of any operating system or platform specific dependencies. The detail part includes the implementation details which are necessarily aware of and make use of operating system or platform specific dependencies.

From a code composition point of view, OS-independent parts are collected into static libraries. OS-dependent parts are collected into final link-targets such as binaries (.exe) and shared object libraries (.so, .dll) and link to the OS-independent static libraries. In this way, code duplication is kept to a bare minimum. To achive this, all OS-independent code should be placed in the src/common project tree, and all OS-dependent code should be placed in a separate project tree under src with the corresponding OS name, for example, src/linux and src/windows.

The below diagram shows an example of the netremote-server component composition:

NetRemote Component Composition

A CMake target naming convention for major components is used where the OS-independent static library is named with the component name, and the corresponding OS-dependent targets are named with the component name suffixed with the OS name, separated by a dash. Eg.

CMake OS-indepdendent static library target name = <component-name>
CMake OS-dependent binary/shared library target name = <component-name>-<os-name>

For example, for the component that produces the gRPC server executable, the following CMake targets are defined:

Target Name Target Type Source Location
netremote-server static library src/common/server
netremote-server-linux executable src/common/linux
netremote-server-windows executable src/windows/server

CMake's OUTPUT_NAME function is used to remove the OS-suffix from the final produced artifact name. So in the example above, the executable binary produced by the netremote-server-linux CMake target is netremote-server instead of netremote-server-linux.

Project Tree Structure

root

Directory Purpose Contents
.devcontainer Build Infrastructure Development container collateral
.docker Build Infrastructure Docker-based development resources
.github Build Infrastructure GitHub resources (actions, templates, workflows)
cmake Build Infrastructure Custom CMake build scripts
doc Documentation Documentation and resources
packaging Packaging Packaging scripts and tooling (eg. .deb creation)
protocol API, Protocol API definitions and client bindings
src Source Code Source sub-directories
tests Source Code Source code for tests
vcpkg Source Code git sub-module for vcpkg dependency management

Source Code Structure and Composition (src)

Directory Contents
common Source that is common to all platforms
linux Source that is specific to Linux
windows Source that is specific to Windows

common

The components in this tree typically define the majority of the core logic needed to implement the component. Thus they are commonly built as static library targets via CMake add_library(<name> STATIC) and then linked into their corresponding component in the OS-dependedent trees.

For example, the server component here defines a static library target netremote-server which is then listed as a link library via CMake target_link_libraries of its corresponding Linux component which defines the executable target via CMake add_executable() under src/linux/server.

Directory Contents
client Client/consumer related code.
dotnet Microsoft .NET projects, mostly for non-C++ sanity checks
server Server code that exposes the gRPC API services externally
service Service code that implementes the gRPC API
shared Shared utility code for use throughout the project
tools Project tools like command-line and dev helper programs
wifi Shared Wi-Fi related utility and helper code

OS Dependent Trees (linux, windows)

These source trees closely mirror that of the common tree and typically define the final output artifact for a particular component, and link corresponding common, OS-independent components.

Packaging Structure and Composition (packaging)

Directory Contents
deb Scripts to create Linux debian-style package .deb.
vcpkg Scripts to create netremote vcpkg port

Protocol and API Structure and Composition (protocol)

Directory Contents
protos Protocol buffer definitions for the API (.proto)
include Headers defining common protocol properties such as ip, port, etc.

Coding Guidelines

Where possible, C++ Standard Library primitives should be used for interoperability between common and OS-dependent code. The use of OS-specific primitives and libraries is reserved for scenarios where they are strictly needed (eg. calling an OS/System API), or where the highest possible performance is required and only the OS implementation can provide this.

The coding style is dictated by both .clang-format and .clang-tidy files at the root of the project. Please configure your editor to format and lint sources accordingly. Above all, the coding style should be kept as consistent as possible. The exact style used is not overly important.

To help keep the code consistent, please follow these general guidelines:

  • DO use spaces instead of tabs.
  • DON'T prefix variable names to describe their type, scope, or visibility.
  • DO use std::filesystem for storage and UNIX path separators (/) where possible.
  • DO use std::format for string formatting.
  • DO use complete words for variable and function names.
  • DO use the following table for variable naming:
Block Style Example
Types PascalCase struct AccessPoint {};
Functions PascalCase AccessPoint GetAccessPointById(std::string_view id)
Variables camelCase AccessPoint accessPoint{};
Parameters camelCase void registerEventCallback(AccessPointEventCallback& eventCallback)
Namespaces Uppercase namespace Microsoft::Net::Remote
Public Members PascalCase struct AccessPoint { std::string Id; }
Private Members camelCase with m_ prefix class AccessPoint { uint64_t m_sessionId; }
Directory Names lowercase include/microsoft/net/wifi
File Names PascalCase Ieee80211.hxx
#include Paths lowercase #include <microsoft/net/wifi/Ieee80211.hxx>

Development Environment Setup

Pre-requisities:

Compiler

Both a compiler and standard C++ library supporting C++23 are required. The C++ reference pages for C++23 core language features and C++23 library features provide complete details about current compiler and library support.

CMake

CMake may be installed in any form, as long as the version meets the minimum. One popular way of installing it on Windows is to use Chocolately with choco install -y cmake. On Linux, all standard package managers provide a cmake package (eg. apt-get install -y cmake, yum install -y cmake, etc.).

To bootstrap the build environment, instruct CMake to generate the build files. It is strongly recommended to do this in a directory that is separate from the source; this allows one to easily destroy and recreate the build environment without affecting the checked-out source and changes in progress. Typically, a new directory called build at the top-level project tree is used for this purpose:

git clone [email protected]:microsoft/netremote.git
cd netremote
cmake -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -Bbuild 
cmake --build build

CMake with Visual Studio Code

Alternatively, Microsoft provides a CMake Tools Visual Studio Code extension that automates this process. After installing the extension and cloning the repository in VSCode, hit Ctrl + Shift + P or F1, then find the command CMake: Delete Cache and Reconfigure. This will generate the build configuration in a build folder at the top-level of the source tree. Once done, you can build the ALL target (default) with the CMake: Build command again (Ctrl + Shift + P or F1, type cmake, find the command).

In general, you set a build target and variant, then use the CMake: Build command to build incrementally. All build targets can be found in the CMake: Project Outline activity bar, but a list of them will also be shown when invoking actions that involve targets.

You may need to enable unsupported presets versions. To do this, press File > Preferences then search for 'presets' and enable the option CMake: Allow Unsupported Presets Versions.

Development on Linux

As described above, a C++ 23 compiler and CMake are required. So, any distribution satisfying these requirements may be used. A known working environment for Linux is Ubuntu 24.04 (noble numbat) with a few development packages. Instructions for setting up this environment are provided below.

  1. Install Ubuntu 24.04 (noble)

    If development on Windows is desired, Ubuntu may be installed in WSL using a rootfs image. To install WSL, on newer versions of Windows 11, use the following command: wsl --install --no-distribution. For complete instructions, refer to https://learn.microsoft.com/en-us/windows/wsl/install. Then follow these steps to install Ubuntu 24.04 noble for WSL:

    • Choose a location to store the WSL Ubuntu root filesystem (rootfs) image and switch to this directory. For optimal performance, use a non-system drive and create a top-level folder for all WSL content such as d:\wsl:

      md d:\wsl
cd d:\wsl

    • Download the Ubuntu 24.04 wsl rootfs archive. The development environment has been tested and confirmed to work with the 20240627 image. More recent images may work as well, however, no guarantees are made for such images.

      Invoke-WebRequest -Uri https://cloud-images.ubuntu.com/wsl/noble/20240627/ubuntu-noble-wsl-amd64-24.04lts.rootfs.tar.gz -OutFile .\

    • Import the downloaded rootfs image to create a new WSL distribution for Ubuntu 24.04 noble:

      md d:\wsl\noble
wsl --import noble d:\wsl\noble .\ubuntu-noble-wsl-amd64-24.04lts.rootfs.tar.gz
wsl -d noble

    • You are now running in a root shell on a fresh Ubuntu 24.04 installation. The rootfs image has no suitable non-root user accounts, so create one, adding them to the sudo and root groups to enable privileged execution:

      adduser dev --comment dev
usermod dev -aG root,sudo

    • Create a /etc/wsl.conf file.

      • Add a [boot] section with a systemd=true setting to enable systemd.

      • Add a [user] section with a default=username setting to specify the default logon user to be the one you just created (eg. dev).

      [boot]
systemd=true

[user]
default=dev

      For example, you can create the file with those contents using echo in execution mode:

      echo -e '[boot]\nsystemd=true\n\n[user]\ndefault=dev' > /etc/wsl.conf

    • Exit Ubuntu 24.04 noble:

      exit

    • Shutdown WSL so it can pick up the new default user changes, and re-run Ubuntu 24.04 noble:

      wsl --shutdown
wsl -d noble

    You should now be logged on as the dev user.

  2. Install Linux Development Dependencies

    Execute the following commands in a shell to install all the tools required to compile, lint, and debug the project:

    • Update the package cache and upgrade to latest packages:

      sudo apt-get update -y && sudo apt-get upgrade -y

    • Install core build tools and dependencies:

      sudo apt-get install -y autoconf automake autopoint build-essential ca-certificates cmake curl git gnupg libltdl-dev libmount-dev libtool linux-libc-dev libstdc++-14-dev ninja-build pkg-config python3-jinja2 tar unzip zip

    • Install LLVM 18 toolchain dependencies:

      sudo apt-get install -y libllvm-18-ocaml-dev libllvm18 llvm-18 llvm-18-dev llvm-18-doc llvm-18-examples llvm-18-runtime clang-18 clang-tools-18 clang-18-doc libclang-common-18-dev libclang-18-dev libclang1-18 clang-format-18 python3-clang-18 clangd-18 clang-tidy-18 libclang-rt-18-dev libpolly-18-dev  libfuzzer-18-dev lldb-18 libc++-18-dev libc++abi-18-dev libomp-18-dev libclc-18-dev libunwind-18-dev libmlir-18-dev mlir-18-tools libbolt-18-dev bolt-18 flang-18 libclang-rt-18-dev-wasm32 libclang-rt-18-dev-wasm64 libc++-18-dev-wasm32 libc++abi-18-dev-wasm32 libclang-rt-18-dev-wasm32 libclang-rt-18-dev-wasm64 libllvmlibc-18-dev

    • Install other development dependencies and helpful tools:

      sudo apt install -y --no-install-recommends bc bison dwarves flex libelf-dev dos2unix file gnupg2 iproute2 mtools neofetch rsync ssh sudo gdb kmod nano policycoreutils-python-utils python-is-python3 vim debconf-utils iw

    • Install hostapd and wpa_supplicant development dependencies:

      sudo apt-get install -y libnl-3-200-dbg libnl-3-dev libssl-dev libnl-genl-3-dev libdbus-c++-dev libnl-route-3-dev

    [!NOTE] The source of truth for installing development dependencies is found in the GitHub build-with-host host action `Install Linux build dependencies' step. The instructions there always supercede the instructions here as that action is used to build official release packages.

  3. Install Docker on Windows (optional)

    Download and install docker from https://docs.docker.com/desktop/install/windows-install (amd64). During the installation, ensure the Use WSL 2 instead of Hyper-V (recommended) box is checked.

  4. Configure commit-signing (optional)

Development on Windows

Visual Studio 2022 generally satisfies the requirements, however, the full integrated development environment (IDE) is not needed. A much leaner alternative for those using other editors such as Visual Studio Code can instead install Visual Studio Build Tools.

As described above, a C++ 23 compiler and CMake are required. A known, minimal working environment for Windows is using Visual Studio 2022 Build Tools which provides all of the necessary development requirements. Note that the build tools do not provide the Visual Studio IDE, so if this is desired, the Visual Studio product of choice may be installed instead. Instructions for setting up this environment are provided below:

  1. Install the build tools

    Download the latest version of Visual Studio 2022 Build Tools or the full Visual Studio IDE and run the installer. Select the following 'Workloads':

    • ASP.NET and web development
    • Desktop development with C++
    • .NET desktop development
    • Windows Universal Platform development

    Select the 'Individual Components' tab at the top, and select the following required components:

    • C++ CMake tools for Windows
    • MSVC v143 - VS 2022 C++ x64/x86 build tools (Latest)
    • MSVC v143 - VS 2022 C++ x64/x86 Spectre-mitigated libs (Latest)
    • Windows 11 SDK (10.0.22621.0)
    • .NET 7.0 Runtime

    Additionally if you are using the Visual Studio 2022 Build Tools install the followng under 'Individual Components'

    • C++ Build Tools core features
    • C++ core features

    The following components are optional, but strongly recommended for a more complete development experience:

    • C++ AddressSanitizer
    • C++ Clang Compiler for Windows (15.0.1)
    • C++ Clang-cl for v143 build tools (x64/x86)

    Once installed, the project can be built using CMake as described here.

  2. Configure git commit-signing (optional)

    Prior to following the generic instructions for configuring a git signing key here, the gpg tools must be installed:

    1. Download the latest version of gpg4win from https://www.gpg4win.org/.

    2. Run the installer, deselect all optional components, leaving only GnuPG selected. Accept all other defaults and complete the installation.

    3. Configure git with the location of the Windows gpg tools:

      git config --global gpg.program "C:\Program Files (x86)\GnuPG\bin\gpg.exe"

    4. Follow the common instructions here.

Commit Signing

While not required, it is strongly recommended to configure git with a GNU Privacy Guard (GPG) signing key. This allows GitHub to verify commits were pushed by a specific user and will show a green Verified status beside each verified commit. Follow these steps to configure a signing key for commit verification:

  1. Install the gpg tools for the target operating system (Linux, Windows).
  2. Generate a gpg key
  3. Add the gpg key to your github account
  4. Configure git to use the gpg key. This link will also tell you how to use gpg signing with an ssh key. Additionally, if you wish tell git to use a type any type of signing by default, be it gpg, ssh or X.509, you need to run the following command git config --global commit.gpgsign true

Docker-based Environment

A Docker image is available that provides a known working Linux development environment with all of the tools necessary pre-installed:

  • abeltrano/netremote-dev: A container image for development of the project, including hostapd and building modules for the WSL2 kernel.

To use the Docker container image, install Docker on your development machine, then start an interactive instance, which will bring you to a bash shell:

docker run -it abeltrano/netremote-dev

container shell

This provides the full development environment but does not sync source code. So instead of using the docker image directly, it is recommended to use a development container described in the Dev Container section below.

Dev Container

The docker images are also configured for use with dev containers. Many IDEs have built-in support for development containers, including but not limited to VSCode. This method is the easiest and fastest way to get a working development environment, especially if you're already using VSCode.

VSCode

Use of dev containers in VSCode is the recommended and officially supported development environment. To use this, install Docker on your development machine and install the Dev Containers VSCode extension. The Dev Containers extension has two (2) primary ways to use it:

  1. [Recommended] Cloning directly into a named volume in the container.

    Clone the repository with the dev containers command to use a named volume: press Ctrl + Shift + P or F1 and select the Dev Containers: Clone Repository in Named Container Volume dev container extension command. A named volume is a persistent filesystem that is mounted in the container and used to store the repository source code. This allows the source code to survive across container stops, removals, and deletions. As part of the command, you will be prompted to select the named volume to use. On first run, no volumes will have been created so select Create a new volume and give it any name you want (eg source). In subsequent command invocations to clone the repository, the source volume will exist and you can select it from the list.

    VSCode Clone in Named Container Volume

  2. Cloning locally and sharing source workspace with the container.

    Clone the repository as normal. With this method, VSCode will detect a dev container specification and prompt you to open it in a container:

    VSCode Open In Dev Container Prompt

    Alternatively, you can open the project in the container at any time by invoking the Dev Containers: Open Folder in Container dev container extension. Press Ctrl + Shift + P or F1 and select Reopen in Container.

    A prompt will ask to select a devcontainer.json file:

    VSCode Select a devcontainer.json file

    Select NetRemoteDev-Stateless, which is configured to mount the workspace directory in a bind mount instead of a volume. The NetRemoteDev configuration will not work.

Warning

A major drawback of this method is that the source tree is mounted in the container using a bind mount, which has extremely poor performance to the point where git operations are essentially unusable. Hence, this method typically requires two (2) VSCode windows: one for the repository on the host where git operations are carried out, and one for the container where the source is modified and built. Therefore, this method is strongly discouraged.

Clone this wiki locally