[Feature] OpenWRT with APK support

[pedorich-n]

OpenWRT is currently migrating from OPKG to APK. As of 2024-11-28 the snapshot build already uses APK and 24.10-SNAPSHOT still uses OPKG. Although there are some other changes to the directory structure in both of these branches.
I want to use this issue to track what needs to be done to make nix-openwrt-imagebuilder support these changes and the newer version of OpenWRT.

Changes present in both branches

• kmods are not included in packages anymore
  • They used to be included in https://downloads.openwrt.org/releases/<release>/targets/<target>/<variant>/packages, for example https://downloads.openwrt.org/releases/23.05.5/targets/mediatek/filogic/packages/Packages, look for Package: kmod-.
  • Now they're in https://downloads.openwrt.org/releases/<release>/targets/<target>/<variant>/kmods/<kmod-target>. For example: https://downloads.openwrt.org/releases/24.10-SNAPSHOT/targets/mediatek/filogic/kmods/6.6.58-1-8aeafca30161030b0854d761688a400e/Packages
    • The <kmod-target> as I call it, consists of the kernel version, kernel release, and hash of some kind.
    • Thanks to build: add kernel version information to index.json openwrt/openwrt#17042 these fields are now present in profiles.json. For example: https://downloads.openwrt.org/releases/24.10-SNAPSHOT/targets/mediatek/filogic/profiles.json, field linux_kernel.
  • So all we need to do is just "inject" kmods as another "feed" in generate-hashes.nix and files.nix and use a different URL to download kmod-* packages

Changes present only in SNAPSHOT (as of 2024-11-28)

• No more Packages file in APK builds
  • It's used to build a static dependency tree for all packages that need to be included in the build
  • Thankfully there's packages.adb file. For example: https://downloads.openwrt.org/snapshots/targets/mediatek/filogic/packages/packages.adb This file is APK's "apkv3 repository index file" created using apk mkndx command
  • There's also a apk adbdump command which allows to dump this DB into a "yaml-like" format. This plaintext file contains the depends field which we can use to replace Depends from the Packages file
    • But there are issues with quoting currently, so the file is not always valid YAML. Meaning it can't be convered to Nix easily. The issue has been open for 3+ years and it seems like it's not a priority for the developers. So we might need to resort to plain-text parsing 😞
      • OpenWRT currently uses awk & sed to generate index.json from this yaml-like code.
    • Another problem is that these commands are only available in unstable v3 preview builds of APK: https://gitlab.alpinelinux.org/alpine/apk-tools/-/releases/v3.0.0_pre4. So we need to wait for a stable apk release and ideally for quoting issues to be resolved.
• APK requires adb file instead of just src custom file:///path/to/packages to consider something a repository
  • We can use apk mkndx just like like OpenWRT does. Perhaps it makes sense to package all packages into a separate derivation that would build this adb file.

Proposed plan

• Wait for APK v3 release
• Wait for OpenWRT 24.10 release or at least wait until snapshot builds are "stable" (optional?)
• Add support for "separate" kmods to generate-hashes.nix and files.nix
• Add support for parsing packages.adb and building custom static.adb containing only required packages

Again, this is just a proposal and a place to track my findings.
Let me know what you think. 🙇

[pedorich-n]

Update about APK's adbdump output: it might support JSON output "soon". Which great news for Nix, as we could just use builtins.fromJSON instead of using yq + IFD 🚀

[pedorich-n]

An update about adbdump: JSON support was added! This is not in any of the releases yet, but I built the package locally and tested it.
The output JSON file is valid and it'll help simplify the code here.

Sample of the JSON

{
  "packages": [
    {
      "name": "arptables-nft",
      "version": "1.8.10-r1",
      "hashes": "17a6d609cb8223cf8d9695b1ceca2c922e37e08936851c2feec42c3b59bb0c3d",
      "description": "ARP firewall administration tool nft",
      "arch": "aarch64_cortex-a53",
      "license": "GPL-2.0",
      "origin": "feeds/base/network/utils/iptables",
      "url": "https://netfilter.org/",
      "installed-size": 77824,
      "file-size": 3407,
      "provider-priority": 1,
      "depends": [
        "kmod-arptables",
        "kmod-nft-arp",
        "libc",
        "xtables-nft"
      ],
      "provides": [
        "arptables"
      ]
    },
    ....
  ]
}