apps: Parse the docker store to obtain layer size and disk usage

[mike-sul]

• Parse the docker store to obtain a size and a disk usage for each App layer (unarchived).
• Store the obtained layer metadata as a CI run artifact. Then, it can be used by the CI publish run to add the layer metadata (disk usage & size) into App manifest.
• The embedded layer metadata can be used by the aklite/update client to determine whether an App update can fit into a device disk.

[mike-sul]

Each build run generates layers_meta.json with the following content

{
  "fs_block_size": 4096,
  "layers": {
    "sha256:48ecbb6b270eb481cb6df2a5b0332de294ec729e1968e92d725f1329637ce01b": {
      "size": 4206620,
      "usage": 4781861
    },
   ...
  }
}

The publish download layers_meta.json of each arch/build, merges them into a single dict, dumps the dict into a file and pass it to the compose publish tool.
The compose publish tool extracts metadata of all App layers for each arch, upload a blob with the App layers meta to Registry, and adds the App layers metadata reference to the App manifest.

[mike-sul]

App manifest with references to the two layers, the App bundle/blob and the App layers metadata.

skopeo inspect --raw docker://hub.foundries.io/msul-dev01/app-04@sha256:927e670305d2984088c45ec0837b032a466a2250d21de6e482fdfd8f5ba1c65b
{
   "annotations": {
      "compose-app": "v1"
   },
   "config": {
      "digest": "sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
      "mediaType": "application/vnd.oci.image.config.v1+json"
   },
   "layers": [
      {
         "digest": "sha256:4ec01af7d04fa694b03b3f96c665cfbc6ddc14bf9293cac12144e44ad8eaf56d",
         "mediaType": "application/octet-stream",
         "size": 456
      },
      {
         "annotations": {
            "layers-meta": "v1"
         },
         "digest": "sha256:f8e0b960f955dc633b78a2b8ba79ae4dad94d397cbfa0222bd1b9188766e49af",
         "mediaType": "application/octet-stream",
         "size": 3344
      }
   ],
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.index.v1+json",
         "size": 1465,
         "digest": "sha256:84a2b686ac33f5dce1c15ee575fae14d89bdca2207eaec3b06fad6fdeba56632",
         "platform": {
            "architecture": "amd64",
            "os": ""
         }
      },
      {
         "mediaType": "application/vnd.oci.image.index.v1+json",
         "size": 1623,
         "digest": "sha256:272d1c0ba9cb31e4fcc916e4530cf0c016b8e00508fd0f121ed4f5edf0231eef",
         "platform": {
            "architecture": "arm",
            "os": ""
         }
      },
      {
         "mediaType": "application/vnd.oci.image.index.v1+json",
         "size": 1625,
         "digest": "sha256:4f81d5330f22fa1f2dfd415b9b3b4b5230d95f357b014079362730a293766b0c",
         "platform": {
            "architecture": "arm64",
            "os": ""
         }
      }
   ],
   "mediaType": "application/vnd.oci.image.manifest.v1+json",
   "schemaVersion": 2
}

[mike-sul]

This is the blob/json posted to Registry and referenced from the App manifest. aklite gets it and does its app update size calculation based on it.

{
  "amd64": {
    "fs_block_size": 4096,
    "layers": {
      "sha256:3c7be240f7bfb19ec575d8547832a9f20b95eec9b4cc94fe717dd047ad661159": {
        "size": 32,
        "usage": 33243,
        "archive_size": 230
      },
      "sha256:48ecbb6b270eb481cb6df2a5b0332de294ec729e1968e92d725f1329637ce01b": {
        "size": 4206620,
        "usage": 4781861,
        "archive_size": 2107173
      },
      "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": {
        "size": 0,
        "usage": 20562,
        "archive_size": 32
      },
      "sha256:4fc258d8a87d2d03d0274969d18131d18b4e11755ada73d6a7784129371ef717": {
        "size": 0,
        "usage": 28928,
        "archive_size": 120
      },
      "sha256:6439819450d10d1aae92561f3ffff722137aada46d509644e8de4ca82bb26b07": {
        "size": 70486558,
        "usage": 78985591,
        "archive_size": 25904160
      },
      "sha256:692f29ee68fa6bab04aa6a1c6d8db0ad44e287e5ff5c7e1d5794c3aabc55884d": {
        "size": 555812,
        "usage": 1621362,
        "archive_size": 308482
      },
      "sha256:8f7d2488f433fa6b71ddf27eb3d441a27e349b1830260c676706e4354d6640fd": {
        "size": 4,
        "usage": 41279,
        "archive_size": 156
      },
      "sha256:ca4b349df8ed83a59776df8f3868ece2783aa1ee2e9f052c9c9f3b54ae51a593": {
        "size": 6035089,
        "usage": 7755003,
        "archive_size": 1811993
      }
    }
  },
  "arm": {
    "fs_block_size": 4096,
    "layers": {
      "sha256:4271bf13258b585def40291fcb3b0490ce089b76fe80a5f1aa672e925ec81bcb": {
        "size": 127,
        "usage": 28962,
        "archive_size": 175
      },
      "sha256:479b55051d8d4fb9d10c3958275f30e41e80fda3a591153b8ca051ca6767a9b6": {
        "size": 6030081,
        "usage": 7751409,
        "archive_size": 1811163
      },
      "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": {
        "size": 0,
        "usage": 20562,
        "archive_size": 32
      },
      "sha256:71fb506fd73eb15932a6b3c1549a7339eab8b6d5fe4e93e81ab8d391263835e7": {
        "size": 0,
        "usage": 28929,
        "archive_size": 120
      },
      "sha256:786f62ef66aef5b82d01dc251c095789af4cc746f6279dcc8011023d019b70fb": {
        "size": 3832456,
        "usage": 4409248,
        "archive_size": 2050505
      },
      "sha256:ad61c271c57f705cc5795972f5ba6eb39b7e7ac3a14611f6c61863739ca8b450": {
        "size": 554583,
        "usage": 1621537,
        "archive_size": 309260
      },
      "sha256:d2154f8b624d1e3ee45dd1e3bffc688d8d370e41ffc1058bf29d5a10ae95469b": {
        "size": 61349551,
        "usage": 69884925,
        "archive_size": 24278773
      },
      "sha256:d6968846807f3e3d11b9e15bcef09607398091a85f102f47c1f254bc98617915": {
        "size": 32,
        "usage": 33242,
        "archive_size": 230
      },
      "sha256:db846e3b51640cfe430fa1d39248decbf8c71154d11dc7a599c2a952707dc789": {
        "size": 4,
        "usage": 41278,
        "archive_size": 156
      }
    }
  },
  "arm64": {
    "fs_block_size": 4096,
    "layers": {
      "sha256:06ade89b14b08b385b7add0a1c420a11d7840018ef39e96da50912076a94b580": {
        "size": 6030081,
        "usage": 7751054,
        "archive_size": 1811031
      },
      "sha256:0db621febe29710a4ae149746af391f701bc3a0eb8705910c1401f516e1a0aa3": {
        "size": 4,
        "usage": 41278,
        "archive_size": 155
      },
      "sha256:0fef6ea0df27793b9f4c29c5532d42d82a22db9b46d5420712c33ad3279850c3": {
        "size": 559631,
        "usage": 1625534,
        "archive_size": 308703
      },
      "sha256:2d459117171f80a57721f05c86bb49df9f9b56974652306d63bddba85353bd82": {
        "size": 0,
        "usage": 28929,
        "archive_size": 120
      },
      "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1": {
        "size": 0,
        "usage": 20562,
        "archive_size": 32
      },
      "sha256:836a8c2b8815affbbd8c0c1e8bde3d6a77b336e2171db263b929771dee2ee23d": {
        "size": 32,
        "usage": 33241,
        "archive_size": 231
      },
      "sha256:8bfe4843495e56e3546d4d49cf476ee7bc1650156062151aa38cf3e2fb397a70": {
        "size": 127,
        "usage": 28962,
        "archive_size": 176
      },
      "sha256:c223065212e836d8e54e480f4c8e0a978393f15db88bc4413fd921ba6e9b325e": {
        "size": 78581101,
        "usage": 87096704,
        "archive_size": 25134244
      },
      "sha256:f0599ed267cb750a3c0ebeb55004b5a818b7e9a7e8071b49e687d11349556868": {
        "size": 4010014,
        "usage": 4593408,
        "archive_size": 1999171
      }
    }
  }
}

[mike-sul]

@doanac I think it's ready for review along with the compose publish part foundriesio/compose-publish#13.