Connect new API to the old dashboard

[alexandrasp]

As we work on the new API for Kernel CI, we experiment with using it in the current dashboard. The goals are twofold: test the state of the API to build a Web Dashboard and provide visualization of test runs with the new API in the current legacy dashboard; meanwhile, we don't have a new one.

Below you will see a description of the work and screens.

In the first page:

You can see all job reports sent to the API, and using the pre-existent template, we bring data from the new API and show information about a job, restringing to tree, branch, kernel version, date and status.

Once you select a row, combining kernel version, tree and branch, we will show, on the second page, which test plans we have for that given combined information:

Finally, in the third view, you can see all details of a given test plan combined with kernel version, date and branch.
Example page 3:
Example page 3.1:
Example page 3.2:

We already have some discussion around a new dashboard in progress (see some user stories being discussed ) the idea here is not to bring these views as the solution for this problem but a way to interact and gather information from the new API and experiment with it in a reasonable way to visualize data from new integrations as well (fstests, kunit).

• New Kernel CI API is a work in progress, but we have been missing kernel revision information for over a week. I don't know if it is expected or even if it will be something permanent.

• Another thing, the API is not yet saving the logs in the DB. As we know, having the log information about the build is helpful in case of any problem or regression.

• It was tested locally, as shown in the screenshot above, using an API key and setting the BACKEND_URL in the config file to the new API on staging.

[nuclearcat]

It looks like it is failing on staging (not sure yet):

fatal: [staging.kernelci.org]: FAILED! => {"changed": true, "cmd": ["nodejs", "/srv/staging.kernelci.org/app/dashboard/static/js/lib/r.js", "-o", "/srv/staging.kernelci.org/app/dashboard/static/js/build.js"], "delta": "0:00:00.323137", "end": "2022-12-16 07:03:10.389187", "msg": "non-zero return code", "rc": 1, "start": "2022-12-16 07:03:10.066050", "stderr": "", "stderr_lines": [], "stdout": "Error: Error: ERROR: module path does not exist: kci-new-api-job-branch-kernel-plan.js for module named: kci-new-api-job-branch-kernel-plan.js. Path is relative to: /home/kernelci\n    at /srv/staging.kernelci.org/app/dashboard/static/js/lib/r.js:26773:35", "stdout_lines": ["Error: Error: ERROR: module path does not exist: kci-new-api-job-branch-kernel-plan.js for module named: kci-new-api-job-branch-kernel-plan.js. Path is relative to: /home/kernelci", "    at /srv/staging.kernelci.org/app/dashboard/static/js/lib/r.js:26773:35"]}

[nuclearcat]

Strange, but seems on second try it went thru

[gctucker]

Thanks for this, it should be a very helpful tool during development of the new API & Pipeline as well as an extra way of prototyping things for the new web dashboard.

I would suggest to create a branch for this and host it on staging.kernelci.org with an alternative port number, so we don't risk breaking anything on the production web dashboard. Would that be OK? If so I can create the branch e.g. api-experimental and create a SysAdmin task to set up the instance.

[gctucker]

Some notes about the API:

• There aren't currently any kernel builds run with the new pipeline, but that's the very next step now we have KUnit and fstests running (they do their own builds under the hood) and we have the code in place for sending artifacts to storage.
• The API hasn't been tuned at all for a web dashboard use-case, it only has primitives and the database indexes haven't really been designed yet. We could add a websocket interface to get real-time updates from the API, as well as extra API entry points to make things easier with queries that return data in an efficient way for a web dashboard. It's a bit premature to do all this right now but it's useful to keep in mind we'll need it at some point while we're adding new data schema and features etc.
• Please keep track of any ideas and improvements you might have while working on this, even if they don't seem realistic or useful. Any kind of brainstorming is good at this stage in preparation for making some plans about the new web dashboard. Feel free to create some GitHub issues or add comments to the on-going discussion. This can include suggestions about how to evolve the API for a web dashboard use-case.

[gctucker]

One thought, if you have some time to spend on this: as a follow-up it would be nice to have a login page since we already have user accounts with the new API. It's using OAuth2 with JWT tokens, so I believe there are standard libraries to cover the implementation and we just need a basic web form to interact with the users. See the API docs for more details.

[mgalka]

Putting staging-skip label as it looks there are some issues with deploying this changes to staging

[alexandrasp]

Thanks for this, it should be a very helpful tool during development of the new API & Pipeline as well as an extra way of prototyping things for the new web dashboard.

I would suggest to create a branch for this and host it on staging.kernelci.org with an alternative port number, so we don't risk breaking anything on the production web dashboard. Would that be OK? If so I can create the branch e.g. api-experimental and create a SysAdmin task to set up the instance.

Thanks, @gctucker. I believe this is the best alternative. Keep a separate branch with these experimental changes.

[alexandrasp]

One thought, if you have some time to spend on this: as a follow-up it would be nice to have a login page since we already have user accounts with the new API. It's using OAuth2 with JWT tokens, so I believe there are standard libraries to cover the implementation and we just need a basic web form to interact with the users. See the API docs for more details.

I will create a ticket to have this in mind for next year so I can discuss it with Gustavo Padovan, and we can decide together how much effort is to put into this.