Add TeamCity CI config files

[LecrisUT]

Reserving the top comment to add notes on the usage of the CI

[LecrisUT]

This also includes sstate-cache server, currently proposed as https://sstate-cache.asteroid.org. Need to change the parameters and double-check what is being published before adding the .teamcity folder.

[FlorentRevest]

I appreciate that some sort of modern automation tool could potentially bring AsteroidOS a lot of benefits and thanks for looking into this! However, I have concerns about how things are done here.

First of all, you never stated clearly what your goals were. What are you trying to achieve, exactly ? For example: if the goal is to verify that pull requests don't cause regressions before they are merged, then the CI system should really build all of asteroid-image, not just the part that got modified (otherwise it wouldn't catch things that depended on it and broke). Alternatively, if your goal is to distribute a sstate cache on asteroidos.org, we already have a super dumb cron script that builds asteroid for all watches and updates https://release.asteroidos.org/nightlies why couldn't this be trivially extended to share its sstate ? If your goal is to rebuild asteroid as soon as something is pushed, why isn't that enough to just build it once a day ? If you'd clarify what you're trying to achieve, it would make the technical tradeoffs much more apparent and make our lives as maintainers considerably easier.

Then, it's fairly obvious that you spent a lot of time working on the implementation of all of this and we appreciate the effort. However, it does not seem like any of your big design decisions were discussed or documented anywhere ? For example, why TeamCity and not something else ? Maybe you had excellent reasons to choose it but from our perspective as maintainers we just get a giant commit full of xml files which are virtually impossible to review to configure a random tool we never heard of. We won't merge this unless you can make a really strong case for it (as opposed to something simpler). Also consider this situation which already happened in the past: someone contributed Jenkins configs without documentation or assistance and moved on with their life: https://github.com/AsteroidOS/jenkins-job-builds because no maintainers were onboarded to it so this just rotted in a dark corner. I fear that history is repeating itself here with whatever is the latest trendy tool. One of the questions I would like to see answered for example is: why rebuild something from scratch instead of reviving the old jenkins automation ?

Also, the changes required by this automation seem fairly intrusive. You originally proposed splitting a Yocto layer (although that seems to no longer be the case) or changing our commit policy. All of these big changes deserve discussions and as long as we won't have a clear picture of what you're trying to achieve and why this way we won't be merging this.

Anyway, bear with us, I'm sure your contributions could be very useful so don't loose sight of your goals. I'm just dubious that the current implementation is the best way to get there. ;)

[LecrisUT]

Just a heads up, I can't respond to the comments until next week due to conference, but please remind me around that time.

Overall TeamCity itself is not a strictly necessary tool, but it can be useful for the end-user to setup their own environment on their own machine without docker or anything. We can try out other CI tools that the team is more comfortable with, and if there are any specific features in any of the tools we can migrate the workflow to that tool.

Otherwise, the structural changes, job workflow and comment structure are applicable to any CI tool. I'll walk through the ideas here when I get back.

[LecrisUT]

@FlorentRevest I am almost back in a stable schedule, so let me address some of your issues:

• Firstly I did add a CI.md file to put all of the discussions about the minimum requirements for a CI and some benefits that TeamCity might have. Let's move some of the conversation there, and also please give me some prospective on other CI solutions.
• About the hundreds of file changes added here, I found that I can change to a kotlin script so that it becomes a single (or 2-3) file so that it's more manageable to review
• Contact me on matrix and I will give you an account on my teamcity instance to navigate the project in a more UI friendly manner. You can also confirm that the settings are identical by adding the project directly from the relevant branch on your teamcity instance.
• If I miss some of the issues in our discussions in the CI.md please remind me to add them there

As for the specific issues:

First of all, you never stated clearly what your goals were.

There are 2 goals: To confirm that PR builds are buildable outside the dev's personal environment; and to periodically rebuild the dev environment from scratch to confirm that newcommers will be able to do the same builds themselves. With a CI as opposed to a cron job, we have more finer details on where things go wrong, and we can track specific upstream third-party repos.

This PR also includes the sstate-cache addition, but maybe we should separate that and add it sooner.

However, it does not seem like any of your big design decisions were discussed or documented anywhere ?

Indeed we still need to discuss this, and I am fully open to switching to different environment, but here are the reason for my choice of teamcity:

• It can run dockerless, so any dev can implement their own teamcity instance on their own machine.
• I haven't tested it yet, but according to the documentation we can link different teamcity servers so each maintainer for a specific watch can run/manage their own server
• We can track third-party repositories for changes
• Builds can be triggered/ignored by commit message, and by specific changes to some folders

Of course it's not a perfect solution:

• Github action/app is still missing, but it's in development
• Connection to cloud platform to start/stop agents is still not fully developed
• Limit of 3 agents and 100 build configurations. Applying for an open source license and/or distributing the project across multiple developers can circumvent this issue.
• Closed source, however jetbrains are so far one of the more ethical companies out there

The only other CI solution I have looked at is CircleCI so please let me know which ones you want me to look into and I'll make a table of the required and helpful features of each.

someone contributed Jenkins configs without documentation or assistance and moved on with their life

This is always a risk, so I want to document as much of my journey as possible, at the very least in CI.md and later in a page on the website.

You originally proposed splitting a Yocto layer (although that seems to no longer be the case) or changing our commit policy

Yes, I got a better understanding of Yocto, and just meta-asteroid#89 is sufficient for this. As for the commit policy, these are not requirements, just suggestions for the core team if they want some finer control of the CI tool. Certainly I will not want external contributors to follow those, so I tried to make the build as flexible as possible. The format itself still needs to be discussed, and I will appreciate any help with the Regex part of it.

Anyway, sorry for the wall of text, so let's split up our discussions in smaller sections in CI.md

[LecrisUT]

Made a drastic simplification to the teamcity CI by switching to a Kotlin settings file. The configurations still need a bit of work to get it to build, but I'll work on documenting the CI.md a bit more.

Edit: I will start putting it in production on my side, and I am up for a new round of discussion.