[concourse] No new nightly images since Apr 21.

[OscarL]

Expected behavior: builds trigger once a day (ideally, only if there has been any new commit at all, see issue #76).

(BTW, "issues templates" could use an "other" option for things like this. Edit: Doh! I missed the "Open a blank issue." link at the bottom of the "new issue page" :-D)

[leavengood]

We had a power outage at my house, probably around that time. I left my builder off, thinking @kallisti5's builder would pick up the slack, but I guess not. I just started it back up and it looks like Concourse started a build. Though it seems to be using kallisti5's builder. So I guess mine being off stops all the builds? 🤔

We really need a better system to keep track of builder status. My device is headless and half the time I don't really know if it is still even getting jobs from Concourse. Also when both builders are running Concourse does stupid stuff like try to pass around artifacts. See #107.

[kallisti5]

Weird, so concourse seems to think my builder is healthy, but it's seemingly not..

name    containers  platform  tags     team  state    version  age 
hbav01  0           linux     site_av  none  running  2.5      67d 
hbrl01  7           linux     none     none  running  2.5      5h2m

Notice the tag "site_av", that is what keeps the builds running on my physical site by default. Since we can't let builds run "anywhere" due to the streaming issues Ryan mentioned, I set a tag on concourse to keep them to a single "physical" site.

https://github.com/haiku/infrastructure/blob/master/concourse/apply-pipeline.sh#L8

The idea was we can "switch" sites when a worker is down, however that requires someone to reapply the pipelines to "switch".

None of this is ideal. I really love concourse, but the rigidity of their worker logic (and their seemingly stubborn stance they have on it) is kinda killing it as a solution as we grow (unless we want a pool of workers within the same physical site / network, but the amount of volume streaming / sharing still seems overdramatic by concourse)

Anyway, I restarted my worker. I'm about to head out of town so i'll make sure things are stable again before leaving.

$ fly -t haiku workers
name    containers  platform  tags     team  state    version  age  
hbav01  0           linux     site_av  none  running  2.5      15s  
hbrl01  7           linux     none     none  running  2.5      5h11m

[kallisti5]

Small note for Ryan, this is what adds that "site tag"..

/opt/concourse/worker.env, Add:

CONCOURSE_TAG="site_av"

[kallisti5]

As for alternatives, i've been playing with doing builds directly in kubernetes. It's not functional yet though since building "containers in containers" is kind of a pain 😓

https://github.com/haiku/infrastructure/tree/master/playground/build/toolchain

[OscarL]

Forgot to close this one, sorry. Can be reopen (or more specific issues created) if needed.