Use NixOS on the production site

[ForNeVeR]

I have a log-standing target of using OS with declarative and reproducible configuration instead of constantly failing LTS Ubuntu that I have to mess up manually.

(And we're already messing it up much: we have some custom PPAs for GHC, Prosody and dotnet-cli, we have some custom configuration of firewall, we have fail2ban and I don't know whether it works or not, we have VPN server set up and I have no freakin' idea how to manage it all; everything is ad-hoc and undocumented, although I'm trying to keep the copies of the changed configurations in a safe place.)

So, I have the following plan:

1. Experiment with a fresh VM conversion.
   1. Create a new virtual machine on DigitalOcean.
   2. Set up the same Ubuntu version as we use on the production site.
   3. Try using something like nixos-assimilate or nixos-in-place on that machine.
2. Experiment with our VM backup.
   1. Make a backup of our current production VM (even if it'll cost me a bit of money).
   2. Deploy the backup to some staging site.
   3. Convert that machine on the staging site using nixos-assimilate while preserving the existing services (prosody, loglist, ctor, nginx) and documenting the experience (ideally in form of deployable config file).
3. Decide whether it's still worth based on the results of 1 and 2.
4. Convert the production machine!
5. Store the configs somewhere accessible for the ops team members. Probably even in a publish repository (while removing the security and password parts to some private file, for sure).

[ForNeVeR]

@rexim, @hagane, @Minoru: I request for your comments. Maybe it'll be worth to check CoreOS instead of NixOS or something else? Maybe you have some other ideas?

[rexim]

@ForNeVeR I find this experiment really interesting and I vote for it. The results of this experiment can produce pretty useful artificats (which I hope to use later on my Stream Hub, which is also on DO with Ubuntu).

If you need any help from my side, feel free to ask.

[Minoru]

That sounds nice and I vote for this, even though I'm not interested in any of the ops stuff at the moment.

Step 3 sounds like you'll abandon the idea if nixos-assimilate or nixos-in-place will fail to convert all existing services. Is that on purpose? I think it's still worth to automate configuration even if you'll have to convert configs by hand for some of the services .

[hagane]

@ForNeVeR CoreOS is only good for when you want to mangle your brain with big-D containers, microservices and whatever jolly insanity happening out there in the #gifee land (NB: I do intend to mangle my brain with all this go-se at once -- as soon as I find some interesting problem)

If I understood your intent correctly, definitely do give NixOS a try. OTOH you could try something less system-invasive, like ansible, or chef

[hagane]

@ForNeVeR also, i'd like to offer a donation to cover our infrastructure expenses and/or time to reconfigure our boxen.

[ForNeVeR]

@Minoru

Step 3 sounds like you'll abandon the idea if nixos-assimilate or nixos-in-place will fail to convert all existing services. Is that on purpose?

Nope, I didn't meant that. I'll abandon the idea if nixos-assimilate will fail to convert the whole box, and at the same time I'll fail to convert the services manually. That's not the task of nixos-assimilate to convert the services.

@hagane

you could try something less system-invasive, like ansible, or chef

I've already tried these, and they aren't so interesting: they won't help us to manage stuff at the level I'd want to. I am already using NixOS on one of my servers and one of my notebooks, so I think that it'll actually help.

i'd like to offer a donation to cover our infrastructure expenses and/or time to reconfigure our boxen.

I'll definitely let you know when your help will be useful. Currently it's just a plan, and I'm not sure if I'll have enough time and courage to implement it next week or next month. But thanks anyway, I'll keep that in mind.

[aszlig]

You could try Disnix, which I've been using at work during the transition from Debian GNU/Linux to NixOS.