RFC: Cargo replication

[leon3s]

Could be nice to brainstorm about how we can define a cargo replication.

We should be able to :

• run only one instance on all nodes
• run only one instance on targeted nodes
• run one instance in one node
• run x defined instances in all node
• run x defined instances in targeted nodes
• run x defined instances in one node
• autoscale on all nodes
• autoscale on targeted nodes
• autoscale on one node

What else we should be able to do ?
And how we define the data ?

Example of the enum that came in my mind:

/// Auto is used to automatically define that the number of replicas in the cluster
/// Number is used to manually set the number of replicas
/// Note: auto will ensure at least 1 replica exists in the cluster
#[derive(Debug, Clone)]
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[cfg_attr(feature = "serde", serde(rename_all = "PascalCase"))]
#[cfg_attr(feature = "dev", derive(ToSchema))]
pub enum ReplicationMode {
  /// Auto is used to automatically define that the number of replicas in the cluster
  /// This will ensure at least 1 replica exists in the cluster
  /// And automatically add more replicas in the cluster if needed for redundancy
  Auto,
  /// Unique is used to ensure that only one replica exists in the cluster
  Unique,
  /// UniqueByNode is used to ensure one replica is running on each node
  UniqueByNode,
  /// UniqueByNodeGroups is used to ensure one replica is running on each node group
  UniqueByNodeGroups { groups: Vec<String> },
  /// UniqueByNodeNames is used to ensure one replica is running on each node name
  UniqueByNodeNames { names: Vec<String> },
  /// Number is used to manually set the number of replicas in one node
  Number(i64),
  /// NumberByNodes is used to manually set the number of replicas in each node
  NumberByNodes(i64),
  /// NumberByNodeGroups is used to manually set the number of replicas in each node group
  NumberByNodeGroups { groups: Vec<String>, number: i64 },
  /// NumberByNodeNames is used to manually set the number of replicas in each node name
  NumberByNodeNames { names: Vec<String>, number: i64 },
}

[lsabi]

Just to be clear: when you say

run `x` instances in targeted nodes

etc., do you mean to run, e.g., x instances on a certain subset of nodes y, which may be smaller, equal or even greater than x?

With replicas do you mean what k8s means with ReplicaSet? Or just the number of instances without state, e.g. a normal deployment?

https://kubernetes.io/docs/concepts/workloads/controllers/replicaset/

I'm asking, because I'm very interested in using such a project

[leon3s]

etc., do you mean to run, e.g., x instances on a certain subset of nodes y, which may be smaller, equal or even greater than x?

Nanocl is already thinked to be able to group nodes, so you could choose in what group of node you will run your workload and in this case it will be equal to X statically.

With replicas do you mean what k8s means with ReplicaSet? Or just the number of instances without state, e.g. a normal deployment?

In this particular case it will be like just the number of instances without state, e.g. a normal deployment.

Im thinking to do a separate program for horizontal autoscaler with his own yaml format.
That way people can write their own if needed.

Also adding something like NodeAffinity for normal deployment and the autoscaler
eg: you want node that have a graphic card, where a specific cargo or vm doesn't exist. to have maximum control of your workload placement.

Overall this is still being brainstormed and it's not implemented yet.
So im open to sudgestions!

[lsabi]

Nanocl is already thinked to be able to group nodes, so you could choose in what group of node you will run your workload and in this case it will be equal to X statically.

I see, so run x defined instances in all node is intended as run x instances on y nodes, regardless on which nodes. For any x < y. It was not clear from for the initial issue.

Im thinking to do a separate program for horizontal autoscaler with his own yaml format.
That way people can write their own if needed.

I think there should be a default one, batteries included. Or multiple options to choose from. Like having a list of pre-loaded strategies/policies and some user defined/coded policies to choose from. One may need different policies depending on the type of service.

[leon3s]

I think there should be a default one, batteries included. Or multiple options to choose from. Like having a list of pre-loaded strategies/policies and some user defined/coded policies to choose from. One may need different policies depending on the type of service.

Like for the proxy and the dns it will be included by default, but i like to keep things plug and play, so if you want to rewrite your own it's a possibility, for example if you wanna use traefik as a proxy instead of nginx that's possible.

Any tought on what the yaml could look like for the replication ?

[lsabi]

Any tought on what the yaml could look like for the replication ?

I don't know the structure behind nanocl and thus the best format. That being said, I think a possibility is that it should be as close as possible to the format of k8s, so that it can make learning simpler, attracting k8s users and allowing them to try nanocl without going crazy (it happened to me when try out some new frameworks). Otherwise, there could be a simple cli tool that converts from k8s's format to nanocl's format.

The underlying idea is to make transitioning from k8s to nanocl simpler.

[leon3s]

That's definitly a possibility, the main reason why we don't follow the same spec for most of k8s objects it's that it would have been less creative to just copy everything, and im my opinion our spec is simpler to use, currently you write less yaml to expose a service then k8s, but there is some concept that could be integrated and that are already similar, eg: jobs, secrets.

If you are familiar with docker, we currently have the exact same spec as their api to create cargoes and jobs, with some additions.
Same for the CLI command, they are pretty similar to what docker cli offer.

Having a tools to convert K8s yaml format to nanocl file format is definitly something planned in the future, we have already done that for docker-compose files with c2ncl. As it allow easier migrations.

The replication feature is the next big things to integrate right now, so i'll update the issue with what i think could be nice to have as yaml format.