Skip to content

Commit

Permalink
Implement profile inheritance
Browse files Browse the repository at this point in the history
  • Loading branch information
LucasPickering committed Jul 13, 2023
1 parent 04a05eb commit 446ba4b
Show file tree
Hide file tree
Showing 8 changed files with 880 additions and 92 deletions.
6 changes: 4 additions & 2 deletions .env-select.toml
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
[applications.server.profiles.base]
variables = {PROTOCOL = "https"}
[applications.server.profiles.dev]
extends = ["base"]
variables = {SERVICE1 = "dev", SERVICE2 = "also-dev"}
[applications.server.profiles.prd]
extends = ["base"]
variables = {SERVICE1 = "prd", SERVICE2 = "also-prd"}

[applications.empty]

# TODO move into tests/ directory
[applications.integration-tests.profiles.p1.variables]
VARIABLE1 = "abc"
Expand Down
23 changes: 23 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ toml = {version = "0.7.5", features = ["preserve_order"]}

[dev-dependencies]
assert_cmd = {version = "2.0.11", default-features = false, features = ["color-auto"]}
pretty_assertions = "1.4.0"
rstest = {version = "0.17.0", default-features = false}
rstest_reuse = "0.5.0"
serde_test = "1.0.165"
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Easily switch between predefined values for arbitrary environment variables Feat
- Cascading config system, allowing for system and repo-level value definitions
- Grab values dynamically via local commands or Kubernetes pods
- Modify your shell environment with `es set`, or run a one-off command in a modified environment with `es run`
- Re-use common variables between profiles with inheritance

## Table of Contents

Expand Down
69 changes: 69 additions & 0 deletions USAGE.md
Original file line number Diff line number Diff line change
Expand Up @@ -250,6 +250,75 @@ variables = {SERVICE1 = "prd", SERVICE2 = "also-prd"}

To see where env-select is loading configs from, and how they are being merged together, run the command with the `--verbose` (or `-v`) flag.

## Profile Inheritance

In addition to top-level merging of multiple config files, env-select also supports inheritance between profiles, via the `extends` field on a profile. For example:

```toml
[applications.server.profiles.base]
variables = {PROTOCOL = "https"}
[applications.server.profiles.dev]
extends = ["base"]
variables = {SERVICE1 = "dev", SERVICE2 = "also-dev"}
[applications.server.profiles.prd]
extends = ["base"]
variables = {SERVICE1 = "prd", SERVICE2 = "also-prd"}
```

During execution, env-select will merge each profile with its parent(s):

```sh
> es set server
❯ === base ===
PROTOCOL=https

=== dev ===
SERVICE1=dev
SERVICE2=also-dev
PROTOCOL=https

=== prd ===
SERVICE1=prd
SERVICE2=also-prd
PROTOCOL=https
```

Note the `PROTOCOL` variable is available in `dev` and `prd`. The profile name given in `extends` is assumed to be a profile of the same application. To extend a profile from another application, use the format `application/profile`:

```toml
[applications.common.profiles.base]
variables = {PROTOCOL = "https"}
[applications.server.profiles.dev]
extends = ["common/base"]
variables = {SERVICE1 = "dev", SERVICE2 = "also-dev"}
[applications.server.profiles.prd]
extends = ["common/base"]
variables = {SERVICE1 = "prd", SERVICE2 = "also-prd"}
```

#### Multiple Inheritance and Precedence

Each profile can extend multiple parents. If two parents have conflicting values, the **left-most** parent has precedence:

```toml
[applications.server.profiles.base1]
variables = {PROTOCOL = "https"}
[applications.server.profiles.base2]
variables = {PROTOCOL = "http"}
[applications.server.profiles.dev]
extends = ["base1", "base2"]
variables = {SERVICE1 = "dev", SERVICE2 = "also-dev"}
```

The value from `base1` is used:

```sh
> es run server dev -- printenv PROTOCOL
https
```

Inheritance is applied recursively, meaning you can have arbitrarily large inheritance trees, **as long as there are no cycles**.

## Configuration Reference

### Value Source
Expand Down
217 changes: 166 additions & 51 deletions src/config/inheritance.rs
Original file line number Diff line number Diff line change
@@ -1,69 +1,184 @@
//! Utilitied related to profile inheritance resolution
use crate::config::{Config, Profile, ProfileReference};
use anyhow::anyhow;
use indexmap::IndexSet;
use crate::config::{Config, DisplayKeys, Name, Profile, ProfileReference};
use anyhow::{anyhow, bail};
use indexmap::{IndexMap, IndexSet};
use log::trace;
use std::{collections::HashMap, fmt::Display, hash::Hash};

impl Config {
/// Resolve `extends` field for all profiles
/// Resolve inheritance for all profiles. Each profile will have its parents
/// (as specified in its `extends` field) merged into it, recursively.
pub(super) fn resolve_inheritance(&mut self) -> anyhow::Result<()> {
// Step 1 - Make sure application is set for all profile references
self.qualify_profile_references();
let mut resolver = InheritanceResolver::from_config(self);
resolver.resolve_all()
}
}

struct InheritanceResolver<'a> {
profiles: HashMap<QualifiedReference, &'a mut Profile>,
unresolved: IndexMap<QualifiedReference, IndexSet<QualifiedReference>>,
}

// Profiles we've *started* (and possibly finished) resolving
let mut visited: IndexSet<ProfileReference> = IndexSet::new();
// Profiles we've *finished* resolving
let mut resolved: IndexSet<ProfileReference> = IndexSet::new();
impl<'a> InheritanceResolver<'a> {
fn from_config(config: &'a mut Config) -> Self {
let mut profiles = HashMap::new();
let mut unresolved = IndexMap::new();

// Step 2 - resolve dependency tree
todo!();
// Flatten profiles into a map, keyed by their path. For each profile,
// we'll also track a list of parents that haven't been resolved+merged
// in yet
for (application_name, application) in &mut config.applications {
for (profile_name, profile) in &mut application.profiles {
let reference =
QualifiedReference::new(application_name, profile_name);
// Qualify relative references using the parent application
let parents = QualifiedReference::qualify_all(
application_name,
profile.extends.iter(),
);

// Step 3 - merge configs together
todo!();
// Any profile with parents is deemed unresolved
profiles.insert(reference.clone(), profile);
if !parents.is_empty() {
unresolved.insert(reference, parents);
}
}
}

trace!(
"Detected {} profiles needing inheritance resolution: {}",
unresolved.len(),
unresolved.display_keys()
);
Self {
profiles,
unresolved,
}
}

/// Resolve inheritance for all profiles
fn resolve_all(&mut self) -> anyhow::Result<()> {
// Resolve each profile. A profile has been resolved when its `parents`
// list is empty, so keep going until they're all done
while let Some((reference, parents)) = self.unresolved.pop() {
self.resolve_profile(reference, parents, &mut IndexSet::new())?;
}
Ok(())
}

/// Fully qualify all profile references. It'd be nice to have a different
/// type to enforce the reference is resolved, but it's not worth drilling
/// that all the way down the tree so we'll just do runtime checks later to
/// be safe
fn qualify_profile_references(&mut self) {
// First, go through and
for (application_name, application) in &mut self.applications {
for profile in application.profiles.values_mut() {
// We have to drain the set and rebuild it, since the hashes
// will change
profile.extends = profile
.extends
.drain(..)
.map(|mut reference| {
if reference.application.is_none() {
reference.application =
Some(application_name.clone());
}
reference
})
.collect();
/// Resolve inheritance for a single profile, recursively. This will also
/// resolve its parents, and their parents, and so on.
fn resolve_profile(
&mut self,
reference: QualifiedReference,
parents: IndexSet<QualifiedReference>,
visited: &mut IndexSet<QualifiedReference>,
) -> anyhow::Result<()> {
trace!("Resolving inheritance for profile {reference}");
visited.insert(reference.clone());

for parent in parents {
trace!("Resolving parent {reference} -> {parent}");

// Check for cycles
if visited.contains(&parent) {
bail!("Inheritance cycle detected: {}", display_cycle(visited));
}

// Check if parent needs to be resolved. If parent is an unknown
// path, we'll skip over here and fail down below
if let Some(grandparents) = self.unresolved.remove(&parent) {
// Parent is unresolved - resolve it now
self.resolve_profile(
parent.clone(),
grandparents,
// When we branch, we have to clone the `visited` list so
// it remains linear. This doesn't seem like a good
// solution, and yet it works...
&mut visited.clone(),
)?;
} else {
trace!("{parent} is resolved");
}

// We know parent is resolved now, merge in their values
Self::apply_inheritance(
(*self
.profiles
.get(&parent)
.ok_or_else(|| anyhow!("Unknown profile: {}", parent))?)
.clone(),
self.profiles
.get_mut(&reference)
.ok_or_else(|| anyhow!("Unknown profile: {}", reference))?,
);
}
Ok(())
}

/// Get a profile by reference. This should only be called *after*
/// qualifying all profile references
fn get_profile(
&self,
reference: &ProfileReference,
) -> anyhow::Result<Option<&Profile>> {
let application = reference.application.as_ref().ok_or_else(|| {
anyhow!(
"Unqualified profile reference {:?} during inheritance \
resolution. This is a bug!",
reference
)
})?;
Ok(self.applications.get(application).and_then(|application| {
application.profiles.get(&reference.profile)
}))
/// Merge a parent into a child, i.e. any data in the parent but *not* the
/// child will be added to the child
fn apply_inheritance(parent: Profile, child: &mut Profile) {
// TODO can we use Merge for this? Right now it's parent-prefential,
// but we may change that
for (variable, parent_value) in parent.variables {
// Only insert the parent value if it isn't already in the child
child.variables.entry(variable).or_insert(parent_value);
}
}
}

/// A [ProfileReference] that has been qualified with its application name,
/// such that it is globally unique
#[derive(Clone, Debug, Eq, Hash, PartialEq)]
struct QualifiedReference {
application: Name,
profile: Name,
}

impl QualifiedReference {
fn new(application: &Name, profile: &Name) -> Self {
Self {
application: application.clone(),
profile: profile.clone(),
}
}

/// Qualify all references for a given application. Any relative
/// references will be qualified with the application name
fn qualify_all<'a>(
parent_application: &'a Name,
references: impl IntoIterator<Item = &'a ProfileReference>,
) -> IndexSet<Self> {
references
.into_iter()
.map(|reference| QualifiedReference {
application: reference
.application
.as_ref()
.unwrap_or(parent_application)
.clone(),
profile: reference.profile.clone(),
})
.collect()
}
}

impl Display for QualifiedReference {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "{}/{}", self.application, self.profile)
}
}

/// Pretty print a cycle chain
fn display_cycle<T: Display>(nodes: &IndexSet<T>) -> String {
let mut output = String::new();
for node in nodes {
output.push_str(&node.to_string());
output.push_str(" -> ");
}
// Duplicate the first node at the end, to show the cycle
output.push_str(&nodes[0].to_string());
output
}
Loading

0 comments on commit 446ba4b

Please sign in to comment.