env var AWS_REGION - will this requirement ever go away??

[pjfoley]

tldr - I am curious if the AWS_REGION is a legacy code artifact that will eventually go away. My question is if the provider allows you to specify a region in the puppet manifest, why doesn't the provider just default to that region?

I think that this section a note on regions needs to have 20 foot sized flashing text that reads - "here be dragons - set this environment variable or add a large delay for each aws puppet type processed".

I incorrectly assumed (up until tonight) that by setting the region in the puppet manifest for each aws type that was all I needed to do.

While trying to understand some of the provider code I stumbled on puppet_x/puppetlabs/aws.rb#L26-L32 which appears to be called for every provider (or at least the ones I checked). For example ec2_vpc_routetable.

Don't believe me, unset the environment variable and run any of the examples. For example:

$ unset AWS_REGION
$ env | grep AWS_

$ pwd
/root/dev/puppet-modules-development/aws/examples/vpc-example
$ time puppet apply init.pp --modulepath ~/dev/puppet-modules-development/
Notice: Compiled catalog for atom in environment production in 0.18 seconds
Notice: /Stage[main]/Main/Ec2_vpc[sample-vpc]/ensure: created
Notice: /Stage[main]/Main/Ec2_vpc_internet_gateway[sample-igw]/ensure: created
Notice: /Stage[main]/Main/Ec2_securitygroup[sample-sg]/ensure: created
Notice: /Stage[main]/Main/Ec2_vpc_routetable[sample-routes]/ensure: created
Notice: /Stage[main]/Main/Ec2_vpc_subnet[sample-subnet]/ensure: created
Notice: Finished catalog run in 107.09 seconds

real    1m49.708s
user    0m3.632s
sys 0m0.732s

$ pwd
/root/dev/puppet-modules-development/aws/examples/vpc-example
$ export AWS_REGION=ap-southeast-2
$ env | grep AWS_
AWS_REGION=ap-southeast-2
$ time puppet apply init.pp --modulepath ~/dev/puppet-modules-development/
Notice: Compiled catalog for atom in environment production in 0.18 seconds
Notice: /Stage[main]/Main/Ec2_vpc[sample-vpc]/ensure: created
Notice: /Stage[main]/Main/Ec2_vpc_internet_gateway[sample-igw]/ensure: created
Notice: /Stage[main]/Main/Ec2_securitygroup[sample-sg]/ensure: created
Notice: /Stage[main]/Main/Ec2_vpc_routetable[sample-routes]/ensure: created
Notice: /Stage[main]/Main/Ec2_vpc_subnet[sample-subnet]/ensure: created
Notice: Finished catalog run in 5.27 seconds

real    0m7.967s
user    0m2.900s
sys 0m0.288s

Essentially if the AWS_REGION environment variable is not set all valid regions will be returned to the provider, and than the provider will iterate across every region looking for valid information.

[garethr]

As you note the AWS_REGION variable is intended as a performance optimisation, although as you've found it's both under-documented and underspecified at the moment.

Some discussions are happening around #132 and #117 which addresses some of that. Once this is clearer we'll provide some examples of usage and more discussion of context.

There are also a number of performance optimisations that are possible, the first batch in #102.

[pjfoley]

Two follow on questions

1. Is there a reason why it cannot just use the region the user sets as part of the function call?
2. In addition to an environment variable could it check for a puppet global variable with the preferred region?

[garethr]

1. I presume you mean the regions specified in the various resources, like ec2_instance? Puppet works by first building a graph of all the resources, it then triggers a prefetch action to load all of the resources of a specific type - it's at this point we loop through the regions. It may be possible to restrict this loop to only those regions specified, we'll look into that. This means we don't trigger one call per resource, so we do one prefetch per resource type, not one per instance.
2. That's an interesting idea, we'll definitely have a look at that as an option too

In addition, whether configured via an environment variable or a config setting we should allow you to provide a list of regions. We definitely want to support graphs of resources across regions, but it's balancing that with the performance characteristics.

[aharden]

I wanted to pick up the discussion on point 2 above regarding the use of a puppet global variable in addition to or instead of an environment variable. I'm just getting back into writing AWS infrastructure code intended to be run on an arbiter during regular puppet agent runs and those runs can't access environment variables. After unsuccessfully trying to figure out how to consume variables set in the Puppet Enterprise console in aws.rb (which is run completely agent-side), I decided to inject the values as local facts from the arbiter node itself. That works. Before I put in a pull request to add Facter lookups for the region and proxy variables, I was wondering if anyone's made progress (or would know how to) consume PE console variables for this purpose. Using Facter for this seems hackish, but since I can manage the disk-based facts as part of the arbiter's resources it's acceptable to me.

[earsdown]

+1 for global variable option, preferably by honouring value set in ~/.aws/config

[garethr]

Just to note that you can now use a config file in Puppet's confdir, specifically called puppetlabs_aws_configuration.ini.

11c4916

It doesn't use ~ as the HOME environment variable gets blatted by the agent.

[pjfoley]

Hi @garethr,

Thanks that is perfect!

Peter