-
Notifications
You must be signed in to change notification settings - Fork 36
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Resource implementations for AWS::CloudFormation::ResourceVersion and AWS::CloudFormation::ResourceDefaultVersion #4
Resource implementations for AWS::CloudFormation::ResourceVersion and AWS::CloudFormation::ResourceDefaultVersion #4
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This falls into the same trap that Lambda has where you can't create two versions in a single stack.
What if you rename So you can have multiple Although, I'm not really aware of an use case that needs multiple versions registered. MyType:
Type: AWS::CloudFormation::Type
Properties:
Version: !Ref MyTypeVersion1
TypeName: Organisation::Service::Resource
ExecutionRoleArn: arn:aws:iam::123456789012:role/MyExecutionRole
LoggingConfiguration:
LogRoleArn: arn:aws:iam::123456789012:role/MyLoggingRole
LogGroupName: MyResourceLogGroups
MyTypeVersion1:
Type: AWS::CloudFormation::TypeVersion
Properties:
Kind: Resource
TypeName: Organisation::Service::Resource
DefaultVersion: "00000001"
SchemaHandlerPackage: my-artifact-bucket
|
In my ideal world, it's three resource types, MyResource:
Type: AWS::CloudFormation::Resource
Properties:
TypeName: Organisation::Service::Resource
MyResourceVersion:
Type: AWS::CloudFormation::ResourceVersion
Properties:
ResourceType: !Ref MyResource
ExecutionRoleArn: arn:aws:iam::123456789012:role/MyExecutionRole
LoggingConfiguration:
LogRoleArn: arn:aws:iam::123456789012:role/MyLoggingRole
LogGroupName: MyResourceLogGroups
SchemaHandlerPackage: my-artifact-bucket
MyResourceDefaultVersion:
Type: AWS::CloudFormation::ResourceDefaultVersion
Properties:
ResourceType: !Ref MyResource
DefaultVersion: !GetAtt MyResourceVersion.VersionId This would allow multiple versions to be defined in a single template. This may become important when versioning gains more semantics (aliases representing major versions). I take the view that if you can't represent any state of the service achievable through the console in a single template, the resource model is broken. Another benefit is that resource versions would get deregistered individually as the resources that represent them get deleted, which would help in the case that, for example, credentials get accidentally hard coded in the source rather than stored externally. That said, it's verbose, and maybe there's a way to define |
Thanks for the feedback @pgarbe and @benkehoe. Let's experiment with these ideas. I'm going to be somewhat offline for a couple of days, but I'll aim to try and bootstrap the repo with API calls and setup so we can see how this plays out. Or if one of you want to have a crack at a PR, you could refer to one of the working examples like AWS::Logs::LogGroup for some of the conventions we use as a starting point. |
This is actually not possible in the underlying APIs anyway; one registration is allowed to be in flight at the same time, and (currently) only one version can be set as the |
Sure. And you'll argue that since you can't deregister a type unless it has only one version, even though a type with lots of versions would require a long sequence of stack updates to restore, deregistering it by accident is unlikely and so we don't need to account for that case. And so given the API I guess it's probably ok to leave it without multiple version representation. But I do want to register my disappointment that even on the CloudFormation team you're not starting with CloudFormation resource design, ensuring proper infrastructure as code lifecycle management, and working backwards from there to API design. The fact that there can be only one default would point to
|
That's even more confusing. From the CLI, I'd assume that you can register multiple versions for a type and point the "default" to one of the registered versions. Can you elaborate what you mean by this is not possible? |
That's true. What Ryan was saying is that you can't have two simultaneous RegisterType calls trying to create new versions at the same time, which is what might happen if there is a resource for |
FYI I've added a basic implementation against this schema, haven't tested it yet. Just wanting to show progress and that I haven't walked away. :) Also - build is failing because I didn't meet my own coverage bar (yet). 💔 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This immutable update style of this implementation won't work, because CloudFormation won't let you create a second resource with the same primaryIdentifier (which is Type
/TypeName
). if you ever try to update it, the update will fail because the type/typeName already exists.
We can solve this by adding an update handler which basically does the same thing as create (but doesn't need to check for existence). The only problem then (aside from hiding versions, which may or may not be desired) is that during delete we would probably have to list and then deregister all type versions.
aws-cloudformation-type/src/main/java/software/amazon/cloudformation/type/CreateHandler.java
Outdated
Show resolved
Hide resolved
aws-cloudformation-type/src/main/java/software/amazon/cloudformation/type/DeleteHandler.java
Outdated
Show resolved
Hide resolved
aws-cloudformation-type/src/main/java/software/amazon/cloudformation/type/CreateHandler.java
Outdated
Show resolved
Hide resolved
aws-cloudformation-type/src/main/java/software/amazon/cloudformation/type/Translator.java
Outdated
Show resolved
Hide resolved
So the PR currently 'works' here, except that the This template snippet demonstrates usage (you need your own code packages and S3 object links - make sure the code is actually different in each package or you'll eventually run into a separate bug which we need to fix);
And if you are interested, the issue blocking |
I like this resource layout, but I think the names should be changed to reflect the shift in responsibility. I'd propose AWS::CloudFormation::TypeVersion and AWS::CloudFormation::TypeVersionAlias. So it would look like this: Resources:
InitialType:
Type: AWS::CloudFormation::TypeVersion
Properties:
Type: RESOURCE
TypeName: Sample::CloudFormation::Resource
SchemaHandlerPackage: s3://cloudformationmanageduploadinfrast-artifactbucket-123456789012abcdef/sample-cloudformation-resource.zip
UpdatedType:
Type: AWS::CloudFormation::TypeVersion
Properties:
Type: RESOURCE
TypeName: Sample::CloudFormation::Resource
SchemaHandlerPackage: s3://cloudformationmanageduploadinfrast-artifactbucket-123456789012abcdef/sample-cloudformation-resource-update.zip
DependsOn: InitialType
DefaultVersion:
Type: AWS::CloudFormation::TypeVersionAlias
Properties:
Arn: !Ref UpdatedType
AliasName: $default # only valid value currently
# VersionId: !GetAtt UpdatedType.VersionId
VersionId: 00000002 I'd argue for a) having an alias name that required to be a specific value, rather than making default version different from possible future aliases from the IaC perspective (templates don't need to match the API calls) and b) going with I'm sad that it's I get that this will save duplication of code between the resources and matches the API, but it feels like optimizing this handler implementation at the expense of the user experience. |
I could probably be convinced that this is OK. You're right that I'm optimizing for implementation detail here and that's probably the wrong approach. Can always share 99% of the code behind slightly different schemas if needed. There's a bit of YAGNI here too and I should worry about what we have more than what we may have later. 😄 |
Oh, fabulous! So you can be convinced--what do I need to do to convince you? |
Send beer. 👌 |
I tried to buy you all beer at re:Invent for exactly this kind of eventuality but Amjad wouldn't let me |
Latest revision has the new type names as suggested. Still a problem on setting the |
Pinging on my question about MyVersion:
Type: AWS::CloudFormation::TypeVersionAlias
Properties:
Arn: !Ref UpdatedType
AliasName: SomeAliasName
# VersionId: !GetAtt UpdatedType.VersionId
VersionId: 00000002 Given that, it seems like the |
Yeah that makes sense. I'm still fixated on the current nomenclature having only a Right now I'm blocked on completion by a change I need to make on the backend - which I've done, just needs to be rolled out. |
cdb7689
to
b5a2753
Compare
I'm not sure if this is already provided by the code/rpdk (my java is very rusty), but to come back on: DefaultVersion: !GetAtt UpdatedType.VersionId
DefaultVersion: "00000002"
DefaultVersion: 00000002 I think it might make sense to actually fail and give an error message in the case that the supplied version is a number (even if the API would be able to set the default version.
|
It's definitely unclear whether version numbers are strings or numbers, and whether or not the leading zeros are required. If they are...are you sure you have enough? I might need 100 million versions, I am constantly screwing things up 😜 |
The string/number uncertainty would be consistent with the rest of the cfn service though
Almost consistent.
IIRC Elastic Beanstalk Option Settings fail if the values are not strings.
|
Did I get it right that the type I'm not sure if makes sense to compare it with Lambda Version/Alias. A typical use case that I've in mind is to just change the package version that is provided by the Vendor. So my first deployment is: Resources:
MySampleResource:
Type: AWS::CloudFormation::Resource
Properties:
TypeName: Sample::CloudFormation::Resource
SchemaHandlerPackage: s3://cloudformationmanageduploadinfrast-artifactbucket-123456789012abcdef/sample-cloudformation-resource-v0.0.1.zip and in the second deployment, I just change the package location: Resources:
MySampleResource:
Type: AWS::CloudFormation::Resource
Properties:
TypeName: Sample::CloudFormation::Resource
SchemaHandlerPackage: s3://cloudformationmanageduploadinfrast-artifactbucket-123456789012abcdef/sample-cloudformation-resource-v0.0.2.zip Everything else should be handled by CloudFormation and as user I don't care how many and which API calls are necessary. And in that case I wouldn't call it |
- AWS::CloudFormation::ResourceVersionAlias converted to AWS::CloudFormation::ResourceDefaultVersion - Implemented as callChain and tests updated
One thing to note - the example template represents a stack which may fail deletion on the first pass due to the order in which the `InitialType` and `UpdatedType` are deleted. If this occurs, a second delete will succeed. This version requires the following PRs from the Java Plugin to be published before this can be finalised; * aws-cloudformation/cloudformation-cli-java-plugin#287 * aws-cloudformation/cloudformation-cli-java-plugin#288
5f11fcd
to
e13eb3e
Compare
I believe this version of the PR to be finalised. It depends on 2 pending changes in the Java plugin repo if testing;
It also needs a Maven-published version of https://github.com/aws-cloudformation/cloudformation-cli-java-plugin-testing-support for completion. |
- cd "$TRAVIS_BUILD_DIR/aws-cloudformation-resourcedefaultversion" | ||
# from Maven 3.6.1+, should use `--no-transfer-progress` instead of Slf4jMavenTransferListener | ||
- > | ||
mvn | ||
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn | ||
-B | ||
clean verify | ||
- cd "$TRAVIS_BUILD_DIR/aws-cloudformation-resourceversion" | ||
# from Maven 3.6.1+, should use `--no-transfer-progress` instead of Slf4jMavenTransferListener | ||
- > | ||
mvn | ||
-Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn | ||
-B | ||
clean verify |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should we have something like this?
jobs:
include:
- script: |
for directory in $TRAVIS_BUILD_DIR/aws-*; do
cd "$directory"
mvn -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn -B clean verify || travis_terminate 1
done
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/aws-cloudformation/devtools/pull/11 (this is for CodeBuild)
} | ||
], | ||
"primaryIdentifier": [ | ||
"/properties/Arn" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so this is provided by customers?
return initiator | ||
.translateToServiceRequest(Translator::translateToUpdateRequest) | ||
.makeServiceCall((awsRequest, sdkProxyClient) -> sdkProxyClient.injectCredentialsAndInvokeV2(awsRequest, sdkProxyClient.client()::setTypeDefaultVersion)) | ||
.done(setTypeDefaultVersionResponse -> new ReadHandler().handleRequest(proxy, request, callbackContext, proxyClient, logger)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do u even need this delegate if there are no read onlies?
"description": "The schema that defines the type.\n\nFor more information on type schemas, see Resource Provider Schema in the CloudFormation CLI User Guide.", | ||
"maxLength": 16777216, | ||
"minLength": 1, | ||
"type": "string" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should that be multityped? can we use Map as well?
'oneOf': [
{
"type": "string",
"maxLength": 16777216,
"minLength": 1
},
{
"type": "object"
}
]
}, | ||
"SchemaHandlerPackage": { | ||
"description": "A url to the S3 bucket containing the schema handler package that contains the schema, event handlers, and associated files for the type you want to register.\n\nFor information on generating a schema handler package for the type you want to register, see submit in the CloudFormation CLI User Guide.", | ||
"type": "string" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe regex with ^(s3://)[a-zA-Z0-9:/_-]+$
// pre-read to capture required metadata fields in model for Delete | ||
return new ReadHandler().handleRequest(proxy, request, callbackContext, proxyClient, logger) | ||
// now deregister the type | ||
.onSuccess(progress -> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
slick
.translateToServiceRequest(model -> | ||
Translator.translateToDeleteRequest(progress.getResourceModel(), logger)) | ||
.makeServiceCall((awsRequest, proxyInvocation) -> proxyInvocation.injectCredentialsAndInvokeV2(awsRequest, proxyInvocation.client()::deregisterType)) | ||
.success()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no stabilization? DeregisterTypeResponse
contains RequestId
. should we describe to make sure?
CallChain.Initiator<CloudFormationClient, ListTypeVersionsResponse, CallbackContext> | ||
listApi = initiator.rebindModel(ListTypeVersionsResponse.builder().build()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this looks a bit hacky to bind response instead of a model but I guess we cant avoid it for now..
ProgressEvent<ListTypeVersionsResponse, CallbackContext> response = | ||
listApi.translateToServiceRequest(incoming -> | ||
ListTypeVersionsRequest.builder() | ||
.deprecatedStatus(context.getDeprecatedStatus()) | ||
.nextToken(incoming.nextToken()) | ||
.typeName(model.getTypeName()) | ||
.type(RegistryType.RESOURCE).build()) | ||
.makeServiceCall((r, c) -> c.injectCredentialsAndInvokeV2(r, c.client()::listTypeVersions)) | ||
.handleError( | ||
(request_, exception, client, model_, context_) -> { | ||
if (exception instanceof TypeNotFoundException) { | ||
// registration can be assumed to be the first version for a type | ||
return ProgressEvent.success( | ||
ListTypeVersionsResponse.builder().typeVersionSummaries( | ||
getPredictedSummary(request, model) | ||
).build(), | ||
context | ||
); | ||
} | ||
throw exception; | ||
} | ||
) | ||
.done(response_ -> ProgressEvent.success(response_, context)); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe just use a direct api call as u dont really register anything in the callgraph but it makes code less verbose
...ourceversion/src/main/java/software/amazon/cloudformation/resourceversion/CreateHandler.java
Show resolved
Hide resolved
The backend allows for 180s of runtime to allow for buffer and latency in timing calculations. 60s was causing most contract tests to fail in SAM Local though the type is functioning correctly in CloudFormation. See: aws-cloudformation/cloudformation-cli#505
Contract tests for
NOTE: ListHandler is removed from the schema for now as it requires careful implementation to change to list versions not just types. That would be simpler as a Registry API than as pagination-within-pagination against the |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
lets get this out and we can iterate on it |
Issue #, if available: #3
Description of changes:
Schema and handler implementations for
AWS::CloudFormation::ResourceVersion
andAWS::CloudFormation::ResourceDefaultVersion
.Sample template for use of these type;
Pending contract test verification.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.