sam.schema.json: Multiple valid CloudFormation constructs not recognized

[jmalins]

Describe the bug

Many advanced CloudFormation features are not currently recognized by the sam.schema.json JSON schema. This makes the AWS Toolkit unusable without disabling the YAML schema.

See details of issues below.

Number Parameter types should allow numeric defaults
If a parameter is type Number, CloudFormation accepts a numeric default value.

Parameters:
  IdPosition:
    Type: Number
    Default: 0 # <-- error: Incorrect type. Expected "string".

Functions in Conditions block
Rich nested functional expressions (most using arrays) need to be supported in the condition block. It is currently only supports one level of object type.

Conditions:
  HasEmailNotification: !Not [ !Equals [ !Ref NotificationEmailAddress, '' ] ] # <-- error: 'Incorrect type. Expected "object".'

Resource-Level Conditions
All resources can have a string-valued Condition: <condition-id> property.

Resources:
  # optional e-mail subscription #
  MonitoringEmailSubscription:
    Type: AWS::SNS::Subscription
    Condition: HasEmailNotification # <-- error: 'Property Condition is not allowed.'
    Properties:
      TopicArn: !Ref MonitoringTopic
      Protocol: email
      Endpoint: !Ref NotificationEmailAddress

String Valued Expressions
Functional expressions that evaluate to a string should be supported for all string-valued properties.

Resources:
  Database:
    Type: AWS::Glue::Database
    Properties:
      CatalogId: !Ref AWS::AccountId
      DatabaseInput:
        Name: !Join [ '_', !Split [ '-', !Ref AWS::StackName ] ] # <-- error: Incorrect type. Expected "string"
        LocationUri:
          Fn::Sub: # <-- error: Incorrect type. Expected "string"
            - 's3://${DataBucket}/databases/${__database_name__}/'
            - __database_name__: !Join [ '_', !Split [ '-', !Ref 'AWS::StackName' ] ]

Deep Fn::Transform Operators
The CloudFormation specification allows inserting a Fn::Transform operation at arbitrary places within the template hierarchy. This is the equivalent of every object allowing additionalProperties of Fn::Transform, instead of simply false.

Resources:
  ClaimsTable:
    Type: AWS::Glue::Table
    Properties:
      CatalogId: !Ref AWS::AccountId
      DatabaseName: !Ref Database
      TableInput:
        Name: claims
        TableType: EXTERNAL_TABLE
        Parameters:
          classification: "csv"
          has_encrypted_data : true
        StorageDescriptor:
          Location: !Sub 's3://${DataBucket}/databases/${Database}/tables/claims/'
          Fn::Transform: # <-- error: Property Fn::Transform is not allowed.
            Name: AWS::Include
            Parameters:
              Location: ./tables/claims.yaml

Custom Resources
Custom CloudFormation resources should be supported, as defined by any resource type starting with Custom:: prefix. These should require a ServiceToken property, but allow arbitrary additional properties.

Resources:
  MyCustomResource:
    Type: Custom::MyResource # <-- error: Value is not accepted. Valid values: "AWS::ACMPCA::Certificate"
    Properties: # <-- all additional properties marked as invalid
      ServiceToken: !Ref SomeResourceArn
      ArbitraryProp1: Value1
      ArbitraryProp2: Value2

IAMPolicyDocument type is missing Version property
Hand-coded IAM policies do not work... the policy schema type allows only the Statement property, without the (required) Version property. This causes the anyOf operator to fall back to expecting a string.

Resources:
  ExtractDataStep:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: ./dist
      Runtime: nodejs14.x
      Handler: ExtractDataStep.handler
      Tracing: Active
      Policies:
        - Version: '2012-10-17' # <-- error: Incorrect type. Expected "string".
          Statement:
            - Sid: GetS3SourceArchive
              Effect: Allow
              Action:
                - s3:GetObject
                - s3:GetObjectVersion
              Resource:
                - !Sub ${DataBucket.Arn}
                - !Sub ${DataBucket.Arn}/data_intake/*

To Reproduce

Create a CloudFormation template with any of the above constructs.

Expected behavior

All valid CloudFormation syntax would be correctly recognized. The examples above should not produce any of the marked errors.

Desktop (please complete the following information):

• OS: MacOS 10.15.7 (Catalina)
• Visual Studio Code Version: v1.59.1
• AWS Toolkit Version: v1.29.0

[justinmk3]

Thanks for the report, and sorry for the trouble! How are you disabling the schema?

For reference, the schema is based on https://github.com/awslabs/goformation .

  Type: AWS::Serverless::Function
      Properties:
      Policies:
        - Version: '2012-10-17' # <-- error: Incorrect type. Expected "string".
          Statement:
            - Sid: GetS3SourceArchive

at a glance, it seems like that should be allowed by the schema: https://github.com/awslabs/goformation/blob/master/generate/sam-2016-10-31.json#L114-L131

We will continue to improve the support. Thanks for the concrete examples.

[bryceitoc9]

Just for clarification: the above file isn't the schema, but is used instead to define sam types for the schema that's generated in the end (all native CFN types are pulled from the official CFN spec). The actual schemas we're loading into the YAML extension can be found here: https://github.com/awslabs/goformation/tree/master/schema (ending in schema.json)

[jmalins]

Thanks for the prompt reply.

I disabled validation globally, in Settings > YAML > Disable validation. I should have said disable validation, not disable the schema... though I believe you can tell the language server to override the schema for a specific file by using the $schema directive.

My reading of https://github.com/awslabs/goformation/blob/master/generate/sam-2016-10-31.json#L1321 is that AWS::Serverless::Function.IAMPolicyDocument does not support the version property, but I may be missing something.

[jmalins]

At least one element is an open issue in the goformation repository:
awslabs/goformation#360

It has been open since March 25, 2021.

[2underscores]

+1

Few more examples we're having:

Resources:
  ParentCert:
    Type: AWS::CertificateManager::Certificate
    Properties:
      DomainName:
        Fn::Sub:  # <--- Incorrect type. Expected "string". yaml-schema: aws://sam.schema.json
          - "*.${LowerCaseEnv}.${DomainBase}"
          - LowerCaseEnv:
              Fn::Transform:
                Name: "String"
                Parameters:
                  InputString: !Ref ParamEnvironment
                  Operation: Lower

Resources:
  AWSOpenSearch:
    Type: AWS::OpenSearchService::Domain
    Properties:
      ClusterConfig:
        DedicatedMasterEnabled: false
        InstanceType:
          Fn::If:  # <--- Incorrect type. Expected "string". yaml-schema: aws://sam.schema.json
            - ProdLikeSetup
            - r6g.large.search
            - t3.small.search

Resources:
  UpdateIpoolCustom:
    Type: Custom::UpdateIpool
    DependsOn: AWSOpenSearch  # <--- Property DependsOn is not allowed. yaml-schema: aws://sam.schema.json
    Properties:
      ServiceToken: !GetAtt UpdateIpoolFn.Arn
      IPoolId: !Ref IPoolOpenSearch

Desktop (please complete the following information):
OS: MacOS 12.6 (Monterey)
Visual Studio Code Version: v1.72.0
AWS Toolkit Version: v1.51.0

[zsiv]

One I ran into:

 FunctionName:
    Type: AWS::Serverless::Function
    Properties:
      Events:
        SQSEvent:
          Type: SQS
          Properties:
            Queue: { "Fn::GetAtt" : ["QueueRef" , "Arn"] } # <--- Incorrect type. Expected "string".

Passes 'sam validate' and appears to deploy with the correct ARN for 'Queue'.

Desktop (please complete the following information):

OS: Win 11
Visual Studio Code Version: v1.72.2
AWS Toolkit Version: v1.51.0

[sj-porter-knime]

I just ran into this same issue - number parameter is showing a syntax highlighting error if I set min/max/default to integers. If I convert the values to strings, the resource I'm trying to create complains that it needs a number.

[sj-porter-knime]

It looks like a Developer: Reload Window command fixed it for me, though.

[mattgillard]

I have discovered 3 of these today.

SQSConsumerFunction:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: sqsconsumer/
      Handler: app.lambda_handler
      Runtime: python3.8
      Architectures:
        - x86_64
      Policies:
        - CloudWatchLambdaInsightsExecutionRolePolicy
      VPCConfig: # Property VPCConfig is not allowed.yaml-schema: aws://sam.schema.json
        SubnetIds:
          - !Ref privateSubnet01
          - !Ref privateSubnet02
        SecurityGroupIds:
          - !Ref SQSConsumerSG

ApiGateway:
    Type: AWS::Serverless::Api
    DependsOn: ApiCWLRoleArn
    Properties:
      EndpointConfiguration: 
        Type: PRIVATE
        VPCEndpointIds: # Property VPCEndpointIds is not allowed.yaml-schema: aws://sam.schema.json
          - !Ref APIGWVPCEndpoint
      Name: !Ref apiName
      StageName: "Prod"
      TracingEnabled: True
      DefinitionBody:
        'Fn::Transform':
          Name: 'AWS::Include'
          Parameters:
            Location: './api.yml'

     Auth:
        DefaultAuthorizer: AADAuthoriser
        ResourcePolicy: #Property ResourcePolicy is not allowed.yaml-schema: aws://sam.schema.json
          IntrinsicVpceWhitelist:
            - !Ref APIGWVPCEndpoint

Desktop (please complete the following information):
OS: MacOS 13.1 (Ventura)
Visual Studio Code Version: v1.75.1
AWS Toolkit Version: v1.63.0

[justinmk3]

Thank you for the reports! In order to keep things manageable, it will help to open new issues for each particular missing field.

[justinmk3]

Since AWS Toolkit 1.65 (which includes #3163 ), we now use the full schema provided by the SAM team.

If any of the problems reported here is still an issue, please let us know the specific case (and ideally, in a new issue). Thanks!