This projects demonstrates usage of AWS API for collecting various types of usage data.
- Amazon EventBridge rule invokes Step Function of every every deployed data collection module. based on schedule.
- The Step Function launches a Lambda function Account Collector that assumes Read Role role in the Management accounts and retrieves linked accounts list via AWS Organizations API
- Step Functions launches Data Collection Lambda function for each collected Account.
- Each data collection module Lambda function assumes IAM role in linked accounts and retrieves respective optimization data via AWS SDK for Python. Retrieved data aggregated in Amazon S3 bucket
- Once data stored in S3 bucket, Step Functions triggers AWS Glue crawler which creates or updates the table in Glue Data Catalog
- Collected data visualized with the Cloud Intelligence Dashboards using Amazon QuickSight to get optimization recommendations and insights
List of modules and objects collected:
Module Name | AWS Services | Collected In | Details |
---|---|---|---|
organization |
AWS Organizations | Management Accounts | |
budgets |
AWS Budgest | Linked Accounts | |
compute-optimizer |
AWS Compute Optimizer | Management Accounts | Requires Enablement of Compute Optimizer |
trusted-advisor |
AWS Trusted Advisor | Linked Accounts | Requires Enterpriso or OnRamp Support Level |
support-cases |
AWS Support | Linked Accounts | Requires Business, Enterprise On-Ramp, or Enterprise Support plan |
cost-explorer-cost-anomaly |
AWS Anomalies | Management Accounts | |
cost-explorer-rightsizing |
AWS Cost Explorer | Management Accounts | DEPRECATED. Please use Data Exports for Cost Optimization Hub |
inventory |
Various services | Linked Accounts | Collects Amazon OpenSearch Domains , Amazon ElastiCache Clusters , RDS DB Instances , EBS Volumes , AMI , EC2 Instances , EBS Snapshot , RDS Snapshot , Lambda , RDS DB Clusters , EKS Clusters |
pricing |
Various services | Data Collection Account | Collects pricing for Amazon RDS , Amazon EC2 , Amazon ElastiCache , AWS Lambda , Amazon OpenSearch , AWS Compute Savings Plan |
rds-usage |
Amazon RDS | Linked Accounts | Collects CloudWatch metrics for chargeback |
transit-gateway |
AWS Transit Gateway | Linked Accounts | Collects CloudWatch metrics for chargeback |
ecs-chargeback |
Amazon ECS | Linked Accounts | |
backup |
AWS Backup | Management Accounts | Collects Backup Restore and Copy Jobs. Requires activation of cross-account |
health-events |
AWS Health | Management Accounts | Collect AWS Health notificaitons via AWS Organizational view |
licence-manager |
AWS License Manager | Management Accounts | Collect Licences and Grants |
aws-feeds |
N/A | Data Collection Account | Collects Blog posts and News Feeds |
quicksight |
Amazon QuickSight | Data Collection Account | Collects Quicksight User and Group information in the Data Collection Account only |
The Management Accounts stack makes use of stack sets configured to use service-managed permissions to deploy stack instances to linked accounts in the AWS Organization.
Before creating the Management Accounts stack, please make sure trusted access with AWS Organizations is activated.
The Management Accounts Stack creates a read role in the Management Accounts and also a StackSet that will deploy another read role in each linked Account. Permissions depend on the set of modules you activate via parameters of the stack:
Deploy Data Collection Stack.
Check Athena tables.