Skip to content

Latest commit

 

History

History

data-collection

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
 
 
 
 
 
 
 
 
 
 
 
 

CID Data Collection

About

This projects demonstrates usage of AWS API for collecting various types of usage data.

Architecture

Architecture

  1. Amazon EventBridge rule invokes Step Function of every every deployed data collection module. based on schedule.
  2. The Step Function launches a Lambda function Account Collector that assumes Read Role role in the Management accounts and retrieves linked accounts list via AWS Organizations API
  3. Step Functions launches Data Collection Lambda function for each collected Account.
  4. Each data collection module Lambda function assumes IAM role in linked accounts and retrieves respective optimization data via AWS SDK for Python. Retrieved data aggregated in Amazon S3 bucket
  5. Once data stored in S3 bucket, Step Functions triggers AWS Glue crawler which creates or updates the table in Glue Data Catalog
  6. Collected data visualized with the Cloud Intelligence Dashboards using Amazon QuickSight to get optimization recommendations and insights

Modules

List of modules and objects collected:

Module Name AWS Services Collected In Details
organization AWS Organizations Management Accounts
budgets AWS Budgest Linked Accounts
compute-optimizer AWS Compute Optimizer Management Accounts Requires Enablement of Compute Optimizer
trusted-advisor AWS Trusted Advisor Linked Accounts Requires Enterpriso or OnRamp Support Level
support-cases AWS Support Linked Accounts Requires Business, Enterprise On-Ramp, or Enterprise Support plan
cost-explorer-cost-anomaly AWS Anomalies Management Accounts
cost-explorer-rightsizing AWS Cost Explorer Management Accounts DEPRECATED. Please use Data Exports for Cost Optimization Hub
inventory Various services Linked Accounts Collects Amazon OpenSearch Domains, Amazon ElastiCache Clusters, RDS DB Instances, EBS Volumes, AMI, EC2 Instances, EBS Snapshot, RDS Snapshot, Lambda, RDS DB Clusters, EKS Clusters
pricing Various services Data Collection Account Collects pricing for Amazon RDS, Amazon EC2, Amazon ElastiCache, AWS Lambda, Amazon OpenSearch, AWS Compute Savings Plan
rds-usage Amazon RDS Linked Accounts Collects CloudWatch metrics for chargeback
transit-gateway AWS Transit Gateway Linked Accounts Collects CloudWatch metrics for chargeback
ecs-chargeback Amazon ECS Linked Accounts
backup AWS Backup Management Accounts Collects Backup Restore and Copy Jobs. Requires activation of cross-account
health-events AWS Health Management Accounts Collect AWS Health notificaitons via AWS Organizational view
licence-manager AWS License Manager Management Accounts Collect Licences and Grants
aws-feeds N/A Data Collection Account Collects Blog posts and News Feeds
quicksight Amazon QuickSight Data Collection Account Collects Quicksight User and Group information in the Data Collection Account only

Installation

1. In Management Account(s)

The Management Accounts stack makes use of stack sets configured to use service-managed permissions to deploy stack instances to linked accounts in the AWS Organization.

Before creating the Management Accounts stack, please make sure trusted access with AWS Organizations is activated.

The Management Accounts Stack creates a read role in the Management Accounts and also a StackSet that will deploy another read role in each linked Account. Permissions depend on the set of modules you activate via parameters of the stack:

2. In Data Collection Account

Deploy Data Collection Stack.

Usage

Check Athena tables.

FAQ

Migration from previous Data Collection Lab

See also

CONTRIBUTING.md