Ingesting Web Application Click Logs into AWS (by HashiCorp)
Threat Model Architecture Representation
Let's use an Official example for Ingesting Web Application Click Logs:
This post provides an API based ingestion application system for websites & applications to push user interactions, click actions from their website into AWS. The ingestion process will be exposed using a web-based interaction with an API Gateway endpoint.
This example gives the source code for the following AWS architecture diagram:
The slp_tfplan processor can generate an accurate threat model by only following the steps below.
Generating my own Terraform files
Clone and prepare the example reporitory:
These previous set-up commands are required because the example needs some lambda code to be compiled.
$ git clone https://github.com/aws-samples/aws-ingesting-click-logs-using-terraform.git
$ cd aws-ingesting-click-logs-using-terraform/source/clicklogger
$ mvn clean package
$ cd ../../terraform/templates
Generate the tf-plan and tf-graph files:
Use the StartLeft CLI to generate the OTM file:
$ startleft parse \
--iac-type TFPLAN \
--default-mapping-file ir-mappings.yaml \
--output-file output.otm \
--project-id "my-project" \
--project-name "My project" \
tf-plan.json tf-graph.gv
After, we need to use the Terraform CLI to create the required files and execute StartLeft parse to generate the final OTM.
trustzones:
- type: b61d6911-338d-46a8-9f39-8dcd24abfe91
name: Public Cloud
risk:
trust_rating: 10
$default: true
- type: f0ba7722-39b6-4c81-8290-a30a248bb8d9
name: Internet
risk:
trust_rating: 1
components:
- label: aws_acm_certificate
type: CD-ACM
$singleton: true
- label: aws_cloudwatch_metric_alarm
type: cloudwatch
$singleton: true
- label: aws_dynamodb_table
type: dynamodb
- label: aws_vpc
type: vpc
- label: aws_instance
type: ec2
- label: aws_subnet
type: empty-component
- label: aws_vpc_endpoint
type: empty-component
- label: aws_internet_gateway
type: empty-component
- label: aws_ecs_service
type: elastic-container-service
- label: aws_ecs_task_definition
type: docker-container
- label: ["aws_lb", "aws_elb", "aws_alb"]
type: load-balancer
- label: aws_kms_key
type: kms
$singleton: true
- label: aws_lambda_function
type: aws-lambda-function
- label: aws_cloudwatch_log_group
type: cloudwatch
$singleton: true
- label: ["aws_db_instance", "aws_rds_cluster"]
type: rds
- label: aws_route53_zone
type: route-53
- label: aws_autoscaling_group
type: CD-EC2-AUTO-SCALING
- label: cloudflare_record
type: empty-component
- label: [aws_s3_bucket, aws_s3_bucket_object]
type: s3
- label: aws_secretsmanager_secret
type: CD-SECRETS-MANAGER
$singleton: true
- label: aws_sqs_queue
type: sqs-simple-queue-service
- label: {$regex: ^aws_ssm_\w*$}
type: CD-SYSTEMS-MANAGER
$singleton: true
- label: aws_synthetics_canary
type: empty-component
- label: {$regex: ^aws_api_gateway_\w*$}
type: api-gateway
$singleton: true
- label: {$regex: ^aws_athena_\w*$}
type: athena
$singleton: true
- label: {$regex: ^aws_mq_\w*$}
type: CD-MQ
$singleton: true
- label: {$regex: ^aws_cloudfront_\w*$}
type: cf-cloudfront
$singleton: true
- label: aws_cloudtrail
type: cloudtrail
- label: ["aws_cognito_user_pool", "aws_cognito_identity_pool"]
type: cognito
- label: {$regex: ^aws_config_\w*$}
type: CD-CONFIG
$singleton: true
- label: {$regex: ^aws_ecr_\w*$}
type: elastic-container-registry
$singleton: true
- label: aws_eks_cluster
type: elastic-container-kubernetes
- label: {$regex: ^aws_elasticache_\w*$}
type: elasticache
$singleton: true
- label: {$regex: ^aws_guardduty_\w*$}
type: CD-GUARDDUTY
$singleton: true
- label: {$regex: ^aws_inspector_\w*$}
type: CD-INSPECTOR
$singleton: true
- label: {$regex: ^aws_macie2_\w*$}
type: CD-MACIE
$singleton: true
- label: aws_networkfirewall_firewall
type: CD-AWS-NETWORK-FIREWALL
- label: aws_redshift_cluster
type: redshift
- label: {$regex: ^aws_ses_\w*$}
type: CD-SES
$singleton: true
- label: {$regex: ^aws_sns_\w*$}
type: sns
$singleton: true
- label: {$regex: ^aws_sfn_\w*$}
type: step-functions
- label: {$regex: ^aws_waf(.*)_\w*$}
type: CD-WAF
$singleton: true
- label: {$regex: ^aws_kinesis_analytics_\w*$}
type: kinesis-data-analytics
$singleton: true
- label: {$regex: ^aws_kinesis_stream\w*$}
type: kinesis-data-analytics
$singleton: true
- label: {$regex: ^aws_kinesis_firehose_\w*$}
type: kinesis-data-firehose
$singleton: true
- label: {$regex: ^aws_iam_\w*$}
type: CD-AWS-IAM
$singleton: true
- label: {$regex: ^aws_cloudwatch_event_\w*$}
type: eventbridge
$singleton: true
- label: {$regex: ^aws_codebuild_\w*$}
type: CD-CODEBUILD
$singleton: true
- label: {$regex: ^aws_codepipeline\w*$}
type: CD-CODEPIPELINE
$singleton: true
- label: aws_ebs_volume
type: elastic-block-store
- label: {$regex: ^aws_shield_\w*$}
type: CD-SHIELD
$singleton: true
- label: {$regex: ^aws_cloudformation_\w*$}
type: CD-CLOUDFORMATION
$singleton: true
- label: aws_glue_job
type: CD-GLUE
- label: aws_glue_registry
type: CD-GLUE-SCHEMA-REGISTRY
- label: aws_efs_file_system
type: elastic-file-system
- label: aws_transfer_server
type: CD-TRANSFER-FML
- label: aws_codecommit_repository
type: CD-CODECOMMIT
- label: aws_globalaccelerator_accelerator
type: CD-GLOBAL-ACC
- label: {$regex: ^aws_dms_\w*$}
type: CD-DMS
$singleton: true
- label: {$regex: ^aws_iot_\w*$}
type: CD-IOT-CORE
$singleton: true
- label: {$regex: ^aws_medialive_\w*$}
type: CD-MEDIALIVE
$singleton: true
- label: {$regex: ^aws_gamelift_\w*$}
type: CD-GAMELIFT
$singleton: true
- label: {$regex: ^aws_directory_service_\w*$}
type: CD-DIR-SERVICE
$singleton: true
- label: {$regex: ^aws_appsync_\w*$}
type: CD-APPSYNC
$singleton: true
- label: {$regex: ^aws_fms_\w*$}
type: firewall-manager
$singleton: true
- label: aws_neptune_cluster
type: CD-NEPTUNE
- label: aws_ec2_transit_gateway
type: CD-AWS-TRANSIT-GW
- label: {$regex: ^aws_batch_\w*$}
type: CD-BATCH
$singleton: true
- label: aws_elastic_beanstalk_application
type: CD-ELASTIC-BEANSTALK
- label: {$regex: ^aws_dx_\w*$}
type: direct-connect
$singleton: true
- label: aws_emr_cluster
type: CD-EMR
- label: aws_msk_cluster
type: CD-MSK
- label: aws_elastictranscoder_pipeline
type: CD-ELASTIC-TRANSCODER
- label: aws_sagemaker_app
type: CD-SAGEMAKER
# AZURE
- label: [ "azurerm_data_share", "azurerm_data_share_account" ]
type: CD-MICROSOFT-AZURE-DATA-SHARE
- label: azurerm_elastic_cloud_elasticsearch
type: CD-MICROSOFT-AZURE-ELASTICSEARCH
- label: [ "azurerm_media_services_account", "azurerm_media_services_account_filter" ]
type: CD-MICROSOFT-AZURE-MEDIA-SERVICES
configuration:
attack_surface:
client: generic-client
trustzone: f0ba7722-39b6-4c81-8290-a30a248bb8d9
# skip:
# - aws_security_group
# - aws_db_subnet_group
# catch_all: empty-component
otmVersion: 0.1.0
project:
name: name
id: id
representations:
- name: Terraform
id: Terraform
type: code
trustZones:
- id: b61d6911-338d-46a8-9f39-8dcd24abfe91
name: Public Cloud
risk:
trustRating: 10
components:
- id: aws_dynamodb_table.click-logger-table
name: click-logger-table
type: dynamodb
parent:
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91
tags:
- aws_dynamodb_table
- id: aws_kinesis_firehose_delivery_stream.click_logger_firehose_delivery_stream
name: click_logger_firehose_delivery_stream
type: kinesis-data-firehose
parent:
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91
tags:
- aws_kinesis_firehose_delivery_stream
- id: aws_lambda_function.lambda_clicklogger
name: lambda_clicklogger
type: aws-lambda-function
parent:
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91
tags:
- aws_lambda_function
- id: aws_lambda_function.lambda_clicklogger_authorizer
name: lambda_clicklogger_authorizer
type: aws-lambda-function
parent:
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91
tags:
- aws_lambda_function
- id: aws_lambda_function.lambda_clicklogger_stream_consumer
name: lambda_clicklogger_stream_consumer
type: aws-lambda-function
parent:
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91
tags:
- aws_lambda_function
- id: aws_s3_bucket.click_logger_firehose_delivery_s3_bucket
name: click_logger_firehose_delivery_s3_bucket
type: s3
parent:
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91
tags:
- aws_s3_bucket
- id: aws_api_gateway_account.click_logger_api_gateway_account
name: api-gateway (grouped)
type: api-gateway
parent:
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91
tags:
- aws_api_gateway_method_response
- aws_api_gateway_method_settings
- aws_api_gateway_integration_response
- aws_api_gateway_method
- aws_api_gateway_model
- aws_api_gateway_request_validator
- aws_api_gateway_integration
- aws_api_gateway_account
- aws_api_gateway_resource
- aws_api_gateway_rest_api
- aws_api_gateway_authorizer
- aws_api_gateway_deployment
- id: aws_cloudwatch_log_group.click_logger_firehose_delivery_stream_log_group
name: cloudwatch (grouped)
type: cloudwatch
parent:
trustZone: b61d6911-338d-46a8-9f39-8dcd24abfe91
tags:
- aws_cloudwatch_log_group
dataflows:
- id: cb8953f8-1436-47a6-8939-ffcd07902614
name: clicklogger-authorizer to lambda_clicklogger_authorizer
source: aws_api_gateway_account.click_logger_api_gateway_account
destination: aws_lambda_function.lambda_clicklogger_authorizer
bidirectional: false
- id: 4f25e0b3-0b8a-4e84-90db-f0a8de34467e
name: integration to lambda_clicklogger
source: aws_api_gateway_account.click_logger_api_gateway_account
destination: aws_lambda_function.lambda_clicklogger
bidirectional: false
- id: 66cecdff-ba09-42b1-a75f-8e98fdae2dc4
name: lambda_click_logger_authorizer_log_group to lambda_clicklogger_authorizer
source: aws_cloudwatch_log_group.click_logger_firehose_delivery_stream_log_group
destination: aws_lambda_function.lambda_clicklogger_authorizer
bidirectional: false
- id: e01769b2-d461-4c4d-b7aa-cc028f4533ab
name: lambda_click_logger_log_group to lambda_clicklogger
source: aws_cloudwatch_log_group.click_logger_firehose_delivery_stream_log_group
destination: aws_lambda_function.lambda_clicklogger
bidirectional: false
- id: 9618b94c-b899-45d0-ba70-61d141e2ad54
name: click_logger_firehose_delivery_stream to lambda_clicklogger_stream_consumer
source: aws_kinesis_firehose_delivery_stream.click_logger_firehose_delivery_stream
destination: aws_lambda_function.lambda_clicklogger_stream_consumer
bidirectional: false
- id: e4431bbb-a8d5-413e-be2b-8c5b0cc06d81
name: click_logger_firehose_delivery_stream to
click_logger_firehose_delivery_s3_bucket
source: aws_kinesis_firehose_delivery_stream.click_logger_firehose_delivery_stream
destination: aws_s3_bucket.click_logger_firehose_delivery_s3_bucket
bidirectional: false
- id: 3c58eb54-f1eb-4762-9f2b-dd88793032e3
name: lambda_clicklogger to click_logger_firehose_delivery_stream
source: aws_lambda_function.lambda_clicklogger
destination: aws_kinesis_firehose_delivery_stream.click_logger_firehose_delivery_stream
bidirectional: false
- id: 50058e60-209a-4a2f-99b3-066bb633e2eb
name: lambda_clicklogger_stream_consumer to click-logger-table
source: aws_lambda_function.lambda_clicklogger_stream_consumer
destination: aws_dynamodb_table.click-logger-table
bidirectional: false
As we can see in the previous example, StartLeft can automatically reproduce an almost exact architecture diagram than the one present in the example website.
Full Architecture Representation
This approach enables StartLeft not only to create threat models but also to explore all the components and the dataflows among them, by configuring this behavior in the mapping file. In that way, you can get this diagram from the example above:
