Setup Snowplow BDP on AWS

info

This documentation only applies to Snowplow BDP Enterprise. See the feature comparison page for more information about the different Snowplow offerings.

To setup Snowplow, simply follow the 'Getting Started' steps in the Snowplow BDP Console. You will receive an account as part of your onboarding.

What are the steps

The first setup steps are designed to get your infrastructure in place and have you sending data as quickly as possible. The initial steps include:

providing the right cloud environment for Snowplow to be installed
setting up your first sandbox environment
selecting which warehouse, if any, you want to load your data into

Completing the forms for these initial steps should take you around 30 minutes.

What will I need

To setup your cloud environment as required you will need

to be able to set up a sub-account and appropriate permissions on AWS
to know which AWS region you’d like us to install your Snowplow pipeline into
to know whether or not you want VPC peering, and for which /21 or /22 CIDR range (note: VPC peering is an additional bolt-on for basecamp and ascent tiers)

We often find our point of contact requires support from their DevOps or Networking colleagues to complete the cloud setup step; in Snowplow BDP Console you can easily create accounts for colleagues who can complete this step for you.

Preparing your AWS sub-account

These instructions are also provided as part of the setup flow in Snowplow BDP Console.

Create sub-account

From your main AWS account, set up an Organisation if you haven't done so already.
Create a member account (the sub-account) in that organization
Sign out and sign into the new sub-account. Everything Snowplow-related will take place within this account from here in.
Follow these instructions to create a policy using the JSON policy document below.

Set up Role and IAM permissions

Access the IAM control panel within the sub-account
Go to Access management > Roles and select Create role
Select "Another AWS account"
(Account ID: 793733611312 Require MFA: true)
Select the policy you created earlier
Call the role "SnowplowAdmin" or similar

You will need to share this role with us as part of filling out the setup form in Snowplow BDP console.

JSON Policy Document

{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Effect":"Allow",
      "Action":[
        "acm:*",
        "application-autoscaling:*",
        "autoscaling:*",
        "aws-marketplace:Subscribe",
        "aws-marketplace:Unsubscribe",
        "aws-marketplace:ViewSubscriptions",
        "cloudformation:*",
        "cloudfront:*",
        "cloudwatch:*",
        "dynamodb:*",
        "ec2:*",
        "ecs:*",
        "eks:*",
        "elasticloadbalancing:*",
        "elasticmapreduce:*",
        "es:*",
        "execute-api:*",
        "events:*",
        "globalaccelerator:*",
        "iam:*",
        "kinesis:*",
        "kinesisanalytics:*",
        "kms:DescribeKey",
        "kms:List*",
        "lambda:*",
        "logs:*",
        "rds:*",
        "redshift:*",
        "route53:*",
        "route53resolver:*",
        "s3:*",
        "secretsmanager:CreateSecret",
        "secretsmanager:DeleteSecret",
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:PutSecretValue",
        "secretsmanager:TagResource",
        "sns:*",
        "sqs:*",
        "ssm:*",
        "support:*"
      ],
      "Resource":"*"
    }
  ]
}
 

For complete documentation from Amazon go here.

Set up Role and IAM Permissions with CloudFormation

We also provide a CloudFormation template that will create a role named SnowplowAdmin with the full permission set here.

Access the CloudFormation service within the sub-account
Go to Stacks select Create stack > With new resources (standard)
Select Template is ready within the Prepare template block
Specify an Amazon S3 URL with the full path to the SnowplowAdmin CloudFormation template and proceed
Provide the stack with a meaningful name such as SnowplowAdmin stack
Now proceed through the remainder of the prompts and choose Create stack

For complete documentation from Amazon go here.

Setup the Snowplow deployment role

The last step is to set up the Snowplow deployment role. This is a role assumed by the machine user to make changes with Terraform.

Navigate to https://console.aws.amazon.com/iam/home#/roles$new?step=type&roleType=crossAccount
Select Create role and for trusted entity type select AWS account.

Account ID: 793733611312
Do not require MFA, as Snowplow needs to be able to assume the role via headless jobs.

Attach the IAMFullAccess policy. If a Permission Boundary was set on the admin role, then add this boundary to the bottom section of permissions page.

Role name: SnowplowDeployment
Role description: Allows the Snowplow Team to programmatically deploy to this account.

Copy the Snowplow deployment role ARN. You will need to share this role with us as part of filling out the setup form in Snowplow BDP console.

Final checklist

If you are sending a request to our team to set up your account for you. Please ensure you provide the following information:

SnowplowAdmin role ARN
SnowplowDeployment role ARN
AWS region to deploy into
VPC CIDR requirements for VPC Peering (if applicable)
The IAM permission boundary ARN (if applicable)

What are the steps​

What will I need​

Preparing your AWS sub-account​

Create sub-account​

Set up Role and IAM permissions​

JSON Policy Document​

Set up Role and IAM Permissions with CloudFormation​

Setup the Snowplow deployment role​

Final checklist​