Setup Guide for AWS
Overviewβ
Snowplow Mini is, in essence, the Snowplow real time stack inside of a single image. It is an easily-deployable, single instance version of Snowplow that serves three use cases:
- Giving a Snowplow consumer (e.g. an analyst / data team / marketing team) a way to quickly understand what Snowplow "does" i.e. what you put it at one end and take out of the other
- Giving developers new to Snowplow an easy way to start with Snowplow and understand how the different pieces fit together
- Giving people running Snowplow a quick way to debug tracker updates
All setup for Snowplow Mini is done within the AWS Console and will incur small amounts of running costs, depending on the size of the EC2 instance you select.
We offer Snowplow Mini in 3 different sizes. To decide on which size of Snowplow Mini to choose, read on.
large & xlarge & xxlargeβ
Mini is available in 3 different sizes:
large
: Opensearch has4g
heap size and Snowplow apps has0.5g
heap size. Recommended machine RAM is8g
.xlarge
: Double the large image. Opensearch has8g
heap size and Snowplow apps has1.5g
heap size. Recommended machine RAM is16g
.xxlarge
: Double the xlarge image. Opensearch has16g
heap size and Snowplow apps has3g
heap size. Recommended machine RAM is32g
.
This service is available as an EC2 image within the AWS Community AMIs in the following regions: ap-northeast-1
, ap-northeast-2
, ap-south-1
, ap-southeast-1
, ap-southeast-2
, ca-central-1
, eu-central-1
, eu-west-1
, eu-west-2
, sa-east-1
, us-east-1
, us-east-2
, us-west-1
and us-west-2
.
Version 0.17.0 (recommended) comes with:
- Snowplow Stream Collector NSQ 2.9.0
- Snowplow Stream Enrich NSQ 3.8.0
- Snowplow Elasticsearch Loader 2.0.8
- Snowplow Iglu Server 0.10.0
- Opensearch 2.4.0
- Opensearch Dashboards 2.4.0
- Postgresql 15.1
- NSQ v1.2.1
Note: All services are configured to start automatically so everything should happily survive restarts/shutdowns.
To understand the flow of data please refer to the following diagram:
IAM
Create a role with the following configuration
- Step 1: For
Select type of trusted entity
, selectEC2
- Step 2.1: For
Attach permissions policies
, create a policy with the following
{
"Version" : "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": ["*"]
}
]
}
- Step 2.2: In step 2 of role creation, select the policy created in the previous step
- Step 3: Tags are optional
- Step 4: Fill in the role name and create it.
CloudWatch
Create a log group named snowplow-mini
so that Mini can emit logs to this log group.
Mini will not function properly if a log group with that name isn't found.
Security Groupβ
In the EC2 Console UI select Security Groups
from the panel on the left.
Select the Create Security Group
button and fill in the name, description and what VPC you want to attach it to.
You will then need to add the following InBound rules:
- Custom TCP Rule | Port Range (80)
- CIDR range
0.0.0.0/0
- Custom TCP Rule | Port Range (443)
- CIDR range
0.0.0.0/0
- SSH (optional):
- Custom TCP Rule | Port Range (22)
- CIDR range
{{ YOUR IP HERE }}/32
For OutBound you can leave the default to allow everything out.
Choose AMIβ
In the EC2 Console UI select the Launch Instance
button then select the Community AMIs
button. In the search bar enter snowplow-mini-0.17.0
to find the needed AMI and then select it.
Choose Instance Typeβ
AMI names explicitly specify which instance type to use.
0.17.0-large
needst2.large
0.17.0-xlarge
needst2.xlarge
0.17.0-xxlarge
needst2.2xlarge
Configure Instanceβ
Select the IAM role created above.
If you created your Security Group in a different VPC than the default you will need to select the same VPC in the Network field.
NOTE: If you select a custom VPC ensure that you select Enable
for the Auto-assign Public IP option.
Add Storageβ
Depending on how long you intend to run Snowplow Mini and how much data you intend to send/store you will need to change the size of the block store accordingly.
For basic testing and debugging;
- 20-50 Gb should suffice for
large
- 50-100 Gb should suffice for
xlarge
- 100-200 Gb should suffice for
xxlarge
We also recommend changing the Volume Type
to GP2 from Magnetic for a smoother experience.
Tag Instanceβ
Add any tags you like here.
Configure Security Groupβ
Select the Security Group you created above.
Reviewβ
Press the Launch
button and select an existing key-pair, or create a new one, if you want to be able to SSH into the box.