S3 loader configuration reference

This is a complete list of the options that can be configured in the S3 loader HOCON config file. The example configs in github show how to prepare an input file.

parameter	description
purpose	Required. Use RAW to sink data exactly as-is. Use ENRICHED_EVENTS to also enable event latency metrics. Use SELF_DESCRIBING to enable partitioning self-describing data by its schema
input.appName	Required. Kinesis Client Lib app name (corresponds to DynamoDB table name)
input.streamName	Required. Name of the kinesis stream from which to read
input.position	Required. Use TRIM_HORIZON to start streaming at the last untrimmed record in the shard, which is the oldest data record in the shard. Or use LATEST to start streaming just after the most recent record in the shard
input.customEndpoint	Optional. Override the default endpoint for kinesis client api calls
input.maxRecords	Required. How many records the client should pull from kinesis each time
output.s3.path	Required. Full path to output data, e.g. s3://acme-snowplow-output/raw/
output.s3.partitionFormat	Optional. Added in version 2.1.0. Configures how files are partitioned into S3 directories.When loading raw files, you might choose to partition by date={yy}-{mm}-{dd}. When loading self describing jsons, you might choose to partition by {vendor}.{name}/model={model}/date={yy}-{mm}-{dd}. Valid substitutions are {vendor}, {name}, {format}, {model} for self-describing jsons; and {yy}, {mm}, {dd}, {hh} for year, month, day and hour. Defaults to {vendor}.{schema} when loading self-describing JSONs, or blank (no partitioning) when loading raw or enriched events
output.s3.filenamePrefix	Optional. Adds a prefix to output
output.s3.compression	Required. Either LZO or GZIP
output.s3.maxTimeout	Required. Maximum Timeout that the application is allowed to fail for, e.g. in case of S3 outage
output.s3.customEndpoint	Optional. Override the default endpoint for s3 client api calls
region	Optional. When used with the output.s3.customEndpoint option, this sets the region of the bucket. Also sets the region of the dynamoDB table. Defaults to the current region
output.bad.streamName	Required. Name of a kinesis stream to output failures
buffer.byteLimit	Required. Maximum bytes to read from kinesis before flushing a file to S3
buffer.recordLimit	Required. Maximum records to read from kinesis before flushing a file to S3
buffer.timeLimit	Required. Maximum time to wait in milliseconds between writing files to S3
monitoring.snowplow.collector	Optional. E.g. https://snplow.acme.com. URI of a snowplow collector. Used for monitoring application lifecycle and failure events
monitoring.snowplow.appId	Required only if the collector uri is also configured. Sets the appId field of the snowplow events
monitoring.sentry.dsn	Optional, for tracking uncaught run time exceptions
monitoring.metrics.cloudwatch	Optional boolean, with default true. This is used to disable sending metrics to cloudwatch
monitoring.metrics.hostname	Optional, for sending loading metrics (latency and event counts) to a statsd server
monitoring.metrics.port	Optional, port of the statsd server
monitoring.metrics.tags	E.g.{ "key1": "value1", "key2": "value2" }. Tags are used to annotate the statsd metric with any contextual information
monitoring.metrics.prefix	Optional, default snoplow.s3loader. Configures the prefix of statsd metric names