Web
info
Some variables are only available in the latest version of our package, or might have changed format from older versions. If you are unable to use the latest version, check the dbt_project.yml file of our package for the version you are using to see what options are available to you.
Package Configuration Variables
This package utilizes a set of variables that are configured to recommended values for optimal performance of the models. Depending on your use case, you might want to override these values by adding to your dbt_project.yml file.
note
All variables in Snowplow packages start with snowplow__ but we have removed these in the below tables for brevity.
Warehouse and tracker
| Variable Name | Description | Default |
|---|---|---|
atomic_schema | The schema (dataset for BigQuery) that contains your atomic events table. | atomic |
database | The database that contains your atomic events table. | target.database |
dev_target_name | The target name of your development environment as defined in your profiles.yml file. See the Manifest Tables section for more details. | dev |
events | This is used internally by the packages to reference your events table based on other variable values and should not be changed. | events |
events_table | The name of the table that contains your atomic events. | events |
ga4_categories_seed | Name of the model for the GA4 category mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | snowplow_web_dim_ga4_source_categories |
geo_mapping_seed | Name of the model for the Geo mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | snowplow_web_dim_geo_country_mapping |
heartbeat | Page ping heartbeat time as defined in your tracker configuration. | 10 |
min_visit_length | Minimum visit length as defined in your tracker configuration. | 5 |
rfc_5646_seed | Name of the model for the RFC 5646 (language) mapping seed table, either a seed or a model (if you want to use a source, create a model to select from it). | snowplow_web_dim_rfc_5646_language_mapping |
sessions_table | The users module requires data from the derived sessions table. If you choose to disable the standard sessions table in favor of your own custom table, set this to reference your new table e.g. {{ ref('snowplow_web_sessions_custom') }}. Please see the README in the custom_example directory for more information on this sort of implementation. | "{{ ref( 'snowplow_web_sessions' ) }}" |
Operation and logic
| Variable Name | Description | Default |
|---|---|---|
allow_refresh | Used as the default value to return from the allow_refresh() macro. This macro determines whether the manifest tables can be refreshed or not, depending on your environment. See the Manifest Tables section for more details. | false |
backfill_limit_days | The maximum numbers of days of new data to be processed since the latest event processed. Please refer to the incremental logic section for more details. | 30 |
conversion_events | (Version 0.15.0+) A list of dictionaries that define a conversion event for your modeling, to add the relevant columns to the sessions table. The dictionary keys are name (required), condition (required), value, default_value, and list_events. For more information see the package documentation. | |
cwv_days_to_measure | The number of days to use for web vital measurements (if enabled). | 28 |
cwv_percentile | The percentile that the web vitals measurements that are produced for all page views (if enabled). | 75 |
days_late_allowed | The maximum allowed number of days between the event creation and it being sent to the collector. Exists to reduce lengthy table scans that can occur as a result of late arriving data. | 3 |
limit_page_views_to_session | A boolean whether to ensure page view aggregations are limited to pings in the same session as the page_view event, to ensure deterministic behavior. If false you may get different results for the same page_view depending on which sessions are included in a run. See the stray page ping section for more information. | true |
list_event_counts | A boolean whether to include a json-type (varies by warehouse) column in the sessions table with a count of events for each event_type in that session. | false |
lookback_window_hours | The number of hours to look before the latest event processed - to account for late arriving data, which comes out of order. | 6 |
max_session_days | The maximum allowed session length in days. For a session exceeding this length, all events after this limit will stop being processed. Exists to reduce lengthy table scans that can occur due to long sessions which are usually a result of bots. | 3 |
page_view_stitching | Determines whether to apply the user mapping to the page views table. Note this can be an expensive operation to do every run. One way to mitigate this is by running this update with less frequency than your usual run by enabling this variable only for that specific run. Please see the User Mapping section for more details. | false |
session_identifiers | A list of key:value dictionaries which contain all of the contexts and fields where your session identifiers are located. For each entry in the list, if your map contains the schema value atomic, then this refers to a field found directly in the atomic events table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the schema field. It will use the specified value in the field key as the field name to access. For Redshift/Postgres, using the schema key the package will try to find a table in your snowplow__events_schema schema with the same name as the schema value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the Utils advanced operation section for more details. | [{"schema" : "atomic", "field" : "domain_sessionid"}] |
session_lookback_days | Number of days to limit scan on snowplow_web_base_sessions_lifecycle_manifest manifest. Exists to improve performance of model when we have a lot of sessions. Should be set to as large a number as practical. | 730 |
session_stitching | Determines whether to apply the user mapping to the sessions table. Please see the User Mapping section for more details. | true |
session_sql | This allows you to override the session_identifiers SQL, to define completely custom SQL in order to build out a session identifier for your events. If you are interested in using this instead of providing identifiers through the session_identifiers variable, please see the Utils advanced operation section for more details on how to do that. | |
session_timestamp | Determines which timestamp is used to build the sessionization logic. It's a good idea to have this timestamp be the same timestamp as the field you partition your events table on. | collector_tstamp |
start_date | The date to start processing events from in the package on first run or a full refresh, based on collector_tstamp. | '2020-01-01' |
total_all_conversions | A boolean flag whether to calculate and add the cv__all_volume and cv__all_total columns. For more information see the package documentation. | false |
upsert_lookback_days | Number of days to look back over the incremental derived tables during the upsert. Where performance is not a concern, should be set to as long a value as possible. Having too short a period can result in duplicates. Please see the Snowplow Optimized Materialization section for more details. | 30 |
user_identifiers | A list of key:value dictionaries which contain all of the contexts and fields where your user identifiers are located. For each entry in the list, if your map contains the schema value atomic, then this refers to a field found directly in the atomic events table. If you are trying to introduce a context/entity with an identifier in it, the package will look for the context in your events table with the name specified in the schema field. It will use the specified value in the field key as the field name to access. For Redshift/Postgres, using the schema key the package will try to find a table in your snowplow__events_schema schema with the same name as the schema value provided, and join that. If multiple fields are specified, the package will try to coalesce all fields in the order specified in the list. For a better understanding of the advanced usage of this variable, please see the Utils advanced operation section for more details. | [{"schema" : "atomic", "field" : "domain_userid"}] |
user_sql | This allows you to override the user_identifiers SQL, to define completely custom SQL in order to build out a user identifier for your events. If you are interested in using this instead of providing identifiers through the user_identifiers variable, please see the Utils advanced operation section for more details on how to do that. | |
user_stitching_id | This is the user_id you want to stitch to sessions (and/or page views) with matching domain_userids. It supports raw sql expressions. | user_id |
tip
When modifying the session/user_identifiers or using session/user_sql in the web package these will overwrite the domain_sessionid and domain_userid fields in tables, rather than being session/user_identifier as in the core utils implementation. This is for historic reasons to mitigate breaking changes. Original values for these fields can be found in original_domain_session/userid in each table.
Contexts, filters, and logs
| Variable Name | Description | Default |
|---|---|---|
app_id | A list of app_ids to filter the events table on for processing within the package. | [ ] (no filter applied) |
enable_consent | Flag to enable the consent module. | false |
enable_cwv | Flag to enable the Core Web Vitals module. | false |
enable_iab | Flag to include the IAB enrichment data in the models. | false |
enable_ua | Flag to include the UA Parser enrichment data in the models. | false |
enable_yauaa | Flag to include the YAUAA enrichment data in the models. | false |
has_log_enabled | When executed, the package logs information about the current run to the CLI. This can be disabled by setting to false. | true |
page_view_passthroughs | Field(s) to carry through from the events table to the derived table. The field is from the page_view event record. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys sql for the SQL code to select the column and alias for the alias of the column in the output. | [] (no passthroughs) |
session_passthroughs | Field(s) to carry through from the events table to the derived table. The field is based on the first page_view or page_ping event for that session. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys sql for the SQL code to select the column and alias for the alias of the column in the output. | [] (no passthroughs) |
ua_bot_filter | Flag to filter out bots via the useragent string pattern match. | true |
user_first_passthroughs | Field(s) to carry through from the events table to the derived table. The field is based on the first session record for that user. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys sql for the SQL code to select the column and alias for the alias of the column in the output. | [] (no passthroughs) |
user_last_passthroughs | Field(s) to carry through from the events table to the derived table. The field is based on the last session record for that user. Aggregation is not supported. A list of either flat column names from the events table or a dictionary with the keys sql for the SQL code to select the column and alias for the alias of the column in the output. Note flat fields will be aliased with a last_ prefix, dictionary provided aliases will not by default. | [] (no passthroughs) |
Warehouse Specific
- Databricks
- Redshift & Postgres
- Bigquery
- Snowflake
| Variable Name | Description | Default |
|---|---|---|
databricks_catalog | The catalogue your atomic events table is in. Depending on the use case it should either be the catalog (for Unity Catalog users from databricks connector 1.1.1 onwards, defaulted to hive_metastore) or the same value as your snowplow__atomic_schema (unless changed it should be 'atomic'). | hive_metastore |
Redshift and Postgres use a shredded approach for the context tables, so these variables are used to identify where they are, if different from the expected schema and table name. They must be passed in a stringified source function as the defaults below show.
| Variable Name | Default |
|---|---|
page_view_context | com_snowplowanalytics_snowplow_web_page_1 |
iab_context | com_iab_snowplow_spiders_and_robots_1 |
ua_parser_context | com_snowplowanalytics_snowplow_ua_parser_context_1 |
yauaa_context | nl_basjes_yauaa_context_1 |
consent_cmp_visible | com_snowplowanalytics_snowplow_cmp_visible_1 |
consent_preferences | com_snowplowanalytics_snowplow_consent_preferences_1 |
| Variable Name | Description | Default |
|---|---|---|
enable_load_tstamp | Flag to include the load_tstamp column in the base events this run model. This should be set to true (the default) unless you are using the Postgres loader or an RDB loader version less than 4.0.0. It must be true to use consent models on Postgres and Redshift. | true |
entities_or_sdes | A list of dictionaries defining the context or self-describing event tables to join onto your base events table. Please use the tool below or see the section on Utilizing custom contexts or SDEs for details of the structure | [] |
| Variable Name | Description | Default |
|---|---|---|
derived_tstamp_partitioned | Boolean to enable filtering the events table on derived_tstamp in addition to collector_tstamp. | true |
| Variable Name | Description | Default |
|---|---|---|
query_tag | This sets the value of the query_tag for all sql executed against the database. This is used internally for metric gathering in Snowflake and its value should not be changed. | snowplow_dbt |
Output Schemas
By default all scratch/staging tables will be created in the <target.schema>_scratch schema, the derived tables, will be created in <target.schema>_derived and all manifest tables in <target.schema>_snowplow_manifest. Some of these schemas are only used by specific packages, ensure you add the correct configurations for each packages you are using. To change, please add the following to your dbt_project.yml file:
tip
If you want to use just your connection schema with no suffixes, set the +schema: values to null
models:
snowplow_web:
base:
manifest:
+schema: my_manifest_schema
scratch:
+schema: my_scratch_schema
sessions:
+schema: my_derived_schema
scratch:
+schema: my_scratch_schema
user_mapping:
+schema: my_derived_schema
users:
+schema: my_derived_schema
scratch:
+schema: my_scratch_schema
page_views:
+schema: my_derived_schema
scratch:
+schema: my_scratch_schema
Config Generator
You can use the below inputs to generate the code that you need to place into your dbt_project.yml file to configure the package as you require. Any values not specified will use their default values from the package.
Project Variables:
vars:
snowplow_web: null