The target-bigquery loader sends data into Google BigQuery after it was pulled from a source using an extractor
Alternate Implementations
Getting Started
Prerequisites
If you haven't already, follow the initial steps of the Getting Started guide:
Then, follow the steps in the "Activate the Google BigQuery API" section of the repository's README.
Installation and configuration
-
Add the target-bigquery loader to your
project using
:meltano add -
Configure the target-bigquery
settings using
:meltano config
meltano add loader target-bigquery --variant adswervemeltano config target-bigquery set --interactiveNext steps
Follow the remaining steps of the Getting Started guide:
If you run into any issues, learn how to get help.
Capabilities
This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!Settings
The
target-bigquery settings that are known to Meltano are documented below. To quickly
find the setting you're looking for, click on any setting name from the list:
project_iddataset_idlocationcredentials_pathvalidate_recordsadd_metadata_columnsreplication_methodtable_prefixtable_suffixmax_cachemerge_state_messagestable_config
You can also list these settings using
with the meltano configlist
subcommand:
meltano config target-bigquery list
You can
override these settings or specify additional ones
in your meltano.yml by adding the settings key.
Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.
Project ID (project_id)
-
Environment variable:
TARGET_BIGQUERY_PROJECT_ID
BigQuery project
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set project_id [value]Dataset ID (dataset_id)
-
Environment variable:
TARGET_BIGQUERY_DATASET_ID -
Default Value:
$MELTANO_EXTRACT__LOAD_SCHEMA
BigQuery dataset.
The default value will expand to the value of the load_schema extra for the extractor used in the pipeline, which defaults to the extractor's namespace, e.g. tap_gitlab for tap-gitlab.
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set dataset_id [value]Region (location)
-
Environment variable:
TARGET_BIGQUERY_LOCATION -
Default Value:
us-central1
Dataset Location. See https://cloud.google.com/bigquery/docs/locations.
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set location [value]Credentials Path (credentials_path)
-
Environment variable:
TARGET_BIGQUERY_CREDENTIALS_PATH -
Default Value:
$MELTANO_PROJECT_ROOT/client_secrets.json
Fully qualified path to client_secrets.json for your service account.
See the "Activate the Google BigQuery API" section of the repository's README and https://cloud.google.com/docs/authentication/production.
By default, this file is expected to be at the root of your project directory.
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set credentials_path [value]Validate Records (validate_records)
-
Environment variable:
TARGET_BIGQUERY_VALIDATE_RECORDS -
Default Value:
false
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set validate_records [value]Add Metadata Columns (add_metadata_columns)
-
Environment variable:
TARGET_BIGQUERY_ADD_METADATA_COLUMNS -
Default Value:
false
Add _time_extracted and _time_loaded metadata columns
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set add_metadata_columns [value]Replication Method (replication_method)
-
Environment variable:
TARGET_BIGQUERY_REPLICATION_METHOD -
Default Value:
append
The Replication method can be:
append: Adding new rows to the table (Default value)
truncate: Deleting all previous rows and uploading the new ones to the table
incremental: Upserting new rows into the table, using the primary key given by the tap connector (if it finds an old row with same key, updates it. Otherwise it inserts the new row)
WARNING: It is not recommended to use the incremental option (which uses MERGE SQL statement). It might result in loss of production data, because historical records get updated. Instead, we recommend using the append replication method, which will preserve historical data.
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set replication_method [value]Table Prefix (table_prefix)
-
Environment variable:
TARGET_BIGQUERY_TABLE_PREFIX
Add prefix to table name
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set table_prefix [value]Table Suffix (table_suffix)
-
Environment variable:
TARGET_BIGQUERY_TABLE_SUFFIX
Add suffix to table name
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set table_suffix [value]Max Cache (max_cache)
-
Environment variable:
TARGET_BIGQUERY_MAX_CACHE -
Default Value:
50
Maximum cache size in MB
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set max_cache [value]Merge State Messages (merge_state_messages)
-
Environment variable:
TARGET_BIGQUERY_MERGE_STATE_MESSAGES -
Default Value:
false
Whether to merge multiple state messages from the tap into the state file or uses the last state message as the state file. Note that it is not recommended to set this to true when using with Meltano as the merge behavior conflicts with Meltano’s merge process.
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set merge_state_messages [value]Table Config (table_config)
-
Environment variable:
TARGET_BIGQUERY_TABLE_CONFIG
A path to a file containing the definition of partitioning and clustering.
Configure this setting directly using the following Meltano command:
meltano config target-bigquery set table_config [value]Something missing?
This page is generated from a YAML file that you can contribute changes to.
Edit it on GitHub!Looking for help?
#plugins-general-brightgreen){kind=icon}
-19,200-c0c0c4){kind=icon}
-44-c0c0c4){kind=icon}
