S3 CSV

target-s3-csv (transferwise variant)🥈

AWS S3 - CSV File Format

The target-s3-csv loader sends data into S3 CSV after it was pulled from a source using an extractor

Getting Started

Prerequisites

If you haven't already, follow the initial steps of the Getting Started guide:

  1. Install Meltano
  2. Create your Meltano project

Installation and configuration

  1. Add the target-s3-csv loader to your project using 
    meltano add
     : 
    2. meltano add loader target-s3-csv
  2. Configure the target-s3-csv settings using 
    meltano config
     : 
    3. meltano config target-s3-csv set --interactive

Next steps

If you run into any issues, learn how to get help.

Capabilities

This plugin currently has no capabilities defined. If you know the capabilities required by this plugin, please contribute!

Settings

The target-s3-csv settings that are known to Meltano are documented below. To quickly find the setting you're looking for, click on any setting name from the list:

You can also list these settings using 

meltano config
with the list subcommand: 

meltano config target-s3-csv list

You can override these settings or specify additional ones in your meltano.yml by adding the settings key.

Please consider adding any settings you have defined locally to this definition on MeltanoHub by making a pull request to the YAML file that defines the settings for this plugin.

Add Metadata Columns (add_metadata_columns)

  • Environment variable: TARGET_S3_CSV_ADD_METADATA_COLUMNS

(Default - False) Metadata columns add extra row level information about data ingestions, (i.e. when was the row read in source, when was inserted or deleted in snowflake etc.) Metadata columns are creating automatically by adding extra columns to the tables with a column prefix SDC. The column names are following the stitch naming conventions documented at https://www.stitchdata.com/docs/data-structure/integration-schemas#sdc-columns. Enabling metadata columns will flag the deleted rows by setting the _SDC_DELETED_AT metadata column. Without the add_metadata_columns option the deleted rows from singer taps will not be recongisable in Snowflake.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set add_metadata_columns [value]

S3 Access Key Id (aws_access_key_id)

  • Environment variable: TARGET_S3_CSV_AWS_ACCESS_KEY_ID

If not provided, AWS_ACCESS_KEY_ID environment variable will be used.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set aws_access_key_id [value]

AWS endpoint URL (aws_endpoint_url)

  • Environment variable: TARGET_S3_CSV_AWS_ENDPOINT_URL

AWS endpoint URL


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set aws_endpoint_url [value]

AWS profile (aws_profile)

  • Environment variable: TARGET_S3_CSV_AWS_PROFILE

Name for profile based authentication. If not provided, AWS_PROFILE environment variable will be used.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set aws_profile [value]

S3 Secret Access Key (aws_secret_access_key)

  • Environment variable: TARGET_S3_CSV_AWS_SECRET_ACCESS_KEY

If not provided, AWS_SECRET_ACCESS_KEY environment variable will be used.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set aws_secret_access_key [value]

AWS Session token (aws_session_token)

  • Environment variable: TARGET_S3_CSV_AWS_SESSION_TOKEN

If not provided, AWS_SESSION_TOKEN environment variable will be used.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set aws_session_token [value]

Compression (compression)

  • Environment variable: TARGET_S3_CSV_COMPRESSION

The type of compression to apply before uploading. Supported options are none (default) and gzip. For gzipped files, the file extension will automatically be changed to .csv.gz for all files.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set compression [value]

Delimiter (delimiter)

  • Environment variable: TARGET_S3_CSV_DELIMITER

(Default - ',') A one-character string used to separate fields.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set delimiter [value]

Encryption Key (encryption_key)

  • Environment variable: TARGET_S3_CSV_ENCRYPTION_KEY

A reference to the encryption key to use for data encryption. For KMS encryption, this should be the name of the KMS encryption key ID (e.g. '1234abcd-1234-1234-1234-1234abcd1234'). This field is ignored if 'encryption_type' is none or blank.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set encryption_key [value]

S3 Access Key Id (encryption_type)

  • Environment variable: TARGET_S3_CSV_ENCRYPTION_TYPE

(Default - 'none') The type of encryption to use. Current supported options are - 'none' and 'KMS'.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set encryption_type [value]

Naming Convention (naming_convention)

  • Environment variable: TARGET_S3_CSV_NAMING_CONVENTION

(Default - None) Custom naming convention of the s3 key. Replaces tokens date, stream, and timestamp with the appropriate values. Supports "folders" in s3 keys e.g. folder/folder2/{stream}/export_date={date}/{timestamp}.csv. Honors the s3_key_prefix, if set, by prepending the "filename". E.g. naming_convention = folder1/my_file.csv and s3_key_prefix = prefix_ results in folder1/prefix_my_file.csv


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set naming_convention [value]

Quote Char (quotechar)

  • Environment variable: TARGET_S3_CSV_QUOTECHAR

(Default - '"') A one-character string used to quote fields containing special characters, such as the delimiter or quotechar, or which contain new-line characters.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set quotechar [value]

S3 Bucket name (s3_bucket)

  • Environment variable: TARGET_S3_CSV_S3_BUCKET

S3 Bucket name


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set s3_bucket [value]

S3 Key Prefix (s3_key_prefix)

  • Environment variable: TARGET_S3_CSV_S3_KEY_PREFIX

(Default - None) A static prefix before the generated S3 key names. Using prefixes you can


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set s3_key_prefix [value]

S3 Access Key Id (temp_dir)

  • Environment variable: TARGET_S3_CSV_TEMP_DIR

(Default - platform-dependent) Directory of temporary CSV files with RECORD messages.


Configure this setting directly using the following Meltano command:

meltano config target-s3-csv set temp_dir [value]

Something missing?

This page is generated from a YAML file that you can contribute changes to.

Edit it on GitHub!

Looking for help?

If you're having trouble getting the target-s3-csv loader to work, look for an existing issue in its repository, file a new issue, or join the Meltano Slack community and ask for help in the 
#plugins-general
channel.

Install

meltano add loader target-s3-csv

Maintenance Status

  • Maintenance Status

Repo

https://github.com/transferwise/pipelinewise-target-s3-csv
  • Stars
  • Forks
  • Last Commit Date
  • Open Issues
  • Open PRs
  • Contributors
  • License

Maintainer

  • Wise

Meltano Stats

  • Total Executions (Last 3 Months)
  • Projects (Last 3 Months)

Keywords