gcp toolkit

io module

class gcp_toolkit.io.IO(bucket_name, staging_dataset, bq_client=None, storage_client=None)

Bases: object

bq_to_bucket(query, path_to_file)

Runs query in BigQuery and stores results in Storage

bq_to_df(query, use_builtin=False)

Runs a query in BigQuery and loads the results into a pandas Data Frame

bucket_to_bq(path_to_file, table_id, schema=[], csv_delimiter=', ')

Loads a csv from Storage into a BigQuery table

bucket_to_df(path_to_file)

Reads a file or a group of files matching a pattern into a pandas Data Frame

df_to_bq(df, table_id, schema=[])

Loads a pandas Data Frame into a BigQuery table.

utils module

gcp_toolkit.utils.change_table_schema(table_id, new_fields, bigquery_client=None)

Updates a table schema with new fields

gcp_toolkit.utils.convert_pandas_gbq_schema(schema)

Converts a BigQuery schema in the format of a list of dicts, used by pandas gbq into a list of SchemaFields

gcp_toolkit.utils.create_bucket_folder(bucket_name, folder_name, storage_client=None)

Creates a Folder in Storage