Importing Data Locker to Big Query

  • Advertisers
  • Developers

af-s3-gcs2.png

Introduction

If your organization uses Google Cloud Storage (GCS), you can transfer data from Data Locker to GCS.

One of the major benefits of transferring data from Data Locker to GCS is that you can import data to Google Big Query. You don't need to rely on your organization's developers to retrieve data from Data Locker. With a simple UI configuration you can transfer the data straight into Google Big Query.

Google Big Query can easily handle data from Data Locker thanks to the standard structure of the CSV files in Data Locker.  Once you import data into Google Big Query you can start analyzing it right away. 

Prerequisites

Before you can configure GCS to pull data from Data Locker, the following is required:

AWS Access Key and Secret

  1. Access the app's dashboard
  2. In the left-hand side menu, click on Data Locker under Integration
  3. Look for Credentials Details in the right-hand side of the screen
  4. Copy your AWS Access Key
  5. Click on Show Secret
  6. Copy your secret key

Your Data Locker Home Folder

In the Data Locker screen, under Credentials Details, copy your Data Locker Home Folder.

Configuring a Transfer Job in GCS

The first step is to configure a transfer job in GCS. Here, you configure GCS to pull data from Data Locker on a daily basis. Once the data gets into GCS, you can import it to Big Query.

  1. Open your GCS Transfer Console
  2. Click on Transfer in the left-hand side menu
  3. Click on Create transfer
  4. Under Select Source choose Amazon S3 bucket
  5. In the Amazon S3 bucket field put af-ext-raw-data
  6. In the Transfer files with these prefixes field put your home folder
  7. Specify your access key and secret in the relevant fields

     

  8. Under Select Destination specify the GCS storage bucket

     

  9. Under Configure Transfer choose when to run the transfer job

     

Importing Data from GCS to Big Query

Once you have data from Data Locker imported into GCS, you can transfer it to Big Query. Big Query handles the standard CSV format of files from Data Locker for seamless table creation. Once you can create table, you can go ahead and analyze your data quickly and easily.

  1. Open your Big Query Console
  2. Click the plus sign next to the dataset to create a new table
  3. In Location select Google Cloud Storage
  4. Specify the GCS bucket
  5. Name the new table in the field Table Name
  6. Check Automatically detected next to Schema
  7. Click Create Table

     

Was this article helpful?
0 out of 0 found this helpful