If your organization uses Google Cloud Storage (GCS), you can transfer data from Data Locker to GCS.
One of the major benefits of transferring data from Data Locker to GCS is that you can import data to Google Big Query. You don't need to rely on your organization's developers to retrieve data from Data Locker. With a simple UI configuration, you can transfer the data straight into Google Big Query.
Google Big Query can easily handle data from Data Locker thanks to the standard structure of the CSV files in Data Locker. Once you import data into Google Big Query you can start analyzing it right away.
Before you can configure GCS to pull data from Data Locker, you need your AWS access key, secret key, and home folder id.
To retrieve your AWS access key, secret key, and home folder ID:
- Go to Integration > Data Locker.
The Data Locker window opens.
- On the right-hand side of the window, locate the Credentials panel.
- Copy the AWS Access Key.
- Click Show Secret.
The Show secret window opens.
- Copy the Secret Key.
- Copy the Home Folder ID.
Configuring a Transfer Job in GCS
The first step is to configure a transfer job in GCS. Here, you configure GCS to pull data from Data Locker on a daily basis. Once the data gets into GCS, you can import it to Big Query.
- Open your GCS Transfer Console.
- In the left-hand side menu, click Transfer.
- Click on Create transfer.
- Under Select Source choose Amazon S3 bucket.
- In the Amazon S3 bucket field put af-ext-reports.
- In the Transfer files with these prefixes field put the following path:
- Specify your access key and secret in the relevant fields.
- Under Select Destination specify the GCS storage bucket
- Under Configure Transfer choose when to run the transfer job
Importing Data from GCS to Big Query
Once you have data from Data Locker imported into GCS, you can transfer it to Big Query. The files are zipped in .gz format but Big Query knows how to unzip the files. Big Query also handles the standard CSV format of files from Data Locker for seamless table creation. Once you can create a table, you can go ahead and analyze your data quickly and easily.