Visit Intelligence Data Structure

The below table represents the structure of the daily file which will be shared with partners. The file contains all the information gathered by the top data providers of Cuebiq. The file will be delivered in TSV format and organized in columns.

Cuebiq reserves the ability to add new fields to the ones above.  New fields will be added at the end of the existing list.

Example of Visit Intelligence data from a TSV file:

2019-02-15T11:08:14-07:00      e132698c-5676-4161-b201-22d63de12306     17.13    82c306f2-2be6-4362-a173-45a097a53ca2  BEST_BUY       BEST_BUY_ANALYTICS

2019-02-15T17:31:49-05:00      8B0EC37F-DD8E-407B-AB0C-2650D45B5BE3     34.55    4ad6c0e1-e16a-47ed-82c6-3b63d76f0144  Sam's Club       SAM_CLUB_ANALYTICS

2019-02-15T09:05:37-05:00      8b0ec09d-d9b0-4792-9b46-8d5ed8e60c75      62.23    1589dcce-b3b5-4434-bda8-046c39cd40f7   Fifth Third Bank & ATM FIFTH_THIRD_BANK_ANALYTICS

Data Access for a Feed in Production

In a production setting where customers are purchasing live ongoing data, Cuebiq would generate Visit Intelligence data on a daily cadence and will share data feed via an Amazon S3 bucket. 

A Cuebiq representative will generate a pair of Amazon AWS keys and provide access to the s3 path where the data feed will be stored.

Data is shared according to the following rules:

• Data is added to the S3 bucket daily, with .gzip compression
• Files are broken into subparts in order to optimize performance
• Files are delimited by tab
• Each file contains a set of data points which can have a 1 to 7-day delay
• Each file is automatically removed 30 days after its creation
• Files can be downloaded more than once within the 30-day window

Cuebiq reserves the right to monitor any activity occurring on Cuebiq owned S3 buckets. Misuse (e.g. downloading the same file multiple times, trying to access a path different from the one provided, performing actions unrelated to data download, etc...) will be notified to the user, and, in severe cases, will be sanctioned deactivating the user’s account.

Data Partition Details

The folder names in Amazon s3 will represent the day on which the data was processed, for example:

2019020100 - meaning the data was processed on February 1, 2019

2019020200 - meaning the data was processed on February 2, 2019

2019020300 - meaning the data was processed on February 3, 2019

Cuebiq processes data at 12AM UTC and there can also be a certain amount of lag in the data as it is first batched on device before being sent to our servers, therefore the timestamps in each dated folder may not represent only that day and could potentially include some data from additional dates.

The rule of thumb is that 90+% of data for a given day can be found by looking at the data for that day and the following day. So for example, to find the data for January 1 2019, the best practice is to look at the folders named:

2019010100 - meaning the data was processed on January 1, 2019

2019010200 - meaning the data was processed on January 2, 2019

Confirming Data Access to Cuebiq’s s3

To confirm that access to Cuebiq’s s3 has been established correctly, a quick check can be run using AWS cli.

Installation documentation for AWS cli can be found here - https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html

Once installed, navigate to the command line and run the below command:

aws -- version

In order to confirm that AWS cli was correctly installed.

Once AWS cli is confirmed to be successfully installed, configure a profile with the below command

aws configure --profile cuebiq_data

You will then be prompted to enter the below fields:

AWS Access Key ID [None]: xxxxx

AWS Secret Access Key [None]: xxxxx

Default region name [None]:

Default output format [None]:

Access key and secret key can be filled in with the credentials provided by your Cuebiq Rep. Default region name and Default output format can remain blank.

Once a profile has been configured, the below command can be used to test that access to Cuebiq's s3 bucket is working as expected:

aws s3 ls s3://<cuebiq_path_provided_goes_here>/

With the s3 path mirroring what Cuebiq has provided exactly. If configured correctly, you should now be able to see the folders within Cuebiq's s3.