Visit Intelligence Data Structure
The below table represents the structure of the daily file which will be shared with partners. The file contains all the information gathered by the top data providers of Cuebiq. The file will be delivered in TSV format and organized in columns.
|Standard Visit Intelligence||High Volume Visit Intelligence|
Timestamp (of Visit)
Timestamp (of first time seen at POI)
|Device ID||Device ID|
|POI||POI (Place ID, Place Name, Geoset)|
Cuebiq reserves the ability to add new fields to the ones above. New fields will be added at the end of the existing list.
Example of Visit Intelligence data from a TSV file:
2019-02-15T11:08:14-07:00 e132698c-5676-4161-b201-22d63de12306 17.13 82c306f2-2be6-4362-a173-45a097a53ca2 BEST_BUY BEST_BUY_ANALYTICS
2019-02-15T17:31:49-05:00 8B0EC37F-DD8E-407B-AB0C-2650D45B5BE3 34.55 4ad6c0e1-e16a-47ed-82c6-3b63d76f0144 Sam's Club SAM_CLUB_ANALYTICS
2019-02-15T09:05:37-05:00 8b0ec09d-d9b0-4792-9b46-8d5ed8e60c75 62.23 1589dcce-b3b5-4434-bda8-046c39cd40f7 Fifth Third Bank & ATM FIFTH_THIRD_BANK_ANALYTICS
Data Access for a Feed in Production
In a production setting where customers are purchasing live ongoing data, Cuebiq would generate Visit Intelligence data on a daily cadence and will share data feed via an Amazon S3 bucket.
A Cuebiq representative will generate a pair of Amazon AWS keys and provide access to the s3 path where the data feed will be stored.
Data is shared according to the following rules:
- Data is added to the S3 bucket daily, with .gzip compression
- Files are broken into subparts in order to optimize performance
- Files are delimited by tab
- Each file contains a set of data points which can have a 1 to 7-day delay
- Each file is automatically removed 30 days after its creation
- Files can be downloaded more than once within the 30-day window
Cuebiq reserves the right to monitor any activity occurring on Cuebiq owned S3 buckets. Misuse (e.g. downloading the same file multiple times, trying to access a path different from the one provided, performing actions unrelated to data download, etc...) will be notified to the user, and, in severe cases, will be sanctioned deactivating the user’s account.
Data Partition Details
The folder names in Amazon s3 will represent the day on which the data was processed, for example:
2019020100 - meaning the data was processed on February 1, 2019
2019020200 - meaning the data was processed on February 2, 2019
2019020300 - meaning the data was processed on February 3, 2019
Cuebiq processes data at 12AM UTC and there can also be a certain amount of lag in the data as it is first batched on device before being sent to our servers, therefore the timestamps in each dated folder may not represent only that day and could potentially include some data from additional dates.
The rule of thumb is that 90+% of data for a given day can be found by looking at the data for that day and the following day. So for example, to find the data for January 1 2019, the best practice is to look at the folders named:
2019010100 - meaning the data was processed on January 1, 2019
2019010200 - meaning the data was processed on January 2, 2019
Confirming Data Access to Cuebiq’s s3
To confirm that access to Cuebiq’s s3 has been established correctly, a quick check can be run using AWS cli.
Installation documentation for AWS cli can be found here - https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html
Once installed, navigate to the command line and run the below command:
aws -- version
In order to confirm that AWS cli was correctly installed.
Once AWS cli is confirmed to be successfully installed, configure a profile with the below command
aws configure --profile cuebiq_data
You will then be prompted to enter the below fields:
AWS Access Key ID [None]: xxxxx
AWS Secret Access Key [None]: xxxxx
Default region name [None]:
Default output format [None]:
Access key and secret key can be filled in with the credentials provided by your Cuebiq Rep. Default region name and Default output format can remain blank.
Once a profile has been configured, the below command can be used to test that access to Cuebiq's s3 bucket is working as expected:
aws s3 ls s3://<cuebiq_path_provided_goes_here>/
With the s3 path mirroring what Cuebiq has provided exactly. If configured correctly, you should now be able to see the folders within Cuebiq's s3.