Table of Contents
Relocation Index Data Structure
Confirming Data Access to Cuebiq’s S3
Visit Index Data Structure
The table below outlines the structure of Visit Index data that will be shared with clients. Files will be delivered in a CSV format. There will be one file that is updated each day with the entire history of available Visit Index data.
Visit Index data can be found in files with the naming convention 'cvi-{sector}-{date}.csv000.gz
Cuebiq reserves the right to add additional columns to the data schema at Cuebiq's discretion. Any additional columns will always be added to the end of data schema.
|
Field |
Description |
Type |
|
Visit Week Cd |
Epoch week code |
String/Varchar |
|
Week Label |
Week label |
String/Varchar |
|
Market Area Code |
Designated Market Area code |
Integer |
|
Market Area |
Designated Market Area name |
String/Varchar |
|
Brand |
Brand name |
String/Varchar |
|
Vertical |
Brand vertical |
String/Varchar |
|
Sector |
Vertical sector |
String/Varchar |
|
CVI |
Cuebiq visit index |
Float/Double |
|
CVI per Store |
Cuebiq visit index per store |
Float/Double |
Example of Data
|
visit_week_cd |
week_label |
market_area_code |
market_area |
brand |
vertical |
sector |
cvi |
cvi_per_store |
|
2020-W01 |
Week of 19-12-30 |
507 |
Savannah (GA) |
Toyota |
Car Brands |
Automotive |
0.3743 |
0.0143 |
|
2020-W02 |
Week of 20-01-06 |
556 |
Richmond-Petersburg (VA) |
Walmart |
Big Box |
Retail |
32.0037 |
0.0949 |
|
2020-W03 |
Week of 20-01-13 |
612 |
Shreveport (LA) |
Chipotle Mexican Grill |
QSR |
Dining |
0.3907 |
0.01 |
|
2020-W04 |
Week of 20-01-20 |
546 |
Columbia (SC) |
Verizon Wireless |
Mobile Carrier |
Telco |
1.3491 |
0.0058 |
|
2020-W05 |
Week of 20-01-27 |
819 |
Seattle-Tacoma (WA) |
US Airports |
Airports |
Transportation |
31.4195 |
0.1023 |
Daily Visit Index Data Structure
The table below outlines the structure of the Daily Visit Index file that will be shared with clients. Files will be delivered in a CSV format. There will be one file that is updated each day with the entire history of available Daily Visit Index data.
Daily Visit Index data can be found in files with the naming convention 'daily-cvi-{sector}-{date}.csv000.gz
Cuebiq reserves the right to add additional columns to the data schema at Cuebiq's discretion. Any additional columns will always be added to the end of data schema.
| Field | Description | Type |
| Ref_dt | Reference Date | Date |
| Market Area Code | Designated Market Area code | Integer |
| Market Area Name | Designated Market Area Name | String/Varchar |
| Sector | Overall Sector the brand belongs to | String/Varchar |
| Vertical | Vertical the brand belongs to | String/Varchar |
| Brand | Brand Name | String/Varchar |
| NAICS6 Code | North American Industry Classification System code | String/Varchar |
| roll_avg_7days_cvi | 7-day rolling average for CVI | Float/Double |
| ly_roll_avg_7days_cvi | 7 Day rolling average of CVI from the same period of time last year | Float/Double |
| roll_avg_7days_cvi_per_store | 7-day rolling average for CVI per store | Float/Double |
| ly_roll_avg_7days_cvi_per_store | 7-day rolling average for CVI per store from the same period of time last year | Float/Double |
Example of Data
| Ref_dt | market_area_code | market_area | sector | vertical | brand | naics6_code | roll_avg_7days_cvi | ly_roll_avg_7days_cvi | roll_avg_7days_cvi_per_store | ly_roll_avg_7days_cvi_per_store |
| 2020-01-01 00:00:00 | 500 | "Portland-Auburn, Me" | Automotive | Auto Parts | Advance Auto Parts | 441310 | 0.71660000000000001 | 0.4123 | 0.0511 | 0.041200000000000001 |
| 2020-01-02 00:00:00 | 500 | "Portland-Auburn, Me" | Automotive | Auto Parts | Advance Auto Parts | 441310 | 0.77310000000000001 | 0.51759999999999995 | 0.048300000000000003 | 0.051700000000000003 |
| 2020-01-03 00:00:00 | 500 | "Portland-Auburn, Me" | Automotive | Auto Parts | Advance Auto Parts | 441310 | 0.75780000000000003 | 0.54220000000000002 | 0.050500000000000003 | 0.049200000000000001 |
| 2020-01-04 00:00:00 | 500 | "Portland-Auburn, Me" | Automotive | Auto Parts | Advance Auto Parts | 441310 | 0.75580000000000003 | 0.58609999999999995 | 0.053900000000000003 |
0.044999999999999998 |
Mobility Index Data Structure
The table below outlines the structure of Mobility Index data that will be shared with clients. Files will be delivered in CSV format and organized by daily partitions. There will be one file that is updated each day with the entire history of available Mobility Index data.
Cuebiq reserves the right to add additional columns to the data schema at Cuebiq's discretion. Any additional columns will always be added to the end of data schema.
|
Field |
Description |
Type |
|
Ref dt |
Reference date |
Date |
|
Week Name |
Epoch week code |
String/Varchar |
|
County Name |
Name of geographical region |
String/Varchar |
|
State Name |
State name of county |
String/Varchar |
|
County FIPS Code |
The Federal Information Processing Standard code that uniquely identifies a county |
String/Varchar |
|
CMI |
Cuebiq Mobility Index |
Float/Double |
|
% Sheltered in Place |
Percent of devices that sheltered in place, defined by a max distance of 100 meters |
Float/Double |
|
% less than 1 mile |
Percent of devices that traveled 1 mile or less |
Float/Double |
|
% less than 10 miles |
Percent of devices that traveled 10 miles or less |
Float/Double |
|
CMI 7 day rolling avg |
7 Day rolling average of CMI |
Float/Double |
|
% Sheltered in Place 7 day rolling avg |
7 day rolling average of percent of devices sheltered in place |
Float/Double |
|
% less than 1 mile 7 day rolling avg |
7 day rolling average of percent of devices that traveled 1 mile or less |
Float/Double |
|
% less than 10 miles 7 day rolling avg |
7 day rolling average of percent of devices that traveled 10 mile or less |
Float/Double |
|
CMI 7 day rolling avg last year |
7 Day rolling average of CMI from the same period of time last year |
Float/Double |
| % Sheltered in Place 7 day rolling avg last year | 7 day rolling average of percent of devices sheltered in place from the same period of time last year | Float/Double |
| % less than 1 mile 7 day rolling avg last year | 7 day rolling average of percent of devices that traveled 1 mile or less from the same period of time last year | Float/Double |
|
% less than 10 miles 7 day rolling avg last year |
7 day rolling average of percent of devices that traveled 10 mile or less from the same period of time last year |
Float/Double |
|
Contact Index 7 day rolling average |
7 day rolling average of Cuebiq contact index |
Float/Double |
|
Contact Index 7 day rolling average last year |
7 day rolling average of Cuebiq contact index from the same period of time last year |
Float/Double |
Example of Data
|
ref_dt |
week_name |
county_name |
state_name |
county_fips_code |
cmi |
% Sheltered |
% less than 1 mile |
% less than 10 miles |
cmi_7days_rolling_avg |
sheltered_in_place_7days_rolling_avg | less_1_mile_7days_rolling_avg | less_10_mile_7days_rolling_avg | cmi_ly_7days_rolling_avg | sheltered_in_place_ly_7days_rolling_avg | less_1_mile_ly_7days_rolling_avg | less_10_mile_ly_7days_rolling_avg | cci_7days_rolling_avg | cci_ly_7days_rolling_avg |
|
2020-01-01 |
2020-W01 |
Philadelphia |
Pennsylvania |
42101 |
3.157410973 |
0.5 |
0.3 |
0.2 |
3.6953978363644349 |
0.33582962789609172 | 0.059208986660425927 | 0.31219283875497311 | 3.5820918780768283 | 0.38722168441432719 | 0.04872539528880284 | 0.30655050016134239 | 0.66959798994974873 | 0.48858447488584472 |
|
2020-01-06 |
2020-W02 |
Denver |
Colorado |
08031 |
3.685635747 |
0.4 |
0.3 |
0.3 |
3.8991554302166174 |
0.28194347115160007 | 0.046484466246204156 | 0.29969633263256251 | 3.8365488634552469 | 0.30667526604321188 | 0.042889390519187359 | 0.30312802321831667 | 0.66455696202531644 | 0.47222222222222221 |
|
2020-01-13 |
2020-W03 |
Orleans |
Louisiana |
22071 |
3.700853257 |
0.6 |
0.2 |
0.2 |
3.9805868336089634 |
0.25361305361305364 | 0.041025641025641026 | 0.30326340326340329 | 3.927586170753838 | 0.27588424437299036 | 0.04212218649517685 | 0.30096463022508041 | 0.65350318471337576 | 0.45775301764159704 |
|
2020-01-20 |
2020-W04 |
Westchester |
New York |
36119 |
3.636130451 |
0.7 |
0.1 |
0.1 |
3.9737639422516713 |
0.24784130688448075 | 0.041073512252042005 | 0.31178529754959161 | 3.9857005377950063 | 0.25489566613162118 | 0.038523274478330656 | 0.30208667736757622 | 0.63589076723016902 | 0.45343367826904984 |
|
2020-01-27 |
2020-W05 |
Santa Barbara |
California |
06083 |
3.59367014 |
0.5 |
0.4 |
0.1 |
3.9502645544277142 |
0.25180863477246207 | 0.040373395565927651 | 0.32112018669778297 | 3.9769525265229753 | 0.25641025641025639 | 0.037179487179487179 | 0.30544871794871797 | 0.63089005235602091 | 0.45650140318054255 |
Total Flow Data Structure
The table below outlines the structure of Total Flow data that will be shared with clients. Files will be delivered in CSV format and organized by daily partitions. There will be one file that is updated each day with the entire history of available Total Flow data.
Cuebiq reserves the right to add additional columns to the data schema at Cuebiq's discretion. Any additional columns will always be added to the end of data schema.
|
Field |
Description |
Type |
|
Country |
Two Letter Country code |
String/Varchar |
|
Local Date |
Reference Date |
String/Varchar |
|
County FIPS Code |
The Federal Information Processing Standard code that uniquely identifies a county |
String/Varchar |
|
Fraction In |
The ratio of devices arriving to the county to total devices seen in that county on that date |
Float/Double |
|
Fraction Out |
The ratio of devices departing the county to total devices seen in that county on that date |
Float/Double |
|
Userbase Share |
The ratio of devices that we see in this county to all the devices seen across all counties |
Float/Double |
Example of Data
| country | local_date | county_code | fraction_in | fraction_out | userbase_share |
| US | 2020-01-08 | 01121 | 0.60641701 | 0.61225066 | 0.00045913 |
| US | 2020-01-08 | 36117 | 0.55981651 | 0.54697248 | 0.00023544 |
| US | 2020-01-08 | 42007 | 0.53843608 | 0.53090608 | 0.00052207 |
| US | 2020-01-08 | 17103 | 0.6244753 | 0.6151114 | 0.00013379 |
| US | 2020-01-08 | 48061 | 0.17291341 | 0.17428604 | 0.001133 |
Flow Network Data Structure
The table below outlines the structure of Flow Network data that will be shared with clients. Files will be delivered in CSV format and organized by daily partitions. There will be one file that is updated each day with the entire history of available Flow Network data.
Cuebiq reserves the right to add additional columns to the data schema at Cuebiq's discretion. Any additional columns will always be added to the end of data schema.
|
Field |
Description |
Type |
|
Country |
Two Letter Country code |
String/Varchar |
|
Week Name |
Epoch week code |
String/Varchar |
|
County Code |
Origin county of the flow |
String/Varchar |
|
Next County Code |
Destination county of the flow |
String/Varchar |
|
Fraction In |
Devices that arrived at next_area_id and departed from area_id within 24 hours prior, as a percentage of total devices seen at area_id. |
Float/Double |
|
Fraction Out |
Devices that departed area_id and arrived at next_area_id within the 24 hours later, as a percentage of total devices seen at area_id |
Float/Double |
Example of Data
| country | week | county_code | next_county_code | fraction_in | fraction_out |
| US | Week of 2019-12-30 | 01003 | 22051 | 0.00776384 | 0.00748035 |
| US | Week of 2019-12-30 | 01015 | 01073 | 0.1025726 | 0.10437404 |
| US | Week of 2019-12-30 | 01059 | 01079 | 0.0553859 | 0.05586449 |
| US | Week of 2019-12-30 | 01085 | 12091 | 0.01154821 | 0.00937627 |
| US | Week of 2019-12-30 | 01091 | 01047 | 0.03138151 | 0.03636364 |
Relocation Index Data Structure
The table below outlines the structure of Relocation Index data that will be shared with clients. Files will be delivered in CSV format and organized by daily partitions. There will be one file that is updated each day with the entire history of available Relocation data.
Cuebiq reserves the right to add additional columns to the data schema at Cuebiq's discretion. Any additional columns will always be added to the end of data schema.
|
Field |
Description |
Type |
|
ref_week_code |
Epoch week code |
String/Varchar |
|
ref_week_name |
Week date |
String/Varchar |
|
new_county_fips_code |
FIPS code for the new county of home location |
String/Varchar |
|
new_county_name |
Name of the new home county |
String/Varchar |
|
old_county_fips_code |
FIPS code for the county of previous home location |
String/Varchar |
|
old_county_name |
Name of the county where prior home location was identified |
String/Varchar |
|
home_switcher_pct |
Calculated as number of users switching home / number of users having home location in the Old County |
Float/Double |
| top10_home_switcher_pct |
Top 10 household income percentile (same a #6 but considering only the users living in the richest census block groups) |
Float/Double |
| bottom10_home_switcher_pct |
Bottom 10 household income percentile (same as #6 but considering only the users living in the poorest census block groups) |
Float/Double |
Example of Data
| ref_week_code | ref_week_name | new_county_fips_code | new_county_name | old_county_fips_code | old_county_name | home_switcher_pct | top10_home_switcher_pct | bottom10_home_switcher_pct |
| 2020-W37 | Week of 2020-09-07 | 1001 | Autauga | 1051 | Elmore | 0.10658307 | 0 | 0.02631579 |
| 2020-W37 | Week of 2020-09-07 | 1001 | Autauga | 1103 | Morgan | 0.00696864 | 0 | 0.01388889 |
| 2020-W37 | Week of 2020-09-07 | 1001 | Autauga | 19049 | Dallas | 0.00380228 | 0.11111111 | 0 |
| 2020-W37 | Week of 2020-09-07 | 1001 | Autauga | 51149 | Prince George | 0.00704225 | 0 | 0 |
| 2020-W37 | Week of 2020-09-07 | 1001 | Autauga | 12033 | Escambia | 0.003125 | 0 | 0 |
| 2020-W37 | Week of 2020-09-07 | 1001 | Autauga | 12091 | Okaloosa | 0.00174216 | 0 | 0.01388889 |
| 2020-W37 | Week of 2020-09-07 | 1001 | Autauga | 24043 | Washington | 0.00540541 | 0 | 0 |
| 2020-W37 | Week of 2020-09-07 | 1001 | Autauga | 45059 | Laurens | 0.00793651 | 0 | 0 |
Data Access in Production
Cuebiq will generate Mobility index and Visit Index data on a daily cadence and will share data via an Amazon S3 bucket. A Cuebiq representative will generate a pair of Amazon AWS keys and provide access to the S3 path where the data feed will be stored.
Data is shared according to the following rules:
- Data is added to the S3 bucket daily, with .gzip compression
- Files are comma-delimited
- Each day’s files will contain the entire history of CMI/CVI data
- Each file is automatically removed 10 days after its creation
- Files can be downloaded more than once within the 10-day window
Cuebiq reserves the right to monitor any activity occurring on Cuebiq owned S3 buckets. Misuse (e.g. downloading the same file multiple times, trying to access a path different from the one provided, performing actions unrelated to data download, etc...) will be notified to the user, and, in severe cases, will be sanctioned deactivating the user’s account.
Confirming Data Access to Cuebiq’s S3
To confirm that access to Cuebiq’s S3 has been established correctly, a quick check can be run using AWS cli.
Installation documentation can be found here: Installing the AWS CLI - AWS Command Line Interface
Navigate to the command line and run the command below to confirm successful installation: aws --version
Once AWS CLI is confirmed to be successfully installed, configure a profile with the command below:
aws configure --profile cuebiq_data
You will then be prompted to enter the below fields:
AWS Access Key ID [None]: xxxxx
AWS Secret Access Key [None]: xxxxx
Default region name [None]:
Default output format [None]:
Access key and secret key will need to be filled in with the credentials provided by your Cuebiq rep. Default region name and Default output format can remain blank.
Once a profile has been configured, the below command can be used to test that access to Cuebiq's S3 bucket is working as expected:
For Mobility Index:
aws s3 ls s3://cuebiq-dataset-nv/offline-intelligence/index=cmi/country=US/ --profile cuebiq_data
For Visit Index:
aws s3 ls s3://cuebiq-dataset-nv/offline-intelligence/index=cvi/sector=<enter_sector_name>/country=US/ --profile cuebiq_data
Comments
1 comment
How I can obtain the AWS Access Key ID and AWS Secret Access Key? thanks! :)
Please sign in to leave a comment.