Enriching Media Data: A Special Report from the U.S. Coalition of Innovative Media Measurement

Data Quality Generally Good (But Needs to Improve)

End users were mostly positive about the overall quality of data provided by third-party firms, particularly the larger ones who have been perfecting their standards and processes for decades. There was recognition that some firms tended to excel in delivering data for specific industry sectors like automotive versus insurance or CPG, and no one firm scored high grades across the entire gamut of marketplace data. Most important to note, end users felt that third-party data is making their marketing and advertising campaigns more effective; however, they would like to see some improvement in data quality. Among their comments:

• “‘Will never be absolutely satisfied with data quality but feel our company has industry edge in performance through data integrations.” (Advertiser)

• “Significant lift in effectiveness driven by 1st party data but third-party data generates better lift than standard targeting.” (Multichannel video programming distributor [MVPD])

• “Big firms that originate their own third-party assets, they tend to do it very well, use same approach on clients' 1st party data.” (Data intermediary)

• “All do a decent job but occasionally make back-end mistakes (calculations, wrong data) when, for example, matching viewer to transaction, lack of quality control.” (Media agency)

Consumer Segments: Top Data Quality Issue

First and foremost, end users expressed concern about accuracy of modeled consumer segments created by data partners for use in look-alike and/or behavioral targeting. They cited the practice of creating target-audience profiles from transactional data sourced from a small base of consumers and extrapolated to the larger population.

End users also sought to understand how effective modeled targets were in driving sales; they called into question sales-predictive behavioral indicators, such as “visited an automobile site” or “read travel content” as markers for automotive or travel intenders, for example. The feeling was that these consumers may be targeted too late, after they've purchased the car or made their travel plans.

Not surprisingly, data recency was cited as another data quality issue that ties back to the ability to identify in-market consumers, as well as capturing timely changes in consumer marketplace dynamics. Finally, it was acknowledged that smaller, low-incidence targets tend to have the highest rate of error when modeling due to the paucity of actual consumer behavior data available to model from. Comments:

• “We have used firm A for modeling television show websites visitors and found little agreement in profile with syndicated sources.” (Television network)

• “Small modeled targets can be land mines with large error.” (MVPD)

• “Recency of information is key, especially for behavioral, in-market aspects.” (TV network)

• “Want to identify in-market prospects, propensity modeling; some advertisers want to break out actual target vs. modeled.” (Media agency)

Transparency and Decision Making

End users recognize the fact that there is no perfect data source; there will always be some amount of error inherent in any dataset depending on the quality/reliability of source, data preparation procedures, survey response accuracy, and so forth. But disclosure of the extent of the error would go a long way to inform how the data should be used in making marketing and advertising investment decisions.

To that end, the callout for greater data quality transparency resounded repeatedly during end-user conversations as they sought to understand, for example, techniques used for modeling target consumers, recency of data and quality, and reliability of underlying data sources. Insight into key-driver model variables as well as modeled ROI results versus actual consumer ROI were of keen interest (See Figure 4).

• “Data not perfect but want to know how imperfect it is.” (Media agency)

• “Need to understand sources of error and how they're are mitigated.” (Advertiser)

• “Integrations introduce error, for example, how wrong are they getting personal data?” (Media agency)

To Be, or Not To Be, Transparent

Inconsistent transparency was a sore spot for many end users when it came to answers received from third-party firms regarding data quality. Some data companies were found to be vague or nonresponsive about the techniques and methods used to create propensity or look-alike models, a key area of concern.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

End Users—Modeled Target Transparency

Another transparency issue that surfaced was the lack of information about original data sources, especially culled from digital media behavior metrics. In general, larger third-party companies were considered to be more revealing of their approaches to consumer target creation than their intermediate-sized counterparts. Some end users trusted the third-party solutions and tolerated limited transparency, whereas others enforced strict disclosure policies. As one advertiser put it, “If they're not forthcoming in answering our questions, we don't work with them.”

• “Insufficient transparency, the more we dig the more inconsistencies and gaps are uncovered.” (Advertiser)

• “Vague about modeling techniques and the sources their models are built on; can be defensive.” (MVPD)

• “Big companies = lots of transparency. Smaller can be elusive. Companies within AdTech space are not very transparent.” (Data intermediary)

• “Firm A won't reveal retail partners. Firm B and Firm C won't break out known vs. modeled results.” (Media agency)

Compound Data Integrations Can Mean Compound Error

When the mortgage crisis hit the United States during the last decade, it was discovered that consumer debt was sold and re-sold so often that it became nearly impossible to trace records back to the point of origination. Similar situations sometimes exist with data integrations, particularly within digital advertising technology targeting tools, where integration may be layered on integration to the point where original source identification is eclipsed.

Furthermore, there may be no quality assurance procedures followed to ensure that these thickly layered solutions deliver superior results; the implication is that data error may be compounded with introduction of each additional data source. And the lack of transparency within and across these layers makes it nearly impossible for end users to assess the sources as well as the data error. It would be valuable to know, for example, which data elements are key drivers of developing a media target, how recent the data are, where they are sourced from, and how accurate they are at capturing the true advertiser target.

• “The farther the distance from the original data source, the greater the chance of error; there's no opportunity truly to understand the underlying target elements and their impact on media decision making.” (Data intermediary)

• “I'm very suspicious when companies sell pre-segmented packages; data could come from any number of sources.” (Media agency)

• “There tends to be transparency for single, independent metric sources but not for multiple sources.” (Television network)

Each data source within an integration comes with an inherent amount of error that end users should know about in order to gauge impact on marketing and advertising decision making. In the digital space, for example, the practice of high-volume cookie pooling can pose a data quality mystery as end users seek to understand information about

• cookies' point of origination,

• active lifespan,

• recency,

• whether from a registered versus non-registered site, original versus look-alike, model selection criteria, etc. (See Figure 5).

There are currently no industry audits or standards to ensure the quality of cookies or at least facilitate disclosure about their level of business vitality.

Data Intermediaries: Muscle in the Middle

The task of ingesting, cleaning, and formatting data across multiple sources can be daunting for many end users who do not possess the in-house staff or expertise to handle complex data integrations.

Data intermediaries, firms that sit between data suppliers and end users, often fill the end-user resource gap for prepping data. These companies will implement quality assurance measures and build them into integrated data products they market to the industry, or provide custom work-for-hire where they will simply prepare and format data for delivery to end users. Data intermediaries have a unique vantage point in that they have visibility into multiple third-party data supplier offerings and capabilities.

• Download figure
• Open in new tab
• Download powerpoint

Figure 5

End Users: Compound Data Integrations = Compound Error

Data Harmonization: Yes, But Quality First

Bearing in mind the proliferation of data integration and onset of automation in the marketing and advertising industry, end users supported the idea of a common set of names for consumer target segments. But many mentioned that the industry should first focus on data quality issues before standardization, citing the current concerns around consumer authentication, modeled targeting, and transparency.

Once these issues were addressed, end users felt that consistent nomenclature would make life simpler by helping to mitigate differences across data coming from multiple third-party firms. For example, one naming convention for “Men-25-54,” “Males aged 25-54,” and “M 25-54” should be made possible.

In addition to common names, it was felt there should be consistent target definition meanings, which likely would work well for standard age and demographics descriptors but be a bit more challenging for targets like “auto intenders” and “travel planners” due to the fluid and customized nature of how these consumer segments are created by third-party firms.

Last, virtually everyone interviewed indicated that their company would be directly involved in any industry effort for data naming standardization, citing the 4As, Association of National Advertisers (ANA), Interactive Advertising Bureau (IAB), and the Advertising Research Foundation (ARF) as industry associations to help steer the process.

Consumer Authentication and Targeting: Top Customer Concerns

Data suppliers were asked to recount the core questions posed by end users regarding data quality. Their responses were consistent with feedback gathered during the end-user interviews: Consumer authentication and modeled targeting were high up on the short list of data quality questions asked of third-party firm customers. According to data suppliers, end users wanted to know, for example, that males ages 18 to 34 years had been accurately classified and gender-validated. Techniques used for modeled targeting, particularly the portion of actual versus estimated data to create the targets, and recency of data also were on the most-asked list, the latter playing a core role in estimating which consumers are in market to purchase:

• “Data source, validation. Is it representative, how do you know its Men 18–34?” (Data firm F)

• “What is underlying data used to build the modeled solutions and how are the models built?” (Data firm B)

• “What goes into the modeled projections?” (Data firm I)

• “How recent is the data? Where is it sourced from?” (Data firm B)

Consumer Cross-Checking To Address Authentication Issues

Consumer authentication is a core aspect of data quality. Misidentification or misclassifications of consumers at the very beginning of the integration process could have a rippling effect of error that compounds as each data source is combined with the next. For this reason, most data suppliers will run consistency checks to determine whether the same type of information about a consumer, for example, age or marital status, is in agreement across multiple sources; they also will validate residential address by frequently checking various sources to determine if there has been a recent move.

Cross-checks also will be made against gold-standard CRM data, if available to the data supplier.

On the digital side, data will be checked for valid domain names, e-mail addresses, search preferences, and recency and frequency measures; the goal is to ensure that electronic activity actually represents live consumers who recently have exhibited the desired in-market behavior. Perhaps the most common bridge for identifying the same consumer in the online, mobile, and offline world is a validated e-mail address, preferably checked against first-party CRM data and/or registrations from trusted websites. A repository of credible e-mail addresses enables data suppliers and data intermediaries to tie smartphones and tablets, for example, to households and consumers through a digital footprint, like hashed e-mail.

As an example of how a data supplier might validate descriptive information about consumers (See Figure 6):

• The data firm cross-checks Consumer A's age, gender, residential address, and income against six sources of information. The firm found that three of its 20 client CRM databases contained information about Consumer A, and these were supplemented with externally sourced data from syndicated survey research, automotive registrations, digital publisher registrations, and television viewing data. Although some data disparities surfaced regarding age, address, and income, there was enough agreement across the sources to conclude that Consumer A was a 47-year-old male living at 123 Smith Street with a household income of more than $125,000.

General purpose third-party data firms—those offering core consumer data across multiple categories—sometimes conduct studies that benchmark the quality of

• Download figure
• Open in new tab
• Download powerpoint

Figure 6

Data Suppliers—Authenticating Consumers

• household size,

• education,

• home market value,

• household income,

• length of residence,

• gender and age,

the range of accuracy varied from a low of 60 percent to a high of 78 percent (See Figure 7) after removing records with missing data (See Appendix for calculation detail).

If accuracy rates were provided for each individual attribute, some scores likely would be much lower than the average range reported here. Typically household income and gender can present accuracy challenges; income because of its personal nature, and gender due to much of the data being collected at the household level, which must be attributed to household members.

Modeled Target Creation

Data suppliers were asked about the procedures used for developing segmentation/targeting models for deployment in look-alike targeting. Some data firms resisted this question, citing the need to maintain confidentiality due to the proprietary nature of the modeling techniques. Those who disclosed details did so in a very generic way.

In general, the most common approach was to identify the demographic and behavioral variables in their databases that exhibited the strongest relationship with the desired consumer activity. For example, a target description for those likely to take a European vacation during the next year might look something like this:

• Traveled to Europe in past 18 months

• Visited European travel sites in past 3 months

• Searched online for European travel destinations in past 3 months

• Read any travel magazine publications in past month

• Has frequent flyer membership with European airline

• Downloaded mobile travel apps

• Searched mobile apps for European travel destinations in past 3 months

• College education

• Household (HHI) $125,000-plus

• Attended graduate school

• Own nondomestic automobile

• Reside in the Northeast, Pacific, or Mid-Central census regions

• Speak a non-English, European language.

Data suppliers generally have access to thousands of behavioral and demographic variables that can be used to generate a target descriptor list that resembles the one above. As an example of the process for combing the database for target

• Download figure
• Open in new tab
• Download powerpoint

Figure 7

Competitive Data Study Example: Accuracy Rate (Attribute X)

Validating Modeled Target Segments

Drawing from multiple streams of consumer touch-point data, suppliers have the opportunity to develop propensity models that identify consumers who are likely to buy. Most data suppliers cited the use of traditional “holdout” testing to determine how well the modeled target will perform versus the benchmark of CRM/first-party or other actual sales data (See Figure 9). If the modeled target does not meet the performance standards of the first-party benchmarks, the data supplier likely will test new combinations of variables to improve the model's effectiveness. It's important to note that end users would like to see more transparency of the holdout test results so they can be confident that the modeled targets are driving results.

• Download figure
• Open in new tab
• Download powerpoint

Figure 8

Data Suppliers: Creating Modeled Targets

• Download figure
• Open in new tab
• Download powerpoint

Figure 9

Data Suppliers: Validating Modeled Targets

Monitoring Data Fluctuations

The blending of multiple data sources requires keeping an eye out for unusual fluctuations from one reporting period to the next to ensure that integrated data set metrics are sound and stable (See Figure 10). To that end, most data suppliers indicated that they have alert systems set up that trigger when unusual swings in key metrics occur across standard time-period reporting. These alerts are made operational in quality assurance processing software and are activated when a metric falls outside the acceptable range of established normative benchmarks.

Data suppliers stated that once a flag is triggered through their automated systems, a manual investigation takes place to determine the reason for the variation. After vetting the anomaly to understand whether there was an actual change in consumer description or behavior versus some type of error, the data firm will adjust its estimates accordingly.

Independent Testing Of Data Supplier Solutions

In general, data suppliers indicated that they test their modeled targeting and segmentation solutions, using techniques such as holdout samples, to ensure that modeled target performance exceeds the norm (See Figure 11). End users who want to assess performance differences across multiple data suppliers have the opportunity to test and compare the various solutions.

One data intermediary interviewed for the study stated that media agencies are in a unique position to objectively test data supplier offerings: “Agencies can look across the providers with an impartial eye, have the best vantage point for evaluation.” Although media agencies hold a distinct position for evaluating data supplier offerings, they also must meet certain logistics and resource requirements before they can take advantage of their vantage point:

• Analytic resources: staff time must be available to design and steward the testing process and results evaluation;

  

  • Download figure
  • Open in new tab
  • Download powerpoint

  Figure 10

  Data Consistency Checks
  

  • Download figure
  • Open in new tab
  • Download powerpoint

  Figure 11

  End Users: Independent Testing Of Data Supplier Solutions

• Number of data suppliers: usually dictated by advertiser client preference and user history, likely to number no more than two data supplier firms;

• Synchronized campaign timing: ideally, testing across multiple providers should occur at the same time mix, seasonality of response, special events, etc.;

• Matched testing technique: design and execution of test should be the same across all data supplier solutions;

• Multiple testing occasions: tests should be repeated to ensure help to understand changes in data supplier targeting or segmentation approach that may impact testing results.