ABSTRACT
Many previous studies on attention have ignored the eye-catching potential of “direct context”—the entire promotional and editorial content an observer can view at the same time—in print media. In the current study, characteristics of 183 magazine advertisements and their direct context were coded systematically and linked to eye-tracking data, producing more than 19,000 observations. Expanding on earlier research, the authors focused on fixations within an advertisement during the first five seconds and attention paid to the combined main elements of an advertisement. Results showed that direct context diverted visual attention, especially when featuring multiple colors and large amounts of text.
MANAGEMENT SLANT
Direct context in magazines influences attention paid to advertisement elements.
The effects of both the use of colors and of a large amount of text in the direct context is notable.
The most effective position for an advertisement is on the right-hand page, next to an article or illustration without too many colors.
Although the top of the page traditionally has been regarded as the most effective placement for an advertisement, the current study showed the opposite: Eye fixations were drawn to the bottom of the page.
Managers should consider these factors when pre-testing advertisements in a two-page setting and purchasing advertisement space.
INTRODUCTION
Attention has become one of the scarcest resources in the contemporary advertising economy (Davenport and Beck, 2001). Clutter is high; advertisements must compete with the content and context—editorial and promotional—surrounding them. For the purposes of the current study, which focuses on print media, the authors narrowed the meaning of context to “direct context”: the entire content an observer can view at the same time he or she views an advertisement. The reader, for example, may view two pages simultaneously—an article about wine on the left page of a magazine spread and an advertisement about color printing on the right. Would the wine article receive all the attention? Or would some elements in the advertisement be strong enough to engage the reader?
The current study examined the role of direct context in the competition for attention to magazine advertising. Previous research mainly has focused on attention to the elements of the advertisement; the current research expands on this by investigating the fuller context in which an advertisement appears that may divert attention away from the advertisement.
The current research started in 2009, when the authors began building a database of eye-tracking tests conducted between 2002 and 2009 by the research firm, GfK in the Netherlands (Boerman, Smit, and Van Meurs, 2010, 2011a, 2011b, 2011c). The eye-tracking data helped the authors explore what elements of the direct context of print advertisements were most arresting. They believe that this approach of integrating context in the study of visual attention may contribute to understanding why some advertisements are better than others at drawing a reader's attention.
In the current work, the definition of “attention” depends on
the level of processing that is taken into account (e.g., Finn, 1988; Greenwald and Leavitt, 1984; Smit, Neijens, and Heath, 2013) and
the measurement that is used (e.g., Appel, 1987, 1993; Smit and Neijens, 2011).
The authors of the current study focused on visual attention as stimulus-based low-level processing measured by eye tracking. The authors invoked an earlier perspective on visual attention as “the selective use of information from one region of the visual field at the expense of other regions of the visual field” (Henderson, 1992, p. 260). Eye-tracking measurement has provided reliable information regarding attention to advertisements (e.g., Pieters and Wedel, 2004; Wedel and Pieters, 2000). Scholars have concluded that people attend to what the eye is fixated on (Fox et al., 1998; Hyönä, 2010; Styles, 2006).
LITERATURE REVIEW
The authors of the current paper believe that scholarly work confirming the general effect of magazine context on advertisement attention is limited. Among the empirical findings is the notion that congruence between an entire magazine and an advertisement may have increased ad recognition (Moorman, Neijens, and Smit, 2002), yet lower levels of recall and recognition have been found when readers were more involved with a magazine (Norris and Colman, 1992).
These studies on the involvement of entire magazines did not incorporate the direct observable context. In a magazine, direct context refers to the two pages one can view at the same time. This direct context is the primary focus of the current study and has hardly been part of earlier research.
Using “Vote Count” to Analyze Related Research
Although not widely accepted among academics as a valid method of analysis, a “vote-count” procedure in a case such as the current study may help analyze limited findings on a particular research topic. In a vote-count procedure, reported relationships in earlier research are categorized in terms of “positive,” “negative,” and “non-significant” effects. Such a categorization is preferable over a statistical “effect-size” procedure when the number of comparable studies is low or when insufficient information is provided on sample-sized, raw data, and means scores (Bushman, 1994; Eisend, 2009).
In the current study, 26 published empirical studies were analyzed via the vote-count procedure. From that analysis, the authors categorized characteristics of print advertisements and direct context in magazines in terms of their relationship with consumers' attention to advertising.
The 26 studies were selected by means of a keyword and reference search in several databases (EBSCO, Communication & Mass Media Complete, PsycINFO, Sociological abstracts, Online Contents, Web of science, and Google Scholar). The following (combinations of) keywords were used:
(visual) attention,
advertising,
ad(vertisement),
eye movements,
eye tracking,
print,
magazine, and
context.
The reference search was conducted by addressing the references of the collected studies as they applied to different characteristics or independent variables. The 26 selected studies on attention to magazine advertising were published in peer-reviewed journals and edited volumes.
Among the 26 studies, 60 percent reported significant direct relationships between a print-advertisement characteristic and advertising attention (See Table 1). The different independent variables (characteristics) were combined in
ten objective advertisement characteristics,
eight subjective advertisement characteristics,
four context characteristics, and
two interaction variables.
The authors use the term “objective” to clarify that a characteristic (i.e., size or
Assessing the Measures
The eye-tracking measures in the published studies consisted of the number of fixations, fixation duration, viewing time or gaze duration, and viewing pattern. The authors also found scenarios in which attention was measured by memory measures, such as
recognition,
time spent in seconds, and
recall.
In the current study, “memory measures” and “eye tracking” refer to different levels of attention and processing (Finn, 1988; Greenwald and Leavitt, 1984; Smit et al., 2013). Whereas memory measures are indicators of active attention and high-level processing, eye tracking identifies passive attention or stimulus-based low-level processing. Because active attention and processing can occur only after passive attention has been established, these measures are necessarily related (Kahneman, 2011). The current authors, therefore, included these additional memory studies in their vote count.
Advertising Factors Observed in The Literature
Generally, more studies were conducted on magazine advertisements without considering their context than on advertisements including their context (See Table 1). Several well-known objective advertising factors have been observed to influence attention, most notably size (with the finding that “bigger is better”) and the use of multiple colors (e.g., Pieters and Wedel, 2004; Pieters, Wedel, and Batra, 2010; Zhang, Wedel, and Pieters, 2009).
The effect of color is not as straightforward as that of size, but it often has proved to have a positive effect on attention (e.g., Grønhaug, Kvitastein, and Grønmo, 1991).
In addition to size and color, the position of advertisement elements has been observed to attract attention. Although the right half of an advertisement is the best position for product shots, pictorials, headlines, and body copy (Rosbergen, Pieters, and Wedel, 1997), pictorials and text are best positioned on the left half of an advertisement (Kroeber-Riel and Barton, 1980). Moreover, pictorials and text in the upper half of an advertisement have been observed to attract more attention than pictorials and text in the lower half (Kroeber-Riel and Barton, 1980).
Different relationships between direct context and attention (DePelsmacker, Geuens, and Anckaert, 2002; Finn, 1988; Hanssens and Weitz, 1980; Malaviya, Kisielius, and Sternthal, 1996; Wedel and Pieters, 2000; Zhang et al., 2009) have shown that
magazine page placement matters; a page on the right of a (two-page) spread attracts more attention than on the left;
pages at the beginning of a magazine receive more attention than pages at the end; and
placement on the page in relation to other advertisements matters: Specifically, facing another advertisement or facing multiple advertisements is less distracting (from the advertiser's point of view) than placement opposite editorial content, especially when the context is not related to the advertisement (See Table 1).
RESEARCH QUESTION
In summary, previous scholarly work has found that at least the following factors should be taken into account when studying attention to magazine advertising:
page number (place in the magazine and right versus left page);
type of context (other advertisements and/or editorial content);
size of the advertisement;
position of the advertisement; and
use of color.
The current authors focused on these objective characteristics and not on other more subjective characteristics (e.g., “humor,” “originality”) in developing their coding scheme.
To understand the attracting and distracting power of the direct magazine context, the current authors asked
RQ1: How much visual attention is paid to the different elements of the advertisement and its direct magazine context?
METHODOLOGY
Materials
The current researchers measured eye movements using the Tobii Eyetracker, a device that estimates the point of gaze using image sensor technology that monitors the eyes and calculates the point of gaze.
Generally, eye tracking provides rich data about gaze behavior more accurately than questionnaires do and cannot be easily manipulated by the study participant (Duchowski, 2003; Holmqvist et al., 2011; Schrammel et al., 2011). The Tobii Eyetracker is similar to the device used in other parts of the literature (Pieters and Wedel, 2004, 2010). One advantage of the Tobii Eyetracker is that invisible near-infrared illuminators make this eye-tracking device unobtrusive; participants do not have to look at the stimuli through a glass sheet (Wedel and Pieters, 2000) and are not wearing an apparatus on their head (Kroeber-Riel, 1984; Kroeber-Riel and Barton, 1980; Rayner, Rotello, and Stewart, 2001; Rayner, Miller, and Rotello, 2008).
For the current study, the Netherlands office of GfK conducted a series of eye-tracking tests over a seven-year period (2002 to 2009). As participants were asked to read a magazine on a computer screen, their eye movements were recorded. The dataset of eye-tracking measurements used in this study consisted of 183 magazine advertisements plus their direct context. Each year of measurement was included in the analysis to control for a possible increase or decrease in attention over the years.
The types of magazines included
general consumer (66.5 percent);
medical journals (24.8 percent); and
management magazines (8.7 percent).
Most of the advertisements (154) covered one full page. The others were placed as follows:
Four covered half a page;
Three were smaller than half a page; and
22 were spreads covering two full pages.
The 22 spreads were excluded from the analyses because they did not have a direct context.
The most common product categories in the dataset were
medicines (21.7 percent);
food and non-alcoholic drinks (15.5 percent);
insurance (10.6 percent);
consumer electronics (8.1 percent); and
clothing and shoes (8.1 percent).
The magazine types did not differ in terms of visual attention (F(2, 156) = 0.002, p = 0.998; F(2, 146) = 0.64, p = 0.53) nor did product type (F(5, 153) = 1.53, p = 0.185; F(5, 143) = 1.01, p = 0.414).
Participants
The average number of participants per study was 105 (min = 60, max = 273). A full 68 tests were conducted with, on average, 2.7 advertisements per test. The total number of observations was 19,278.
In terms of gender and age, the respondents were distributed evenly, in terms of gender and age, among four groups in each eye-tracking test:
men between the ages of 18 and 39 years;
women ages 18 to 39;
men ages 40 to 65; and
women ages 40 to 65.
Procedure
Respondents were placed in front of the eye tracker and asked to read the full magazine as they would at home. During this exposure to the magazine, eye movements and fixations were recorded. There was no time limit; participants could take as long as they wanted to read the magazine. By clicking on a button, they could move back and forth within the magazine. Because they were asked to read only the magazine, participants were not primed to focus on the advertisements.
Dependent Variable: Visual Attention
Visual attention was measured by means of fixations within the magazine content. The eye-tracking scores were linked to different regions of the advertisements, the so-called four “Areas of Interest” (Pieters and Wedel, 2010; See Figure 1):
advertisement (a);
brand (b);
pictorial (p); and
text (t).
Two different eye-tracking scores were added to the advertisement database:
fixations within an advertisement during the first five seconds of viewing (Area of Interest [a], called “Ad Score”) and
fixations on three main advertisement elements (Areas of Interest [b], [p] and [t], called “Three-Element Score”).
A “fixation” is defined as “the state in which the eye remains in one specific point for at least 0.1 second.”
The current researchers thus had two measures of visual attention:
Ad Score is the percentage of all fixations within an advertisement during the first five seconds of viewing. These fixations were measured at the respondent level per advertisement but analyzed on an aggregated level (i.e., the average percentage of fixations within an advertisement during the first 5 seconds of viewing).
For example, if all respondents were to hold their eyes exclusively on an advertisement for the full five seconds, the Ad Score would be 100. Hence, the less attention paid within the first five seconds of viewing, the lower the Ad Score became. This score thus enabled the researchers to classify how well an advertisement was able to attract attention. In this study, advertisements attracted, on average, 45 percent of all fixations during the first five seconds of viewing (M = 44.67; SD = 15.17; Range = 16–90; Skewness = 0.73; Kurtosis = −0.16).
Three-Element Score is an index of the visual attention paid to the three main elements of an advertisement measured by the percentage of respondents that fixated at least once on at least parts of the brand and the pictorial imagery in the advertising and the accompanying text. The advertisements in this study had an average Three-Element Score of 34 percent (M = 34.20, SD = 17.74; Range = 1–75; Skewness = 0.25; Kurtosis = −0.65).
Both attention measures are based on fixations, consistent with the prior eye-tracking research considered in the current study's vote-count analysis (Pieter and Wedel, 2004, 2007, 2010; Rosbergen et al., 1997).
Contrary to prior studies, however, the current research did not focus on the fixation frequency or duration; instead, it used the percentage of fixations that were inside the advertisement within the first five seconds. This provided information about the attracting power of the advertisement and the possible distracting power of the context, as it also took into account all attention that was not inside the advertisement.
Additionally, although earlier research did take into account the brand, text, and pictorial elements (e.g., Pieter and Wedel, 2004, 2007, 2010; Rosbergen et al., 1997), those studies did not focus on the combination of these three elements. In contrast, the authors of the current study believes the new combination metrics provide an important contribution to earlier eye-tracking research.
Independent Variables: Characteristics and Coding
The context and advertisement characteristics scores were determined by content analysis using a coding scheme featuring 50 questions divided into five categories:
advertisement,
brand,
pictorial elements,
text elements, and
direct context (See Table 2).
Most of the coding was based on previous research examining product, size, color, position, content, and headline (See references in Table 1). Some coding was added: Specifically, the current authors found it interesting to know what brand was advertised and to code some specifics related to the eye-tracking measures (AOI, distance between elements, and text).
The coding scheme was pre-tested and discussed in advance with five experts—three from a publishing house and two from an advertising agency. Four graduate students were trained by one of the authors to code all 183 advertisements. The authors selected a random 20 percent sample of their material and had it coded double (that is, two times) to test inter-coder reliability (Neuendorf, 2002; Wimmer and Dominick, 1997; See Table 2). The overall inter-coder reliability was good (0.88); in other words, the coding was deemed reliable (See Appendix).1
When two coders disagreed on an instance of coding, one of the authors made the final decision by examining the material and choosing what he or she considered the best option. An online survey tool was used to enter the data of the content analysis (as if the coding scheme were a questionnaire). These data form the basis of the data file (supported by SPSS software) with advertisements as cases.
The digital versions of the advertisements and their context were uploaded into ImageMagick to calculate the JPEG properties with respect to the amount of red, green, and blue and the number of colors in the advertisement and the direct context. These data were added to the data file.
Analysis
The SPSS analysis (IBM Statistical Package for the Social Sciences software) comprised several steps:
Several variables were created from the content analysis, such as dummy variables for the ordinal variables;
correlation analysis was conducted; and
all significant bivariate relations from the previous step were combined into one final model for each dependent variable.
Some variables established in Step 2 were significantly correlated to only one of the two attention scores. In these cases, the variables still were included in both models to maintain comparability between models. Some independent variables in Step 2 were highly correlated. Of all pairs of variables with a correlation higher than 0.8, one was deleted to avoid multicollinearity.
The models were tested by linear regression analysis (method Enter). SPSS collinearity diagnostics showed no incidences of multicollinearity (tolerance above 0.20 and VIF below 5; see O'Brien, 2007).
RESULTS
Visual-Attention Models
Two regression models were tested, one for each attention measure: Model 1 (Ad Score) and Model 2 (Three-Element Score; See Table 3).
Both models fit the data well, with an adjusted R2 greater than 44 percent, indicating that advertisement and context characteristics were able to explain fixations during the first five seconds of viewing (Ad Score, Model 1) and combined fixations on the three main elements of the ad (Three-Element Score, Model 2).
Advertisement Characteristics. Overall, the models showed that, to a great extent, advertisement size (bigger was better) explained the variance in both visual attention scores, as did the number of colors in the advertisement (more colors resulted in more engagement) and the position of the pictorials (the lower the better; See Table 3). These results confirmed previous research (See references in Table 1) except for position of the pictorial in the advertisement. Although the top of the page traditionally has been regarded as the most effective placement for an advertisement (Rosbergen et al., 1997), the current study showed the opposite with regard to where readers' eyes tracked on the page. Eye fixations, in fact, were drawn to the bottom of the page.
Context Characteristics. Two of the context variables were notable:
the color in the context (the eyes go to the context instead of to the advertisement when multiple colors are part of this context) and
the page of the direct context (the eyes go to the context when the advertisement is placed on the left page and the context on the right page; see “Context Elements” and Table 3).
Advertisement Elements
Overall, advertisements measured by Ad Score, Model 1, attracted attention when the advertisements
were large,
contained many colors but not dominantly blue,
offered a visual image pictorial at the bottom of the advertisement, and
View this table:combined an illustration with a photo.
In measurements with the Three-Element Score, Model 2, larger advertisements also attracted more visual attention. Specifically,
Not only the total advertisement size but also the size of the text within the advertisements mattered;
multiple colors in the advertisements and the background color of the headline (black) attracted attention to the elements;
a low position of the pictorial was shown to attract visual attention to the three elements; and
attention to the elements increased when the distance between text and pictorials was not great and when a headline was included in an advertisement.
Context Elements
Several aspects of direct context were observed to engage readers and reduce the visual attention to the advertisement. In particular, the use of color in the context and the page of the tested advertisement (left-hand pages attracted less attention than right-hand pages) reduced the overall number of fixations on the advertisements (Ad Score, Model 1).
The results of the second regression analysis for visual attention to the three main elements of the advertisements (Three-Element Score, Model 2) confirmed the distracting effects of color and pagination.
In addition, two factors were determined to influence combined fixation to the three elements, which were not observed for the other visual attention measure. The amount of text in the direct context diminished the number of fixations on all three elements in the advertisements. In other words, text in the direct context reduced attention paid to the advertisement elements. In addition, the use of the color red in the direct context distracted overall fixation on the advertisements.
Attracting versus Distracting
Because editorial content and advertisements are created with different goals in mind, context and advertisement characteristics compete for the reader's attention.
A magazine advertisement most likely is designed to draw attention away from editorial content and to the advertisement, whereas editorial content is not designed to divert attention from advertisements but to inform or entertain its readers.
To test the attracting potential of both context and advertisements, separate regression models were run for each group of characteristics:
one including only the ad variable;
one including only the direct context variables.
The adjusted R2 for each model showed how much of the variance in the eye-tracking score is explained by the advertisement variables and how much by the context variables. Because the adjusted R2 was corrected for the number of variables, the researchers could discern which group explained more of the variance in the eye-tracking scores, even though the number of predictors differed between groups.
The results showed that the advertisement variables explained twice as much of the variance in visual attention to the three elements than the context variables (Adjusted R2 Ad versus Context: 0.45 versus 0.18). For the other visual attention score, this was not the case (Adjusted R2 Ad versus Context: 0.29 versus 0.23).
These results thus demonstrated that context potentially was a more powerful distractor for visual attention to the main elements of the advertisement (Three-Element Score) than for the more global visual attention to the advertisement during the first five seconds (Ad Score).
DISCUSSION
The current study sought to test which elements of the direct context of magazine advertisements catch the eye of the reader. Specifically,
RQ1: How much visual attention is paid to the different elements of the advertisement and its direct magazine context?
The results showed that, in some cases, several context characteristics such as size and position divert attention from advertisements. Overall, color in the context and the page of the context (right-hand versus left-hand page), in particular, distracted the reader from advertisements and to the three main elements:
brand,
pictorial, and
text.
The Three-Element Score also was affected negatively by the amount of text in the context and a dominance of the color red in the context.
The comparison of the distracting potential of the direct context with the attracting potential of an advertisement showed two things:
Direct context generally is a stronger distractor for visual attention when it concerns visual attention to the main elements of an advertisement; and
this is not the case when it concerns visual attention during the first five seconds: During these first moments, direct context and advertisement compete equally strongly.
The authors believe their findings contribute to the literature in two key ways:
The current study confirms the positive influence of some classic characteristics, such as advertisement size, position, and colors; and
the results clearly indicate what factors can drive passive attention, referred to as “bottom-up factors” (Pieters, Warlop, and Wedel, 2002); this is a valuable extension of previous research on attention, which mainly focused on active attention and self-reported memory measures.
LIMITATIONS AND FUTURE RESEARCH
To incorporate as many advertisements as possible into this study, the authors did not have the opportunity to consider user-related characteristics. Thus, no “top-down factors” were included.
Previous research has shown that the following “top-down” characteristics could be relevant variables to incorporate in future attention studies:
gender (Grønhaug et al., 1991);
user involvement (Petty, Cacioppo, and Schumann, 1983);
familiarity (Pieters et al., 2002); and
the goal of the user (Radach et al., 2003).
Further research should explore the relation between bottom-up and top-down factors and focus on more types of attention at the same time. This type of research calls for a combination of subconscious measurements, such as eye tracking, and such conscious measures as memory, attitude, and intention measures.
Despite its many advantages, eye tracking does have its constraints, especially concerning external validity. In particular, the experimental eye-tracking setup in which people read magazines must be taken into account. Although habits may change with the more common use of tablets, people usually do not read magazines on a screen. Moreover, in the experiment conducted in this study, participants were placed in front of a screen in an isolated research room where there were no distractions, a situation far removed from the usual living-room setting where people read at home.
Other eye-tracking studies have included measures such as the number of fixations, average number of fixations for each element, viewing patterns, and viewing times. The current study combined data from a large number of unique eye-tracking studies conducted over several years. To include as many studies as possible, the authors were able to include only eye-tracking scores that were available to all studies. This restricted their choice in eye-tracking measurements.
As a result, the authors were able to include only eye-tracking scores concerning the average attention paid during the first five seconds of viewing and whether participants fixated on the main elements. Thus, they were able to test whether the advertisement and specific elements of advertisements were able to attract attention. The use of more extensive eye-tracking scores, however, would have enabled the researchers to make more elaborate measurements and draw further conclusions.
CONCLUSION
To the authors' knowledge, the current research is the first eye-tracking study that also takes magazine direct context into account. Overall, context characteristics appeared to influence the visual attention paid to magazine advertisements, especially visual attention paid to the three main elements of advertisements. Color, page, and the amount of text in the direct context influenced the magazine reader and directed less visual attention to advertisements.
The effect of color especially was notable. The results showed that the eye fixates on advertisements with multiple colors but also on direct context represented in multiple colors. This color effect was observed for both types of visual attention.
Moreover, the results showed that the amount of red or blue makes a difference. Context in red directs less attention to the main elements of the advertisement and more to the context. Blue works differently: It also directs less attention to the advertisement but only during the first five seconds of viewing. This color effect confirmed the findings of previous studies on attention to web advertising (Diao and Sundar, 2004; Moore, Stammerjohan, and Coulter, 2005) and yellow-pages advertising (Lohse, 1997).
Advertisers could benefit from the current study's insights into specific advertisement characteristics, which may better engage the reading consumer, and context characteristics, which can divert the eye from advertisements. Understanding these characteristics could help advertisers in developing and pre-testing advertisements.
Furthermore, the authors believe that the results of the current study demonstrate the advantages of including color and to position the pictorial in the lower half of the advertisement. The distracting potential of the direct context implies that testing should benefit from showing a test advertisement in a two-page setting with direct context varying in color and amount of text.
When a certain context hinders visual attention to the tested advertisement, it may be beneficial for a marketer to invest additional funds to place a magazine advertisement in the best context. This could include placing it on a right-hand page next to an article or illustration without too many colors, (especially not too much red) and not too much text.
About the Authors
Edith G. Smit is professor of media and advertising at the Amsterdam School of Communication Research and dean of the Graduate School of Communication at the University of Amsterdam. She also is a member of the Dutch Dialogue Marketing Association privacy authority and past president of the European Advertising Academy. Her research focuses on advertising effectiveness, media multitasking, and tailored communication. Her work has appeared in the Journal of Advertising, Journal of Advertising Research, International Journal of Advertising, Computers in Human Behavior, and Journal of Business Research.
Sophie C. Boerman is assistant professor of persuasive communication at the Amsterdam School of Communication Research at the University of Amsterdam. Her research mainly focuses on the effects of embedded and/or hidden advertising on the use of persuasion knowledge and its persuasive outcomes. Her work has appeared in the Journal of Communication, Journal of Advertising, International Journal of Advertising, and Psychology & Marketing.
Lex van Meurs is research director at GfK (the Netherlands). He is responsible for the methodology and development of media research, such as television-audience measurement and the online television-appreciation panel. His research focuses on media research methodology, the online television appreciation panel, and outdoor advertising. His work has been published in the Journal of Advertising Research, Quality and Quantity, Communication Studies, Journal of Broadcasting & Electronic Media, International Journal of Public Opinion Research, and European Journal of Communication.
APPENDIX
For the specific variables and their inter-coder reliability scores, please see the article's Appendix online at http://warc.me/ACP1Sg.
Footnotes
↵1 For the specific variables and their inter-coder reliability scores, please see the article's Appendix online at http://warc.me/ACP1Sg. Cohen's Kappa was used for nominal and ordinal variables. Intra-class correlation was used for interval variables. Inter-coder reliability ranged between 0.43 and 1.00)
- © Copyright 2015 The ARF. All rights reserved.
REFERENCES
Note: The references of the studies considered in the current authors' vote count are included in the reference list with an asterisk (*) and refer to Table 1, with the first three letters of the first author and the last two digits of the year, for example, [ZHA09] means Zhang, Wedel, and Pieters (2009).