original.order Title Authors Venue Year URL Experiment_number Type Between Type Within Type Mixed Length (mins) Participant Prerequisites Colorblindness Pre-Test Qualification Training dummy questions Task Type Participants total Participants valid Participants per condition Female percentage USA only USA:India:Europe:other % Age reported Age range Age mean Platform for Recruitment Device & Software Restrictions Measure time Measure error Measure confidence Measure numeracy abilities Measure spatial abilities Measure other Vis static Vis interactive Payment type Payment Bonus Detect Inattentive Participants Prevent Multiple Participation Available Experiment material Available Collected Data 1 Exploring the impact of emotion on visual judgement Harrison, L.; Chang, R.; Aidong Lu VAST 2012 1 x - - NR NR NR NR NR NR Which, A or B is SMALLER , What percentage is the SMALLER of the LARGER, (judge comparing chart values) 963 664/197 (only valid answers with specific properties have been taken into accout for analysis) 30 NR NR NR NR NR NR AMT - - x - - - - x - Per Trial 0.25 - NR - - - 2 Priming Locus of Control to affect performance Ottley, A.; Crouser, R.J.; Ziemkiewicz, C.; Chang, R. VAST 2012 1 - x - NR NR NR NR NR NR search inferential 300 229 NR NR NR NR NR NR NR AMT - x x - - - - x - NR - - NR - - - 3 Graphical Tests for Power Comparison of Competing Designs Hofmann, H.; Follett, L.; Majumder, M.; Cook, D. TVCG 2012 1 - x - NR NR NR NR NR NR Find outlier view in set of views 115 NR NR NR NR NR NR NR NR AMT - x x x - - - x - NR - - NR - - - 4 Graphical Tests for Power Comparison of Competing Designs Hofmann, H.; Follett, L.; Majumder, M.; Cook, D. TVCG 2012 2 - x - NR NR NR NR NR NR Find outlier view in set of views 208 NR NR NR NR NR NR NR NR AMT - x x x - - - x - NR - - NR - - - 5 Learning Layouts for Single-PageGraphic Designs O'Donovan, P.; Agarwala, A.; Hertzmann, A. TVCG 2012 1 - x - NR NR NR NR NR NR Sketch important regions 35 NR NR NR NR NR NR NR NR AMT - - - - - - - x - NR - NR - - - 6 Learning Layouts for Single-PageGraphic Designs O'Donovan, P.; Agarwala, A.; Hertzmann, A. TVCG 2012 2 - x - NR NR NR NR NR NR Subjective evaluation of designs 45 NR NR NR NR NR NR NR NR AMT - - - - - - - x x Per Trial 0.05 - NR - - - 7 Learning Layouts for Single-PageGraphic Designs O'Donovan, P.; Agarwala, A.; Hertzmann, A. TVCG 2012 3 - x - NR NR NR NR NR NR Subjective evaluation of designs 45 NR NR NR NR NR NR NR NR AMT - - - - - - - x - Per Trial 0.05 - NR - - - 8 Does an Eye Tracker Tell the Truth about Visualizations?: Findings while Investigating Visualizations for Decision Making Sung-Hee Kim; Zhihua Dong; Hanjun Xian; Upatising, B.; Ji Soo Yi TVCG 2012 1 x - - NR NR NR NR Active NR Compare values across rows and column and make select best option 176 100 NR 48 NR NR Y 18 - 56 NR AMT - x x - - - - - x Per Trial 0.23 random bonus during experiment NR - - - 9 Perceptual Guidelines for Creating Rectangular Treemaps Kong, N.; Heer, J.; Agrawala, M. TVCG 2012 1 - x - NR NR NR NR NR NR Compare sizes of rectangles 41 5% removed outlier trails (more than 35%errors) NR NR NR NR NR NR NR AMT - - x - - - - x - Per Trial 0.03 - NR - - - 10 Perceptual Guidelines for Creating Rectangular Treemaps Kong, N.; Heer, J.; Agrawala, M. TVCG 2012 2 - x - NR NR NR NR NR NR Compare sizes of rectangles 104 .7% with errors more than 35% NR NR NR NR NR NR NR AMT - - x - - - - x - Per Trial 0.03 - NR - - - 11 Perceptual Guidelines for Creating Rectangular Treemaps Kong, N.; Heer, J.; Agrawala, M. TVCG 2012 3 - x - NR NR NR NR NR NR 432 4.5% trails removed with error above 70% or estimation times greater than 70% NR NR NR NR NR NR NR AMT - - x - - - - x - Per Trial 0.03 - NR - - - 12 Interactive querying of temporal data using a comic strip metaphor J Jin, P Szekely VAST 2010 1 - x - NR NR NR NR Active NR Create a visual query that answers a given question (e.g. How many students submitted a paper within 60 days after proposal?) 50 42 NR NR NR NR NR NR NR NR - x x - - - - - x NR NR - NR - - - 13 Human computation in visualization: Using purpose driven games for robust evaluation ofvisualization algorithms N Ahmed; Z Zheng; K Mueller TVCG 2012 1 - x - 5 NR NR NR NR NR Play the game 261 NR NR NR NR NR NR NR NR Other - - x - - - - - x NR NR - NR - - - 14 How locus of control influences compatibility with visualization style C Ziemkiewicz; RJ Crouser, AR Yauilla VAST 2011 1 x - - NR NR NR NR NR NR search inferential 240 NR NR 47 NR NR Y 18-63 26.7 AMT - x x - - - - - x NR x y NR - - - 15 Perception of Average Value in Multiclass Scatterplots Gleicher, M.; Correll, M.; Nothelfer, C.; Franconeri, S. TVCG 2013 1 x - - 10 NR NR NR NR NR find clusters in scatterplot 352 312 32 42 Y 100:00:00 Y 18-65 32.6 AMT - - x - - - - x - Per Hour 6 - NR - - - 16 Perception of Average Value in Multiclass Scatterplots Gleicher, M.; Correll, M.; Nothelfer, C.; Franconeri, S. TVCG 2013 2 - x - 15 NR NR NR NR NR find clusters in scatterplot 319 NR NR 42 Y 100:00:00 Y 18-65 32.6 AMT - - x - - - - x - Per Hour 6 - NR - - - 17 Ranking Visualizations of Correlation Using Weber's Law Harrison, L.; Fumeng Yang; Franconeri, S.; Chang, R. TVCG 2014 1 x - - NR NR NR NR Active NR correlation in scatterplots 88 NR 30 36 NR NR NR NR NR AMT mobile deviced blocked x x - - - - x - Per study 2.1 - NR x - - 18 Ranking Visualizations of Correlation Using Weber's Law Harrison, L.; Fumeng Yang; Franconeri, S.; Chang, R. TVCG 2014 2 x - - NR NR NR NR Active NR correlation in other visualizations 1,687 NR 30 49.4 NR NR NR NR NR AMT mobile deviced blocked x x - - - - x - Per study 2.1 - NR x - - 19 How Hierarchical Topics Evolve in Large Text Corpora Weiwei Cui; Shixia Liu; Zhuofeng Wu; Hao Wei TVCG 2014 1 - x - NR >=95% HIT approval rate NR none NR NR find elements in network 621 597 300 NR NR NR NR NR NR AMT - x x - - - - x - Per Trial 0.5 - NR x - - 20 Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error Correll, M.; Gleicher, M. TVCG 2014 1 - - x 8 NR NR NR NR NR error estimation in bar charts 96 NR 8 56 Y 100:00:00 NR NR 33.3 AMT - - x x - - - x - NR NR - NR - - http://graphics.cs.wisc.edu/ Vis/ErrorBars. 21 Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error Correll, M.; Gleicher, M. TVCG 2014 2 - - x 8 NR NR NR NR NR error estimation in bar charts 48 NR 8 56 Y 100:00:00 NR NR 33.3 AMT - - x x - - - x - NR NR - NR - - http://graphics.cs.wisc.edu/ Vis/ErrorBars. 22 Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error Correll, M.; Gleicher, M. TVCG 2014 3 - - x 8 NR NR NR NR NR error estimation in bar charts 96 NR 8 56 Y 100:00:00 NR NR 33.3 AMT - - x x - - - x - NR NR - NR - - http://graphics.cs.wisc.edu/ Vis/ErrorBars. 23 The persuasive power of data visualization AV Pandey; A Manivannan; O Nov TVCG 2014 1 x - - 7.5 >=99% HIT approval rate NR NR NR NR asses true and false answers for statements referring to infographics 240 NR NR NR Y 100:00:00 NR NR NR AMT block browser back-button - x - - - - x - Per study 0.5 - NR x - - 24 The persuasive power of data visualization AV Pandey; A Manivannan; O Nov TVCG 2014 2 x - - 7.5 >=99% HIT approval rate NR NR NR NR asses true and false answers for statements referring to infographics 240 NR NR NR Y 100:00:00 NR NR NR AMT block browser back-button - x - - - - x - Per study 0.5 - NR x - - 25 The persuasive power of data visualization AV Pandey; A Manivannan; O Nov TVCG 2014 3 x - - 7.5 >=99% HIT approval rate NR NR NR NR asses true and false answers for statements referring to infographics 240 NR NR NR Y 100:00:00 NR NR NR AMT block browser back-button - x - - - - x - Per study 0.5 - NR x - - 26 How to display group information on node-link diagrams: an evaluation R Jianu; A Rusu; Y Hu; D Taggart TVCG 2014 1 x - - 2 NR NR introductionary example Active NR group tasks, network tasks, mixed group-network tasks 788 NR 45 NR NR NR NR NR NR AMT - - x - - - - x - NR NR - NR - - - 27 Four Experiments on the Perception of Bar Charts J Talbot; V Setlur; A Anand TVCG 2014 1 x - - NR >=99% HIT approval rate NR NR Active NR 50 NR NR NR NR NR Y 18-65 33.3 AMT - x x - - - - x - Per Trial 0.02 - NR - - - 28 Four Experiments on the Perception of Bar Charts J Talbot; V Setlur; A Anand TVCG 2014 2 x - - NR >=99% HIT approval rate NR NR Active NR 50 NR NR NR NR NR Y 18-65 33.3 AMT - x x - - - - x - Per Trial 0.02 - NR - - - 29 Four Experiments on the Perception of Bar Charts J Talbot; V Setlur; A Anand TVCG 2014 3 x - - NR >=99% HIT approval rate NR NR Active NR 50 NR NR NR NR NR Y 18-65 33.3 AMT - x x - - - - x - Per Trial 0.02 - NR - - - 30 Four Experiments on the Perception of Bar Charts J Talbot; V Setlur; A Anand TVCG 2014 4 x - - NR >=99% HIT approval rate NR NR Active NR 50 NR NR NR NR NR Y 18-65 33.3 AMT - x x - - - - x - Per Trial 0.02 - NR - - - 31 The Perception of Visual UncertaintyRepresentation by Non-Experts S Tak; A Toet; J van Erp TVCG 2014 1 NR NR NR NR NR NR numercy scale [18] higher than 4.4 NR NR estimate uncertainty through visual encodings 140 NR NR 51 NR NR Y 18-67 30.6 NR - x x - x - - x - NR NR - NR - - - 32 PEARL: an interactive visual analytic tool for understanding personal emotion style derived from social media J Zhao; L Gou; F Wang; M Zhou VAST 2014 1 x - - 21 NR NR NR Passive NR 50 44 NR NR N 33:NR:NR NR NR NR AMT - x x - - - - x - NR NR - NR - - - 33 Lightness Constancy in Surface Visualization Szafir, D.A.; Sarikaya, A.; Gleicher, M. TVCG 2015 1 - x - NR >=99% HIT approval rate Colorblindness Test NR NR NR color matching in conditions involving shading 17 15 NR 40 NR NR Y 18-65 31.25 AMT - - x - - - - x - NR NR - NR x - - 34 Lightness Constancy in Surface Visualization Szafir, D.A.; Sarikaya, A.; Gleicher, M. TVCG 2015 2 - x - NR >=99% HIT approval rate Colorblindness Test NR NR NR color matching in conditions involving shading 18 15 NR 40 NR NR Y 18-65 31.25 AMT - - x - - - - x - NR NR - NR x - - 35 Lightness Constancy in Surface Visualization Szafir, D.A.; Sarikaya, A.; Gleicher, M. TVCG 2015 3 - x - NR >=99% HIT approval rate Colorblindness Test NR NR NR color matching in conditions involving shading 34 31 15 40 NR NR Y 18-65 31.25 AMT - - x - - - - x - NR NR - NR x - - 36 Lightness Constancy in Surface Visualization Szafir, D.A.; Sarikaya, A.; Gleicher, M. TVCG 2015 4 - x - NR >=99% HIT approval rate Colorblindness Test NR NR NR color matching in conditions involving shading 92 90 30 40 NR NR Y 18-65 31.25 AMT - - x - - - - x - NR NR - NR x - - 37 Lightness Constancy in Surface Visualization Szafir, D.A.; Sarikaya, A.; Gleicher, M. TVCG 2015 5 - x - NR >=99% HIT approval rate Colorblindness Test NR NR NR color matching in conditions involving shading 108 100 25 40 NR NR Y 18-65 31.25 AMT - - x - - - - x - NR NR - NR x - - 38 Guidelines for Effective Usage of Text Highlighting Techniques Strobelt, H.; Oelke, D.; Bum Chul Kwon; Schreck, T.; Pfister, H. TVCG 2016 1 N Y N 35 10,000 or more HITs approved, 99% HIT Approval Rate Colorblindness Test NR Passive NR Find highlights (as many as possible) 63 45 48.9 NR Y 20-60+ NR AMT PC only (excluded 2 tablet users) N N N N N the number of highlighted items Y N Per study 1.5 NR NR Source code, the test system Y 39 Guidelines for Effective Usage of Text Highlighting Techniques Strobelt, H.; Oelke, D.; Bum Chul Kwon; Schreck, T.; Pfister, H. TVCG 2016 2 N Y N 73 10,000 or more HITs approved, 99% HIT Approval Rate Colorblindness Test NR Active NR Find all highlights of a type A 38 30 53.3 NR Y -20-60+ (all) NR AMT PC only N N N N N the number of highlighted items Y N Per study 3 NR NR Source code, the test system Y 40 Guidelines for Effective Usage of Text Highlighting Techniques Strobelt, H.; Oelke, D.; Bum Chul Kwon; Schreck, T.; Pfister, H. TVCG 2016 3 N Y N 63 10,000 or more HITs approved, 99% HIT Approval Rate Colorblindness Test NR NR NR Find only the overlap of highlights 34 24 37.5 NR Y -20-60 NR AMT PC only N N N N N comparison to study 1 Y N Per study 2.5 NR NR Source code, the test system Y 41 Suggested Interactivity: Seeking Perceived Affordances for InformationVisualization Boy, J.; Eveillard, L.; Detienne, F.; Fekete, J.-D. TVCG 2016 1 N N N NR 1000 or more HITs approved, 98% acceptance rate NR intermediate English reading comprehension test Passive NR fact-checking 70 59 NR NR N AMT PC only (excluded 2 mobile users) N N N N N score, interaction N Y NR NR NR total score <= 0 NR NR N 42 Suggested Interactivity: Seeking Perceived Affordances for InformationVisualization Boy, J.; Eveillard, L.; Detienne, F.; Fekete, J.-D. TVCG 2016 2 N N N NR NR NR intermediate English reading comprehension test Passive NR fact-checking 70 47 NR NR N AMT PC only N N N N N score, interaction N Y NR NR NR total score <= 0 NR NR N 43 Suggested Interactivity: Seeking Perceived Affordances for InformationVisualization Boy, J.; Eveillard, L.; Detienne, F.; Fekete, J.-D. TVCG 2016 3 N N N NR NR NR intermediate English reading comprehension test Passive NR fact-checking 70 51 NR NR N AMT PC only N N N N N score, interaction N Y NR NR NR total score <= 0 NR NR N 44 Suggested Interactivity: Seeking Perceived Affordances for InformationVisualization Boy, J.; Eveillard, L.; Detienne, F.; Fekete, J.-D. TVCG 2016 4 Y N N NR NR NR intermediate English reading comprehension test Passive NR fact-checking 120 108 33 in G1, 35 in G2, 40 in G3 NR NR N AMT PC only N N N N N score, interaction N Y NR NR NR total score <= 0 NR NR N 45 Improving Bayesian reasoning: the effects of phrasing, visualization, and spatial ability A Ottley; EM Peck; LT Harrison; TVCG 2016 1 Y N N NR NR NR NR NR NR Bayesian reasoning 100 100 37 in C1, 30 in C2, 33 in C3 35 NR Y 19~65 33.63 AMT NR Y (measured but not compared) Y N N N Y N Per study + Per correct trial $.50 (base) $.50 per correct AMT worker ID NR Y 46 Improving Bayesian reasoning: the effects of phrasing, visualization, and spatial ability A Ottley; EM Peck; LT Harrison; TVCG 2016 2 Y N N NR NR NR NR NR NR Bayesian reasoning 377 377 61-65 34.2 NR Y 18~65 31 AMT NR Y (measured but not compared) Y N Y Y Y N Per study + Per correct trial $.50 (base) $.50 per correct AMT worker ID NR Y 47 Map LineUps: effects of spatial structure on graphical inference Beecham, Roger; Dykes, Jason ;Meulemans, Wouter ; Slingsby, Aidan; Turkay, Cagatay ; Wood, Jo TVCG 2016 1 N N Y 18 (median) 10,000 # HITs approved, 99% HIT Approval Rate NR NR Active NR Judge spatial autocorrelation; Just Noticable Difference 361 30 42 NR N AMT NR NR Y N N N just noticeable difference (JND) Y N Per study $2.18 NR NR Source code Y 48 Optimizing Hierarchical Visualizations with the Minimum Description Length Principle Veras, Rafael ;Collins, Christopher TVCG 2016 1 N N Y 11 (median) NR NR NR Active NR follow a path NR NR NR NR NR NR CrowdFlower NR Y N N N N number of drill-down interaction N Y Per study $2.00 NR NR NR NR N 49 Evaluating the Impact of Binning 2D Scalar Fields Lace Padilla; P. Samuel Quinan; Miriah Meyer; Sarah H. Creem-Regehr TVCG 2016 1 N N Y NR NR NR master-class workers NR NR discovery-based tasks NR NR NR NR NR NR AMT NR Y Y Y N N Y N NR NR NR NR NR Screenshots of each task, display questions N 50 HindSight: Encouraging Exploration through Direct Encoding of Personal Interaction History Mi Feng; Cheng Deng; Evan M. Peck; Lane Harrison TVCG 2016 1 Y N N NR NR NR NR NR NR Explore & Report Insights 92 NR 44 in control, 48 in hindsight NR NR NR AMT NR Y N N N N insights visited : the number of unique charts that a person directly interacts with during exploration.revisited : the number of instances when a user interacts with a previously visited chart.exploration time : the total amount of time spent interacting withcharts. mentions : the number of times a chart is directly referenced in findings during the Insight phase of our experiment. N Y Per study $1.00 NR NR NR all experimental material, analyses script Y 51 HindSight: Encouraging Exploration through Direct Encoding of Personal Interaction History Mi Feng; Cheng Deng; Evan M. Peck; Lane Harrison TVCG 2016 2 Y N N NR NR NR NR NR NR Explore & Report Insights 116 NR 57 in control, 59 in hindsight NR NR NR AMT NR Y N N N N insights visited : the number of unique charts that a person directly interacts with during exploration.revisited : the number of instances when a user interacts with a previously visited chart.exploration time : the total amount of time spent interacting withcharts. mentions : the number of times a chart is directly referenced in findings during the Insight phase of our experiment. N Y Per study $1.00 NR NR NR all experimental material, analyses script Y 52 HindSight: Encouraging Exploration through Direct Encoding of Personal Interaction History Mi Feng; Cheng Deng; Evan M. Peck; Lane Harrison TVCG 2016 3 Y N N NR NR NR NR NR NR Explore & Report Insights 206 NR 99 in contorl 99, 107 in hindsight NR NR NR AMT NR Y N N N N visited : the number of unique charts that a person directly interacts with during exploration. revisited : the number of instances when a user interacts with a previously visited chart. exploration time : the total amount of time spent interacting withcharts. mentions : the number of times a chart is directly referenced in findings during the Insight phase of our experiment. N Y Per study $1.00 NR NR NR all experimental material, analyses script Y 53 A Study On Designing Effective Introductory Materials for Information Visualization Yuzuru Tanahashi; Nick Leaf; Kwan-Liu Ma CGF 2016 1 x - - 20 AMT Approval rate >= 95%, > 50 HITS completed, NR NR NR NR Graph compare, storyline identify, scatter plot identify , treemap compare 800 Some random clickers were cut, but amount not specified 40 NR NR NR NR NR NR AMT NR Y, but did not use it for eval Y N N N NR N Y (but very limited) Per Correct Trial $0.05 for 0–3, $0.80 for 9–10 correct answers. Max equivalent of $1.60 , min equivalent of $0.15 Pay scaled based on number of correct answers NR NR N N 54 An Evaluation of the Impact of Visual Embellishments in Bar Charts Drew Skau; Lane Harrison; Robert Kosara EuroVis 2015 1 N Y N 19 NR NR NR Active Y absolute (value) 103 94 94 NR Y 100:00:00 NR NR NR AMT NR Y Y N N N N Y N Per study $2.00 NR Other NR N N 55 An Evaluation of the Impact of Visual Embellishments in Bar Charts Drew Skau; Lane Harrison; Robert Kosara EuroVis 2015 2 N Y N 19 NR NR NR Active Y relative (percentage) 103 94 94 NR Y 100:00:00 NR NR NR AMT NR Y Y N N N N Y N Per study $2.00 NR Other NR N N 56 GraphUnit: Evaluating Interactive Graph Visualizations Using Crowdsourcing Mershack Okoe; Radu Jianu EuroVis 2015 http://onlinelibrary.wiley.com/doi/10.1111/cgf.12657/pdf 1 N Y N NR NR NR NR NR NR # of connected nodes 112 112 112 NR NR NR NR NR NR AMT Y Y N N N N Y N Per study $0.50 NR NR NR N N 57 GraphUnit: Evaluating Interactive Graph Visualizations Using Crowdsourcing Mershack Okoe; Radu Jianu EuroVis 2015 http://onlinelibrary.wiley.com/doi/10.1111/cgf.12657/pdf 2 N Y N NR NR NR NR NR NR connectivity (1 & 2 hops) 62 62 62 NR NR NR NR NR NR AMT Y Y N N N N Y N Per study $0.55 NR NR NR N N 58 Interaction with uncertainty in visualisations Bertini, E and Kennedy, J and Puppo, E Eurovis-SP 2014 1 N Y N NR NR NR NR Passive NR describe what they saw & reduce the uncertainty (task to evaluate an interface) 39 39 39 NR NR English-speaking countries NR NR NR Other NR Y Y N N N number of drag and drop actions N Y NR NR NR NR NR N N 59 Preconceptions and Individual Differences in Understanding Visual Metaphors Caroline Ziemkiewicz; Robert Kosara EuroVis 2009 1 N N Y 60 Color Blindness, 20/20 full color vision and be able to read and write in English [everything self-reported] Self-reported NR Active NR yes-or-no questions 63 63 63 63.50 NR NR Y 18 to 54 30.6 AMT NR Y Y N N Y reading time, personality, metaphor interpretation Y N Per study $0.50 up to $2.50 NR NR N N 60 Selecting Semantically-Resonant Colors for Data Visualization Sharon Lin; Julie Fortuna; Chinmay Kulkarni; Maureen Stone; Jeffrey Heer Eurovis 2013 1 N N Y NR Full color vision Self-reported NR Active NR which is larger? Color assignments, rate the strength of color-value association 140 140 140 53 Y 100:00:00 NR NR NR AMT NR Y Y N N N N Y N Per study $1.00 NR NR NR Y Y 61 Selecting Semantically-Resonant Colors for Data Visualization Sharon Lin; Julie Fortuna; Chinmay Kulkarni; Maureen Stone; Jeffrey Heer Eurovis 2013 2 Y N N NR Full color vision Self-reported NR Active NR binary forced-choice questions on quantities (individual bars, 2 combination of bars) 302 302 302 74 Y 100:00:00 NR NR NR AMT NR Y Y N N N N Y N Per study $2.00 NR NR NR Y Y 62 Arcs, Angles, or Areas: Individual Data Encodings in Pie and Donut Charts Drew Skau;Robert Kosara EuroVis 2016 1 N Y N 25 NR NR NR Active NR area percent estimation 102 92 102 47.00 NR NR Y 25-29, 35-39 NR AMT NR Y Y Y N N N Y N Per study $3.00 NR Other NR N N 63 Arcs, Angles, or Areas: Individual Data Encodings in Pie and Donut Charts Drew Skau;Robert Kosara EuroVis 2016 2 N Y N 16 Education level collected but not reported NR NR Active NR area percent estimation 117 93 117 36.00 NR NR Y 25-29, 30-39 NR AMT NR Y Y Y N N N Y N Per study $2.00 NR Other NR N N 64 How Ordered Is It? On the Perceptual Orderability of Visual Channels David H. S. Chung; Daniel Archambault; Rita Borgo; Darren J. Edwards; Robert S. Laramee; Min Chen EuroVis 2016 1 N N Y 8 NR Colorblindness Test self-reported Active NR How ordered is it? 115 110 115 44 NR NR NR NR NR CrowdFlower Y Y Y Y N N N Y N Per study $1.00 N Catch questions relevant to tasks, Task completion time threshold, Other Y N N 65 How Ordered Is It? On the Perceptual Orderability of Visual Channels David H. S. Chung; Daniel Archambault; Rita Borgo; Darren J. Edwards; Robert S. Laramee; Min Chen EuroVis 2016 2 N Y N 8 NR Colorblindness Test self-reported Active NR Which is smallest? Which is largest? 88 87 88 52 NR NR NR NR NR CrowdFlower Y Y Y Y N N N Y N Per study $1.00 N Catch questions relevant to tasks, Task completion time threshold, Other Y N N 66 Judgement Error in Pie Chart Variations Robert Kosara; Drew Skau EuroVis Short Paper 2016 1 N Y N 11 NR NR N Passive NR area percent estimation 108 107 107 50 NR NR NR 30-39 NR AMT NR Y Y N N N N Y N Per study $2.00 N NR N Y Y 67 Using icicle trees to encode the hierarchical structure of source code I. Bacher;B. Mac Namee; J. D. Kelleher EuroVis Short Paper 2016 1 Y N N 30 NR NR N Passive NR How many child/descendant/leaf nodes does node “X” contain and what node is the closest common ancestor of nodes “X” and “Y” 39 39 20 NR NR NR NR NR NR Other NR Y Y N N N N Y N NR NR N NR NR N N 68 Evaluating Viewpoint Entropy for Ribbon Representation of Protein Structure J. Heinrich; J. Vuong; C. J. Hammang; A. Wu; M. Rittenbruch; J. Hogan; M. Brereton; S. I. O’Donoghue EuroVis 2016 1 N Y N 15 NR NR N NR NR Image selection - viewpoint preference. 13 10 10 NR NR NR NR NR NR AMT NR Y Y N N N N Y N Per Trial $0.02 per trial, $8 per hr (160 trials total) N Other NR N N 69 Evaluating Viewpoint Entropy for Ribbon Representation of Protein Structure J. Heinrich; J. Vuong; C. J. Hammang; A. Wu; M. Rittenbruch; J. Hogan; M. Brereton; S. I. O’Donoghue EuroVis 2016 2 N Y N 125 NR NR N NR NR Image selection - viewpoint preference. 104 65 65 NR NR NR NR NR NR AMT NR Y Y N N N N Y N Per Trial $0.02 per trial, $8 per hr (160 trials total) N Other NR N N 70 Mimic: visual analytics of online micro-interactions Simon Breslav; Azam Khan; Kasper Hornbæk AVI 2014 http://wallviz.dk/wp-content/uploads/2014/05/AVI2014Breslav.pdf 1 Y N N 5 NR NR AMT HIT approval rate > 95% NR NR Decision making 400 NR 200 NR NR NR N NR NR AMT Y Y Y Y N N NR Y N Per study $0.40 NR Catch questions relevant to tasks NR N N 71 Progressive Parallel Coordinates Rene Rosenbaum, Jian Zhi, Bernd Hamann PacificVis 2012 http://www.meecoda.de/scientific/publications/Rosenbaum-PV12.pdf 1 N Y N NR NR NR NR NR NR Pattern detection 43 40 40 NR NR N AMT NR N Y N N N 1) refinement level and 2) preference Y N NR NR NR arbitrary answers NR NR NR 72 Crowdsourcing graphical perception: using mechanical turk to assess visualization design Heer, Jeffrey;Michael Bostock SIGCHI 2010 1a N Y N NR NR NR NR Active NR proportional judgement 82 NR 50 NR NR NR N NR NR AMT N Y Y N N N NR Y N Per Trial $0.05 NR NR NR N N 73 Crowdsourcing graphical perception: using mechanical turk to assess visualization design Heer, Jeffrey;Michael Bostock SIGCHI 2010 1b N Y N NR NR NR NR Active NR rectangular area judgement 117 NR 75 NR NR NR N NR NR AMT N Y Y N N N NR Y N Per Trial $0.02 NR NR NR N N 74 Crowdsourcing graphical perception: using mechanical turk to assess visualization design Heer, Jeffrey;Michael Bostock SIGCHI 2010 2 N Y N NR NR NR NR Active NR gridline alpha contrast 117 NR 75 NR NR NR N NR NR AMT N Y Y N N N NR N Y Per Trial $0.02 NR NR NR N N 75 Crowdsourcing graphical perception: using mechanical turk to assess visualization design Heer, Jeffrey;Michael Bostock SIGCHI 2010 3 N Y N NR NR NR NR Active NR chart size and gridline spacing 117 NR 75 NR NR NR N NR NR AMT N Y Y N N N NR Y N Per Trial $0.02, $0.04 NR NR NR N N 76 The impact of social information on visual judgments Hullman, Jessica; Eytan Adar; Priti Shah SIGCHI 2011 http://www.cond.org/hullman_adar_shah_camera_ready.pdf 1 N N Y NR NR NR NR NR NR proportional judgement 100 NR 50 NR NR NR N NR NR AMT N N Y N N N NR Y N Per Trial $0.05, $0.08 NR NR NR N N 77 The impact of social information on visual judgments Hullman, Jessica; Eytan Adar; Priti Shah SIGCHI 2011 http://www.cond.org/hullman_adar_shah_camera_ready.pdf 2 N N Y NR NR NR NR NR NR linear association (correlation) estimation 100 NR 50 NR NR NR N NR NR AMT N N Y N N N NR Y N Per Trial $0.10 NR NR NR N N 78 The impact of social information on visual judgments Hullman, Jessica; Eytan Adar; Priti Shah SIGCHI 2011 http://www.cond.org/hullman_adar_shah_camera_ready.pdf 3 N N Y NR NR NR NR NR NR perceptual judgement (with information cascade) 50 NR 5 NR NR NR N NR NR AMT N N Y N N N NR Y N Per Trial $0.08 NR NR NR N N 79 Playable data: characterizing the design space of game-y infographics Diakopoulos, Nicholas; Funda Kivran-Swaine; Mor Naaman SIGCHI 2011 http://www.nickdiakopoulos.com/wp-content/uploads/2007/05/paper1257-diakopoulos.pdf preliminary Y N N NR NR NR NR NR NR play and usability testing -> explore & report insights + subjective impressions 147 127 47,41,39 64 NR NR Y 19-57 29 NR NR N N N N N interaction log N Y Raffle $50 NR NR NR N N 80 Strategies for crowdsourcing social data analysis Wesley Willett?; Jeffrey Heer†; Maneesh Agrawala? SIGCHI 2012 http://vis.stanford.edu/files/2012-CrowdAnalytics-CHI.pdf 1 Y N N NR NR NR NR NR NR explain a visualization NR [910 for both experiments] NR NR NR NR NR N NR NR AMT N Y Y N N N N Y N Per Trial $0.05,$0.2 NR NR N N N 81 Strategies for crowdsourcing social data analysis Wesley Willett?; Jeffrey Heer†; Maneesh Agrawala? SIGCHI 2012 http://vis.stanford.edu/files/2012-CrowdAnalytics-CHI.pdf 2 Y N N NR NR NR NR NR NR explain a visualization NR [910 for both experiments] NR NR NR NR NR N NR NR AMT N Y Y N N N N Y N NR NR NR NR N N N 82 Strategies for crowdsourcing social data analysis Wesley Willett?; Jeffrey Heer†; Maneesh Agrawala? SIGCHI 2012 http://vis.stanford.edu/files/2012-CrowdAnalytics-CHI.pdf 3 Y N N NR NR NR NR NR NR explain a visualization NR [910 for both experiments] NR NR NR NR NR N NR NR AMT N Y Y N N N N Y N NR NR NR NR N N N 83 Strategies for crowdsourcing social data analysis Wesley Willett?; Jeffrey Heer†; Maneesh Agrawala? SIGCHI 2012 http://vis.stanford.edu/files/2012-CrowdAnalytics-CHI.pdf 4 Y N N NR NR NR NR NR NR explain a visualization NR [910 for both experiments] NR NR NR NR NR N NR NR AMT N Y Y N N N N Y N NR NR NR NR N N N 84 Strategies for crowdsourcing social data analysis Wesley Willett?; Jeffrey Heer†; Maneesh Agrawala? SIGCHI 2012 http://vis.stanford.edu/files/2012-CrowdAnalytics-CHI.pdf 5 Y N N NR NR NR NR NR NR explain a visualization NR [910 for both experiments] NR NR NR NR NR N NR NR AMT N Y Y N N N N Y N NR NR NR NR N N N 85 Strategies for crowdsourcing social data analysis Wesley Willett?; Jeffrey Heer†; Maneesh Agrawala? SIGCHI 2012 http://vis.stanford.edu/files/2012-CrowdAnalytics-CHI.pdf 6 Y N N NR NR NR NR NR NR rate an explanation of a vis 243 NR NR NR NR NR N NR NR AMT N N Y N N N N Y N NR NR NR NR N N N 86 Comparing averages in time series data Correll, M.; Albers, D.M; Franconeri, S.; Gleicher, M. SIGCHI 2012 http://viscog.psych.northwestern.edu/publications/CorrellAlbersFranconeriGleicher2012.pdf 1 Y N N 15 95% HIT approval rate Colorblindness Test NR NR NR perceptual judgement 74 NR 18 57.00 Y 100:0:0 Y 18-62 34.7 AMT N Y Y N N N N Y N NR NR NR NR Post study N N 87 Influencing visual judgment through affective priming Harrison, L.; Skau, D.; Franconeri, S.; Lu, A.; Chang, R SIGCHI 2013 http://valt.cs.tufts.edu/pdf/harrison2013influencing.pdf 1 Y N N NR NR NR NR N NR proportional judgement 963 664 60 NR NR NR N NR NR AMT N N Y N N N Self-Assessment Manikin (SAM) scale Y N Per study $0.02, $0.35 NR Catch questions relevant to tasks, Other NR N N 88 Modeling how people extract color themes from images Sharon Lin; Pat Hanrahan SIGCHI 2013 http://vis.stanford.edu/papers/color-themes 1a Y N N 4 NR NR NR NR NR color selection and ordering based on personal preferences 160 160 160 NR Y 100:0:0 NR NR NR AMT NR N N N N N number of colors chosen Y N Per Trial $0.50 N N N Y N 89 Modeling how people extract color themes from images Sharon Lin; Pat Hanrahan SIGCHI 2013 http://vis.stanford.edu/papers/color-themes 1b Y N N NR NR NR NR NR NR color selection based on personal preferences 40 40 40 NR Y 100:0:0 NR NR NR AMT NR N N N N N color ratings Y N Per study $1.00 N N N Y N 90 Extracting references between text and charts via crowdsourcing Nicholas Kong, Marti A. Hearst, Maneesh Agrawala SIGCHI 2014 http://dl.acm.org/citation.cfm?id=2557241 Y N N NR NR NR NR NR NR association: text and charts association, based on user preference 77 77 10 NR NR NR NR NR NR AMT NR N N N N N distance to gold standard Y N Per study $1.00 N N N N N 91 Automatic generation of semantic icon encodings for visualizations Vidya Setlur, Jock D. Mackinlay SIGCHI 2014 http://delivery.acm.org/10.1145/2560000/2557408/p541-setlur.pdf?ip=137.73.11.23&id=2557408&acc=OA&key=BF07A2EE685417C5%2EAC4A495B30CCB9B0%2E4D4702B0C3E38B35%2E7FE3141059987072&CFID=786979221&CFTOKEN=28167725&__acm__=1500543269_e21f58a3f26019bafabac785fa99df4b 1 N N Y 0.3 NR NR NR NR NR preference: personal preference from users 89 89 3 NR NR NR NR NR NR AMT NR N N N N N numebr of votes cast Y N Per Trial $0.05 N N Y N N 92 Task-driven evaluation of aggregation in time series visualization Danielle Albers, Michael Correll, Michael Gleicher SIGCHI 2014 http://dl.acm.org/citation.cfm?id=2556288.2557200 1a Y N N NR NR NR NR Passive Y estimation: Max 64 64 8 47 NR NR NR NR 31.3 AMT NR Y Y N N N N Y N Per study $1.00 N Catch questions relevant to tasks N N N 93 Task-driven evaluation of aggregation in time series visualization Danielle Albers, Michael Correll, Michael Gleicher SIGCHI 2014 http://dl.acm.org/citation.cfm?id=2556288.2557200 1b Y N N NR NR NR NR Passive Y estimation: Min 64 64 8 47 NR NR NR NR 31.3 AMT NR Y Y N N N N Y N Per study $1.01 N Catch questions relevant to tasks N N N 94 Task-driven evaluation of aggregation in time series visualization Danielle Albers, Michael Correll, Michael Gleicher SIGCHI 2014 http://dl.acm.org/citation.cfm?id=2556288.2557200 1c Y N N NR NR NR NR Passive Y estimation: Average 64 64 8 47 NR NR NR NR 31.3 AMT NR Y Y N N N N Y N Per study $1.02 N Catch questions relevant to tasks N N N 95 Task-driven evaluation of aggregation in time series visualization Danielle Albers, Michael Correll, Michael Gleicher SIGCHI 2014 http://dl.acm.org/citation.cfm?id=2556288.2557200 1d Y N N NR NR NR NR Passive Y estimation: Range 64 64 8 47 NR NR NR NR 31.3 AMT NR Y Y N N N N Y N Per study $1.03 N Catch questions relevant to tasks N N N 96 Task-driven evaluation of aggregation in time series visualization Danielle Albers, Michael Correll, Michael Gleicher SIGCHI 2014 http://dl.acm.org/citation.cfm?id=2556288.2557200 1e Y N N NR NR NR NR Passive Y estimation: Spread Experiment 56 56 8 47 NR NR NR NR 31.3 AMT NR Y Y N N N N Y N Per study $1.04 N Catch questions relevant to tasks N N N 97 Task-driven evaluation of aggregation in time series visualization Danielle Albers, Michael Correll, Michael Gleicher SIGCHI 2014 http://dl.acm.org/citation.cfm?id=2556288.2557200 1f Y N N NR NR NR NR Passive Y estimation: Outlier Experiment 48 48 8 47 NR NR NR NR 31.3 AMT NR Y Y N N N N Y N Per study $1.05 N Catch questions relevant to tasks N N N 98 A table!: improving temporal navigation in soccer ranking tables Charles Perin, Romain Vuillemot, Jean-Daniel Fekete SIGCHI 2014 http://dl.acm.org/citation.cfm?id=2557379 Y N N NR football fans NR NR NR NR inverse lookup and comparison, identification definition, identification search, inverse comparison, and relation seeking 143 143 NR NR NR 20.4%:0:65.4% NR NR NR NR NR Y Y N N N interaction N Y Per study NR Y Task completion time threshold, Other N N N 99 Understand users’ comprehension and preferences for composing information visualizations Huahai Yang, Yunyao Li, Michelle X. Zhou TOCHI 2014 http://dl.acm.org/citation.cfm?id=2541288 1a Y N N 10 NR NR NR NR NR insight and comprehension 24 24 24 Y 100%:0:0 NR NR NR AMT NR N N N N N description and perceived insights Y N Per study NR N Other N N N 100 Understand users’ comprehension and preferences for composing information visualizations Huahai Yang, Yunyao Li, Michelle X. Zhou TOCHI 2014 http://dl.acm.org/citation.cfm?id=2541288 1b Y N N 10 NR NR NR NR NR insight and comprehension 500 500 50 NR Y 100%:0:0 NR NR NR AMT NR N N N N N description and perceived insights Y N Per study NR N Other N N N 101 Understand users’ comprehension and preferences for composing information visualizations Huahai Yang, Yunyao Li, Michelle X. Zhou TOCHI 2014 http://dl.acm.org/citation.cfm?id=2541288 2 Y N N NR NR NR NR Active NR Comparison, value estimation, identification of extrema, correlation. Rankings of visualization suitability to task. 240 240 30 NR Y 100%:0:1 NR NR NR AMT NR N N N N N rankings of graphics. Y N Per study NR N Other N N N 102 Visualizing Sets with Linear Diagrams Peter Rodgers, Gem Stapleton, Peter Chapman TOCHI 2015 https://kar.kent.ac.uk/50020/ 1 N Y N NR 95% HIT approval rate NR NR Active NR Line Segments 200 197 100 50 Y 100%:0:0 Y 19-73 35 AMT NR Y Y N N N N Y N Per study $1.0 N Catch questions relevant to tasks Pre-study N N 103 Visualizing Sets with Linear Diagrams Peter Rodgers, Gem Stapleton, Peter Chapman TOCHI 2015 https://kar.kent.ac.uk/50020/ 2 N Y N NR 95% HIT approval rate NR NR Active NR Color 300 203 100 56 Y 100%:0:0 Y 19-70 34 AMT NR Y Y N N N N Y N Per study $1.1 N Catch questions relevant to tasks Pre-study N N 104 Visualizing Sets with Linear Diagrams Peter Rodgers, Gem Stapleton, Peter Chapman TOCHI 2015 https://kar.kent.ac.uk/50020/ 3 N Y N NR 95% HIT approval rate NR NR Active NR Guidelines 200 199 100 52 Y 100%:0:0 Y 18-71 35 AMT NR Y Y N N N N Y N Per study $1.2 N Catch questions relevant to tasks Pre-study N N 105 Visualizing Sets with Linear Diagrams Peter Rodgers, Gem Stapleton, Peter Chapman TOCHI 2015 https://kar.kent.ac.uk/50020/ 4 N Y N NR 95% HIT approval rate NR NR Active NR Set Order 200 193 100 47 Y 100%:0:0 Y 18-69 32 AMT NR Y Y N N N N Y N Per study $1.3 N Catch questions relevant to tasks Pre-study N N 106 Visualizing Sets with Linear Diagrams Peter Rodgers, Gem Stapleton, Peter Chapman TOCHI 2015 https://kar.kent.ac.uk/50020/ 5 N Y N NR 95% HIT approval rate NR NR Active NR Line Width 200 192 100 50 Y 100%:0:0 Y 18-71 32 AMT NR Y Y N N N N Y N Per study $1.4 N Catch questions relevant to tasks Pre-study N N 107 Visualizing Sets with Linear Diagrams Peter Rodgers, Gem Stapleton, Peter Chapman TOCHI 2015 https://kar.kent.ac.uk/50020/ 6 N Y N NR 95% HIT approval rate NR NR Active NR Orientation 200 196 100 52 Y 100%:0:0 Y 18-66 33 AMT NR Y Y N N N N Y N Per study $1.5 N Catch questions relevant to tasks Pre-study N N 108 Visualizing Sets with Linear Diagrams Peter Rodgers, Gem Stapleton, Peter Chapman TOCHI 2015 https://kar.kent.ac.uk/50020/ 7 N Y N NR 95% HIT approval rate NR NR Active NR Overall Study 300 287 100 61 Y 100%:0:0 Y 18-80 33 AMT NR Y Y N N N N Y N Per study $1.6 N Catch questions relevant to tasks Pre-study N N 109 “Without the Clutter of Unimportant Words”: Descriptive Keyphrases for Text Visualization JASON CHUANG, CHRISTOPHER D. MANNING, and JEFFREY HEER, TOCHI 2012 http://vis.stanford.edu/papers/keyphrases 1 N Y N NR NR NR NR NR NR Comparison, ranking NR 576 NR NR N NR NR NR NR AMT NR N N N N N preference and ranking Y N Per Trial $0.10 N NR NR N N 110 Infographic Aesthetics: Designing for the First Impression Lane Harrison, Katharina Reinecke, Remco Chang SIGCHI - Note 2015 http://dl.acm.org/citation.cfm?id=2702545 1 N N Y 10 NR NR NR Active NR qualitative rating of images 1278 NR NR 77.5 N 60%:0:0 Y 6-80 23 NR Y N N N N N appeal Y N NR NR N NR NR Y Y 111 ISOTYPE Visualization: Working Memory, Performance, and Engagement with Pictographs Steve Haroz, Robert Kosara, and Steven L. Franconeri. SIGCHI 2015 http://dl.acm.org/citation.cfm?id=2702275 1 Y N N 35 NR NR NR Passive NR Information recalling (testing working memory capacity, done via entering values previously seen) 30 NR NR NR Y 100%:0:0 NR NR NR AMT NR N Y N N N N Y N Per study $8 N NR NR N N 112 ISOTYPE Visualization: Working Memory, Performance, and Engagement with Pictographs Steve Haroz, Robert Kosara, and Steven L. Franconeri. SIGCHI 2015 http://dl.acm.org/citation.cfm?id=2702275 1 Y N N 30 NR NR NR Passive NR Information recovery (done via value estimation of charts previously seen) 50 NR NR NR Y 100%:0:0 NR NR NR AMT NR Y Y N N N N Y N Per study $8 N NR NR N N 113 How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques. Anshul Vikram Pandey, Katharina Rall, Margaret L. Satterthwaite, Oded Nov, and Enrico Bertini. SIGCHI 2015 http://dl.acm.org/citation.cfm?id=2702608 1a N N Y 5-10mins 99% HIT approval rate NR NR Passive NR Value Estimation 250 NR 37-43 NR Y 100%:0:0 NR NR NR AMT Y N Y Y Y Y effect of differences in personal attributes Y N Per study $0.3 N Catch questions relevant to tasks Y N N 114 How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques. Anshul Vikram Pandey, Katharina Rall, Margaret L. Satterthwaite, Oded Nov, and Enrico Bertini. SIGCHI 2015 http://dl.acm.org/citation.cfm?id=2702608 1b N N Y 5-10mins 99% HIT approval rate NR NR Passive NR Value Estimation 250 NR 40 NR Y 100%:0:0 NR NR NR AMT Y N Y Y Y Y effect of differences in personal attributes Y N Per study $0.3 N Catch questions relevant to tasks Y N N 115 How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques. Anshul Vikram Pandey, Katharina Rall, Margaret L. Satterthwaite, Oded Nov, and Enrico Bertini. SIGCHI 2015 http://dl.acm.org/citation.cfm?id=2702608 1c N N Y 5-10mins 99% HIT approval rate NR NR Passive NR Value Estimation 250 NR 38-42 NR Y 100%:0:0 NR NR NR AMT Y N Y Y Y Y effect of differences in personal attributes Y N Per study $0.3 N Catch questions relevant to tasks Y N N 116 How Deceptive are Deceptive Visualizations?: An Empirical Analysis of Common Distortion Techniques. Anshul Vikram Pandey, Katharina Rall, Margaret L. Satterthwaite, Oded Nov, and Enrico Bertini. SIGCHI 2015 http://dl.acm.org/citation.cfm?id=2702608 2 N N Y 5-10mins 99% HIT approval rate NR NR Passive NR data interpretation 80 NR 40-38 NR Y 100%:0:0 NR NR NR AMT Y N Y Y Y Y effect of differences in personal attributes Y N Per study $0.3 N Catch questions relevant to tasks Y N N 117 A Comparative Evaluation on Online Learning Approaches using Parallel Coordinate Visualization Kwon, Bum Chul, and Bongshin Lee. CHI 2016 https://www.researchgate.net/profile/Bum_Chul_Kwon/publication/294891134_A_Comparative_Evaluation_on_Online_Learning_Approaches_using_Parallel_Coordinate_Visualization/links/56c613d608ae408dfe4cef81.pdf 1 Y N N NR 10,000 # HITs approved, 99 HIT Approval Rate (%); no prior knowledge about PC NR NR Active NR Analytics Tasks 188 30 120 NR NR Y 20-61 33.6 AMT NR Y Y Y N N 1) Engagement, 2) Fun, 3) Interestingness, 4) Easiness of the tutorial; and 5) Understanding of parallel coordinates. Y Y Per study $1.50 NR who completed a question under three seconds for more than three times NR NR NR 118 The Effect of Visual Appearance on the Performance of Continuous Sliders and Visual Analogue Scales Justin Matejka, Michael Glueck, Tovi Grossman, and George Fitzmaurice CHI 2016 http://www.dgp.toronto.edu/~tovi/papers/vas.pdf 1 Y N N NR NR NR NR NR NR perceptual judgement task [rate blackness of shade of grey] 1,425 1007-1273 75 NR NR NR N NR NR AMT NR Y Y N N N N N Y Per study $1 NR Other Pre-study N N 119 The Effect of Visual Appearance on the Performance of Continuous Sliders and Visual Analogue Scales Justin Matejka, Michael Glueck, Tovi Grossman, and George Fitzmaurice CHI 2016 http://www.dgp.toronto.edu/~tovi/papers/vas.pdf 2 Y N N NR NR NR NR NR NR objective task [select a specific value along the scale with as much accuracy as possible] 375 270-360 25 NR NR NR N NR NR AMT NR Y Y N N N N N Y Per study $1 NR Other Pre-study N N 120 When (ish) is My Bus? User-centered Visualizations of Uncertainty in Everyday, Mobile Predictive Systems Kay, Matthew, Tara Kola, Jessica R. Hullman, and Sean A. Munson CHI 2016 http://idl.cs.washington.edu/files/2016-WhenIsMyBus-CHI.pdf 1 Y N N NR NR NR NR NR NR interpreting probabilistic predictions from vis; report probabilities 550 541 250 29 NR NR N NR NR AMT Y N Y N N N N N Y Raffle $25 all participants took part in raffle for 1 $100 Amazon.com gift card NR NR N N 121 A Deeper Understanding of Sequence in Narrative Visualization Hullman, J., Drucker, S., Riche, N. H., Lee, B., Fisher, D., & Adar, E. TVCG 2013 1 x - - 11 NR NR NR N Y Decision Making 82 NR NR NR NR NR NR NR NR AMT - - - - - - preference interactive slideshow Per Trial 0.1 $0.1 NR NR - - 122 A Deeper Understanding of Sequence in Narrative Visualization Hullman, J., Drucker, S., Riche, N. H., Lee, B., Fisher, D., & Adar, E. TVCG 2013 2 - x - 2 NR NR NR N Y Decision Making 143 NR NR NR NR NR NR NR NR AMT - - - - - - preference self-advancing slideshow N Per study 8 N NR NR - - 123 Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing Micallef, L., Dragicevic, P., Fekete, J. TVCG 2012 1 - - x 25 NR Self-reported NR N NR Estimating likelihood 168 168 24 41 N 47%:40%:0%:13% Y 18-64 32 AMT - Y Y [bias + abs error] Y [5-point likert scale] Y [objective + subjective tests] Y [paper folding test (part 1 only)] N Y N Per study 1 N NR Y Y Y 124 Assessing the Effect of Visualizations on Bayesian Reasoning through Crowdsourcing Micallef, L., Dragicevic, P., Fekete, J. TVCG 2012 2 x - - 5 NR Self-reported NR N NR Estimating likelihood 480 480 120 NR NR NR NR NR NR AMT - Y Y [bias + abs error] Y [5-point likert scale] N N N Y N Per study 0.4 N NR Y Y Y 125 Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty Nadia Boukhelifa, Anastasia Bezerianos, Tobias Isenberg, Jean-Daniel Fekete TVCG 2012 1 x - - NR >=95% HIT approval rate NR NR NR NR interpret visual encoding 210 NR 35 NR NR NR NR NR NR AMT - N N N N N Judgement/preference Y N Per Trial 0.34 N NR Y N N 126 Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty Nadia Boukhelifa, Anastasia Bezerianos, Tobias Isenberg, Jean-Daniel Fekete TVCG 2012 2 x - - NR >=95% HIT approval rate NR NR NR NR interpret visual encoding 168 NR NR NR NR NR NR NR NR AMT - N N N N N Judgement/preference Y N Per Trial 0.34 N NR Y N N 127 Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty Nadia Boukhelifa, Anastasia Bezerianos, Tobias Isenberg, Jean-Daniel Fekete TVCG 2012 3 x - - NR >=95% HIT approval rate NR NR NR NR interpret visual encoding 168 NR 28 NR NR NR NR NR NR AMT - N N N N N Judgement/preference Y N Per Trial 0.34 N NR Y N N 128 Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty Nadia Boukhelifa, Anastasia Bezerianos, Tobias Isenberg, Jean-Daniel Fekete TVCG 2012 4 - - x NR >=95% HIT approval rate NR NR NR NR compare visual encoding 160 NR 28 NR NR NR NR NR NR AMT - N N N N N Judgement/preference Y N Per Trial 0.34 N NR Y N N 129 Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty Nadia Boukhelifa, Anastasia Bezerianos, Tobias Isenberg, Jean-Daniel Fekete TVCG 2012 5 x - - NR >=95% HIT approval rate NR NR NR NR select visualization style 129 NR 21 NR NR NR NR NR NR AMT - N N N N N Judgement/preference Y N Per Trial 0.34 N NR Y N N 130 HOLA: Human-like Orthogonal Network Layout. Visualization and Computer Graphics, Steve Kieffer, Tim Dwyer, Kim Marriott, Michael Wybrow. TVCG 2016 2 - x - 8 NR NR NR NR NR Evaluate quality of graph layout 66 65 Not applicable NR NR NR NR NR NR Other N N N N N Graph Quality Y N Raffle 50 N NR N Y, but link no longer active Y, but link no longer active 131 Identifying Redundancy and Exposing Provenance in Crowdsourced Data Analysis Willett, W.; Ginosar, S.; Steinitz, A.; Hartmann, B.; Agrawala, M. TVCG 2013 1 - - - NR NR NR NR NR NR annotate charts 93 93 10 NR NR NR NR NR NR AMT - N N N N N textual annotations N Y Per Trial 0.2 N NR N N N 132 Identifying Redundancy and Exposing Provenance in Crowdsourced Data Analysis Willett, W.; Ginosar, S.; Steinitz, A.; Hartmann, B.; Agrawala, M. TVCG 2013 2 - - - NR NR NR NR NR NR annotate charts 96 96 NR NR NR NR NR NR NR AMT - N N N N N textual annotations N Y Per Trial 0.2 N NR N N N 133 Laws of Attraction: From Perceived Forces to Conceptual Similarity Caroline Ziemkiewicz and Robert Kosara TVCG 2010 1 - x - NR NR NR NR NR NR locate flashing target 45 44 NR 46 NR NR Y 18-64 NR AMT - Y Y N N N - N Y Per study 0.2 $.02 for answers within a certain range up to $.5 NR N N N 134 Laws of Attraction: From Perceived Forces to Conceptual Similarity Caroline Ziemkiewicz and Robert Kosara TVCG 2010 2 - x - NR NR NR NR NR NR remember location of target 48 47 NR 43 NR NR Y 20-62 NR AMT - Y Y N N N - N Y Per study 0.28 $.02 for answers within a certain range, up to $1 NR N N N 135 Laws of Attraction: From Perceived Forces to Conceptual Similarity Caroline Ziemkiewicz and Robert Kosara TVCG 2010 3 - x - 5 NR NR NR NR NR visual decision making (decide on which visual encoding implies stronger relationship) 50 48 NR 60 NR NR Y 18-67 NR AMT - Y Y N N N preference N Y Per study 0.4 $.02 for answers within a certain range, up to $1 NR N N N 136 Learning Perceptual Kernels for Visualization Design Demiralp, C. D., Bernstein, M. S., & Heer, J. TVCG 2014 1 - x - NR NR NR NR Active NR Estimate visual similarty 300 NR 20 NR Y 100:00:00 NR NR NR AMT - Y N N Y y [manual MDS, place elements according to similarity] Y N NR .30-3.50$ for 43sec - 1400sec N NR NR N N 137 Learning Perceptual Kernels for Visualization Design Demiralp, C. D., Bernstein, M. S., & Heer, J. TVCG 2014 2 - x - NR NR NR NR Active NR Estimate visual similarty 300 NR 20 NR Y 100:00:00 NR NR NR AMT - Y N N Y y [manual MDS, place elements according to similarity] Y N NR .30-3.50$ for 43sec - 1400sec N NR NR N N 138 A Crowdsourced Alternative to Eye-tracking for Visualization Understanding Nam Wook Kim, Aude Olivia, Zoya Bylinskii Krzysztof Z. Gajos, Michelle A. Borkin, Hanspeter Pfister CHI EA 2015 http://dl.acm.org/citation.cfm?id=2732934 1 Y N N NR >=95% HIT approval rate NR US resident NR NR write a textual description of a blurred image but clicking of different areas of the image for a clearer version of a small part of it to be shown 374 333 NR NR Y 100:00:00 N NR NR AMT Y N N N N N mouse click locations N Y Per Trial $0.05 NR NR NR N N 139 Quantity estimation in visualizations of tagged text Correll, Michael A., Eric C. Alexander, and Michael Gleicher. CHI 2013 http://dl.acm.org/citation.cfm?id=2481373 1 N Y N NR NR Colorblindness Test AMT HIT approval rate > 95% Active NR relative numerosity estimation of tagged text ( relative count of diferent coloured tags) 210 NR 20 52 Y 100:00:00 Y 18-65 34.8 AMT NR NR Y N N N N Y N NR NR NR Catch questions relevant to tasks, click through behaviour NR N N 140 Quantity estimation in visualizations of tagged text Correll, Michael A., Eric C. Alexander, and Michael Gleicher. CHI 2013 http://dl.acm.org/citation.cfm?id=2481373 2 N Y N NR NR Colorblindness Test AMT HIT approval rate > 95% Active NR relative numerosity estimation of tagged text ( relative count of diferent coloured tags) word of varyying lenth as a factor 210 NR 20 52 Y 100:00:00 Y 18-66 34.8 AMT NR NR Y N N N N Y N NR NR NR Catch questions relevant to tasks, click through behaviour NR N N 141 Quantity estimation in visualizations of tagged text Correll, Michael A., Eric C. Alexander, and Michael Gleicher. CHI 2013 http://dl.acm.org/citation.cfm?id=2481373 3 N Y N NR NR Colorblindness Test AMT HIT approval rate > 95% Active NR relative numerosity estimation of tagged text ( relative count of different coloured tags) multiple tag colors as a factor 210 NR 20 52 Y 100:00:00 Y 18-67 34.8 AMT NR NR Y N N N N Y N NR NR NR Catch questions relevant to tasks, click through behaviour NR N N 142 Quantity estimation in visualizations of tagged text Correll, Michael A., Eric C. Alexander, and Michael Gleicher. CHI 2013 http://dl.acm.org/citation.cfm?id=2481373 4 N Y N NR NR Colorblindness Test AMT HIT approval rate > 95% Active NR relative numerosity estimation of tagged text ( relative count of diferent coloured tags)role of tag lenght and count 210 NR 20 52 Y 100:00:00 Y 18-68 34.8 AMT NR NR Y N N N N Y N NR NR NR Catch questions relevant to tasks, click through behaviour NR N N 143 Quantity estimation in visualizations of tagged text Correll, Michael A., Eric C. Alexander, and Michael Gleicher. CHI 2013 http://dl.acm.org/citation.cfm?id=2481373 5 Y N N NR NR Colorblindness Test AMT HIT approval rate > 95% Active NR relative numerosity estimation of tagged text ( relative count of diferent coloured tags) contrast in area vs numerosity 210 NR 60 52 Y 100:00:00 Y 18-69 34.8 AMT NR NR Y N N N N Y N NR NR NR Catch questions relevant to tasks, click through behaviour NR N N 144 Explaining the Gap: Visualizing One’s Predictions Improves Recall and Comprehension of Data Yea-Seul Kim, Katharina Reinecke, Jessica Hullman CHI 2017 https://dl.acm.org/citation.cfm?id=3025592 1 x 19.4 NR NR NR Passive NR Recall of visualised Data 378 373 42 44 NR NR:NR:NR:NR Y 18 - 55 + NR AMT N x x Y , Trend Error, Survey questions relating to the users experience with the data X Per study 1.5 NR Post study X X 145 Explaining the Gap: Visualizing One’s Predictions Improves Recall and Comprehension of Data Yea-Seul Kim, Katharina Reinecke, Jessica Hullman CHI 2017 https://dl.acm.org/citation.cfm?id=3025592 2 x NR NR NR Passive NR Recall of visualised Data NR 209 NR NR NR NR:NR:NR:NR Y 18 - 55 + NR AMT N x x Y , Trend Error, Survey questions relating to the users experience with the data x Per study 1.5 NR Post study x x 146 Explaining the Gap: Visualizing One’s Predictions Improves Recall and Comprehension of Data Yea-Seul Kim, Katharina Reinecke, Jessica Hullman CHI 2017 https://dl.acm.org/citation.cfm?id=3025592 3 x NR NR NR Passive NR Recall of visualised Data NR 230 NR NR NR NR:NR:NR:NR Y 18 - 55 + NR AMT N x x Y , Trend Error, Survey questions relating to the users experience with the data x Per study 1.5 NR Post study x x 147 Regression by Eye: Estimating Trends in Bivariate Visualizations Michael Correll, Jeffrey Heer CHI 2017 https://dl.acm.org/citation.cfm?id=3025922 1 x NR HIT approval Rate > 90%, USA Based NR NR NR NR Adust slider so trendline fits percieved trend of bivariate data 48 46 48 30 Y 100:0:0:0 Y NR 33.2 AMT N X X Per study $2 catch questions relevant to tasks NR X X 148 Regression by Eye: Estimating Trends in Bivariate Visualizations Michael Correll, Jeffrey Heer CHI 2017 https://dl.acm.org/citation.cfm?id=3025922 2 x NR HIT approval Rate > 90%, USA Based NR NR NR NR estimate values of yintercepts of trends 48 46 48 30 Y 100:0:0:0 Y NR 33.2 AMT N x X Per study $2 catch questions relevant to tasks NR X X 149 Regression by Eye: Estimating Trends in Bivariate Visualizations Michael Correll, Jeffrey Heer CHI 2017 https://dl.acm.org/citation.cfm?id=3025922 3 x NR HIT approval Rate > 90%, USA Based NR NR NR NR Adust slider so trendline fits percieved trend of bivariate data, which includes outliers 48 46 46 30 Y 100:0:0:0 Y NR 33.2 AMT N x X Per study $2 catch questions relevant to tasks NR X X 150 Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data? Boy, Jeremy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. CHI 2017 https://dl.acm.org/citation.cfm?id=3025512 1 x NR HIT approval Rate > 99%, Did not participate in pre-study data selection crowdsourcing,USA Only NR NR NR NR Participants emotional response to anthropomorphic visualization featuring unit glyphs organically grouped and with unique shapes and uniqeue labels 50 48 48 52 Y 100:0:0:0 NR NR NR AMT N participants’ empathic concern, personal distress, perception of story-protagonists’ responsibility, and perception of justice of donating for each narrative on 7-point scales. x Per study $0.3 NR NR Post study x 151 Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data? Boy, Jeremy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. CHI 2017 https://dl.acm.org/citation.cfm?id=3025512 2 x NR HIT approval Rate > 99%, Did not participate in pre-study data selection crowdsourcing or previous experiment, USA Only NR NR NR NR Participants emotional response to anthropomorphic visualization featuring aggregate glyphs organically grouped and with standard iconic shapes 50 46 46 59 Y 100:0:0:0 NR NR NR AMT N participants’ empathic concern, personal distress, perception of story-protagonists’ responsibility, and perception of justice of donating for each narrative on 7-point scales. x Per study $0.3 NR NR Post study x 152 Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data? Boy, Jeremy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. CHI 2017 https://dl.acm.org/citation.cfm?id=3025512 3 x NR HIT approval Rate > 99%, Did not participate in pre-study data selection crowdsourcing or previous experiment, USA Only NR NR NR NR Participants emotional response to anthropomorphic visualization featuring unit glyphs organically grouped and with standar iconoc shapes and generic labels 50 45 45 58 Y 100:0:0:0 NR NR NR AMT N participants’ empathic concern, personal distress, perception of story-protagonists’ responsibility, and perception of justice of donating for each narrative on 7-point scales. x Per study $0.3 NR NR Post study x 153 Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data? Boy, Jeremy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. CHI 2017 https://dl.acm.org/citation.cfm?id=3025512 4 x NR HIT approval Rate > 99%, Did not participate in pre-study data selection crowdsourcing or previous experiment, USA Only NR NR NR NR Participants emotional response to anthropomorphic visualization featuring unit glyphs organically grouped and with unique shapes and unique labels, different story data set to ex 1 50 49 49 55 Y 100:0:0:0 NR NR NR AMT N participants’ empathic concern, personal distress, perception of story-protagonists’ responsibility, and perception of justice of donating for each narrative on 7-point scales. x Per study $0.3 NR NR Post study x 154 Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data? Boy, Jeremy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. CHI 2017 https://dl.acm.org/citation.cfm?id=3025512 5 x NR HIT approval Rate > 99%, Did not participate in pre-study data selection crowdsourcing or previous experiment, USA Only NR NR NR NR Participants emotional response to anthropomorphic visualization comparing pie chart to plain text 50 40 40 40 Y 100:0:0:0 NR NR NR AMT N participants’ empathic concern, personal distress, perception of story-protagonists’ responsibility, and perception of justice of donating for each narrative on 7-point scales. x Per study $0.3 NR NR Post study x 155 Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data? Boy, Jeremy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. CHI 2017 https://dl.acm.org/citation.cfm?id=3025512 6 x NR HIT approval Rate > 99%, Did not participate in pre-study data selection crowdsourcing or previous experiment, USA Only NR NR NR NR Participants emotional response to anthropomorphic visualization featuring unit glyphs organically grouped and with unique shapes and unique labels, same story as ex one ,but only showed the chart part of the narrative 50 47 47 47 Y 100:0:0:0 NR NR NR AMT N participants’ empathic concern, personal distress, perception of story-protagonists’ responsibility, and perception of justice of donating for each narrative on 7-point scales. x Per study $0.3 NR NR Post study x 156 Showing People Behind Data: Does Anthropomorphizing Visualizations Elicit More Empathy for Human Rights Data? Boy, Jeremy, Anshul Vikram Pandey, John Emerson, Margaret Satterthwaite, Oded Nov, and Enrico Bertini. CHI 2017 https://dl.acm.org/citation.cfm?id=3025512 7 x NR HIT approval Rate > 99%, Did not participate in pre-study data selection crowdsourcing or previous experiment, USA Only NR NR NR NR Participants emotional response to anthropomorphic visualization featuring unit glyphs organically grouped and with unique shapes and unique labels, same story as ex one ,but only showed the chart part of the narrative and further reduced reduced text ( simplified and removed focus on children) 50 47 47 55 Y 100:0:0:0 NR NR NR AMT N participants’ empathic concern, personal distress, perception of story-protagonists’ responsibility, and perception of justice of donating for each narrative on 7-point scales. x Per study $0.3 NR NR Post study x 157 Assessing User Engagement in Information Visualization Ya-Hsin Hung,Paul Parsons CHI EA 2017 https://dl.acm.org/citation.cfm?id=3053113 1 Y 2.58 NR NR NR NR NR Answer questions based on a visualization. And then answer a questionnaire on visualization engagement. 3 visualizations total 40 27 27 44 NR NR:NR:NR:NR Y 20-75 37.73 AMT N Y N N N N Engagement, Tracked mouse movement N Y NR NR NR Y; Removed based on short response time NR 158 Narratives in Crowdsourced Evaluation of Visualizations: A Double-Edged Sword? Evanthia Dimara, Anastasia Bezerianos,Pierre Dragicevic CHI 2017 https://dl.acm.org/citation.cfm?id=3025870 1 X 7 Highly rated (level 3) crowdflower participants NR NR NR NR Three Tasks: An Extremum task, where participants had to find the data point with highest value according to the X dimension. A Correlation task , where participants had to find the scatter-plot with the highest correlation among four different ones. A Comparison task, where participants had to compare data points across their two dimensions simultaneously 405 405 80 NR NR:NR:NR:NR Y 18-79 NR CrowdFlower NR N Y Y N N In Task Attention, Post Task Attention, Enjoyablity, Usefulness (All subjective) N Y Per study $0.6 NR N NR Y Y 159 BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention Nam Wook Kim, Zoya Bylinskii, Michelle A. Borkin, Krzysztof Z. Gajos, Aude Oliva, Fredo Durand, Hanspeter Pfister TOCHI 2017 https://dl.acm.org/citation.cfm?id=3131275 1.1 N Y N 9 >=95% HIT approval rate, living in USA NR NR NR NR describe infovis on website: “click and describe the image” NR NR 38,39,40; 3 conditions NR Y 100:0:0:0 NR NR NR AMT N N N N N N We compared how well the distribution of BubbleView clicks approximates the distribution of eye fixations, using two metrics commonly used for saliency evaluation: Pearson’s Correlation Coefficient (CC) and Normalized Scanpath Saliency (NSS) [Bylinskii et al. 2016]. While the two metrics provide complementary evidence for our conclusions, the NSS metric also allows us to account for differences in attentional consistency between participants (inter-observer congruency) across datasets. N Y Per study 0.5USD N other; description at least 150 characters NR Y Y; supplementary material 160 BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention Nam Wook Kim, Zoya Bylinskii, Michelle A. Borkin, Krzysztof Z. Gajos, Aude Oliva, Fredo Durand, Hanspeter Pfister TOCHI 2017 https://dl.acm.org/citation.cfm?id=3131275 1.2 N Y N 9 >=95% HIT approval rate, living in USA NR NR NR NR describe infovis on website: “click and describe the image” NR NR 20,18,11; 3 conditions NR Y 100:0:0:0 NR NR NR AMT N N N N N N We compared how well the distribution of BubbleView clicks approximates the distribution of eye fixations, using two metrics commonly used for saliency evaluation: Pearson’s Correlation Coefficient (CC) and Normalized Scanpath Saliency (NSS) [Bylinskii et al. 2016]. While the two metrics provide complementary evidence for our conclusions, the NSS metric also allows us to account for differences in attentional consistency between participants (inter-observer congruency) across datasets. N Y Per study 0.5USD N other; description at least 150 characters NR Y Y; supplementary material 161 BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention Nam Wook Kim, Zoya Bylinskii, Michelle A. Borkin, Krzysztof Z. Gajos, Aude Oliva, Fredo Durand, Hanspeter Pfister TOCHI 2017 https://dl.acm.org/citation.cfm?id=3131275 1.3 N Y N 9 >=95% HIT approval rate, living in USA NR NR NR NR describe infovis on website: “click and describe the image” NR NR 10; 1 condition NR Y 100:0:0:0 NR NR NR AMT N N N N N N We compared how well the distribution of BubbleView clicks approximates the distribution of eye fixations, using two metrics commonly used for saliency evaluation: Pearson’s Correlation Coefficient (CC) and Normalized Scanpath Saliency (NSS) [Bylinskii et al. 2016]. While the two metrics provide complementary evidence for our conclusions, the NSS metric also allows us to account for differences in attentional consistency between participants (inter-observer congruency) across datasets. N Y Per study 0.5USD N other; description at least 150 characters NR Y Y; supplementary material 162 BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention Nam Wook Kim, Zoya Bylinskii, Michelle A. Borkin, Krzysztof Z. Gajos, Aude Oliva, Fredo Durand, Hanspeter Pfister TOCHI 2017 https://dl.acm.org/citation.cfm?id=3131275 2.1 N Y N 2.8 >=95% HIT approval rate, living in USA NR NR NR NR free-viewing of natural images on websites: “click anywhere you want to look” NR NR 54; 1 condition NR Y 100:0:0:0 NR NR NR AMT N N N N N N We compared how well the distribution of BubbleView clicks approximates the distribution of eye fixations, using two metrics commonly used for saliency evaluation: Pearson’s Correlation Coefficient (CC) and Normalized Scanpath Saliency (NSS) [Bylinskii et al. 2016]. While the two metrics provide complementary evidence for our conclusions, the NSS metric also allows us to account for differences in attentional consistency between participants (inter-observer congruency) across datasets. N Y Per study 0.3USD N NR NR Y Y; supplementary material 163 BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention Nam Wook Kim, Zoya Bylinskii, Michelle A. Borkin, Krzysztof Z. Gajos, Aude Oliva, Fredo Durand, Hanspeter Pfister TOCHI 2017 https://dl.acm.org/citation.cfm?id=3131275 3.1 N Y N 8.5 >=95% HIT approval rate, living in USA NR NR NR NR free-viewing of natural images on websites: “click anywhere you want to look” NR NR 12,12,12; 3 conditions NR Y 100:0:0:0 NR NR NR AMT N N N N N N We compared how well the distribution of BubbleView clicks approximates the distribution of eye fixations, using two metrics commonly used for saliency evaluation: Pearson’s Correlation Coefficient (CC) and Normalized Scanpath Saliency (NSS) [Bylinskii et al. 2016]. While the two metrics provide complementary evidence for our conclusions, the NSS metric also allows us to account for differences in attentional consistency between participants (inter-observer congruency) across datasets. N Y Per study 0.9USD N NR NR Y Y; supplementary material 164 BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention Nam Wook Kim, Zoya Bylinskii, Michelle A. Borkin, Krzysztof Z. Gajos, Aude Oliva, Fredo Durand, Hanspeter Pfister TOCHI 2017 https://dl.acm.org/citation.cfm?id=3131275 3.2 N Y N 9 >=95% HIT approval rate, living in USA NR NR NR NR describe infovis on website: “click and describe the image” NR NR 12; 1 condition NR Y 100:0:0:0 NR NR NR AMT N N N N N N We compared how well the distribution of BubbleView clicks approximates the distribution of eye fixations, using two metrics commonly used for saliency evaluation: Pearson’s Correlation Coefficient (CC) and Normalized Scanpath Saliency (NSS) [Bylinskii et al. 2016]. While the two metrics provide complementary evidence for our conclusions, the NSS metric also allows us to account for differences in attentional consistency between participants (inter-observer congruency) across datasets. N Y Per study 0.5USD N other; description at least 150 characters NR Y Y; supplementary material 165 BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention Nam Wook Kim, Zoya Bylinskii, Michelle A. Borkin, Krzysztof Z. Gajos, Aude Oliva, Fredo Durand, Hanspeter Pfister TOCHI 2017 https://dl.acm.org/citation.cfm?id=3131275 4 N Y N 2.8 >=95% HIT approval rate, living in USA NR NR NR NR free-viewing of natural images on websites: “click anywhere you want to look” NR NR 35; 1 condition NR Y 100:0:0:0 NR NR NR AMT N N N N N N We compared how well the distribution of BubbleView clicks approximates the distribution of eye fixations, using two metrics commonly used for saliency evaluation: Pearson’s Correlation Coefficient (CC) and Normalized Scanpath Saliency (NSS) [Bylinskii et al. 2016]. While the two metrics provide complementary evidence for our conclusions, the NSS metric also allows us to account for differences in attentional consistency between participants (inter-observer congruency) across datasets. N Y Per study 0.3USD N NR NR Y Y; supplementary material 166 BubbleView: An Interface for Crowdsourcing Image Importance Maps and Tracking Visual Attention Nam Wook Kim, Zoya Bylinskii, Michelle A. Borkin, Krzysztof Z. Gajos, Aude Oliva, Fredo Durand, Hanspeter Pfister TOCHI 2017 https://dl.acm.org/citation.cfm?id=3131275 5.1 N Y N 2.8 >=95% HIT approval rate, living in USA NR NR NR NR free-viewing of natural images on websites: “click anywhere you want to look” NR NR 12,12,12; 3 conditions NR Y 100:0:0:0 NR NR NR AMT N N N N N N We compared how well the distribution of BubbleView clicks approximates the distribution of eye fixations, using two metrics commonly used for saliency evaluation: Pearson’s Correlation Coefficient (CC) and Normalized Scanpath Saliency (NSS) [Bylinskii et al. 2016]. While the two metrics provide complementary evidence for our conclusions, the NSS metric also allows us to account for differences in attentional consistency between participants (inter-observer congruency) across datasets. N Y Per study 0.3USD N NR NR Y Y; supplementary material 167 A Crowdsourced Approach to Colormap Assessment Terece L. Turton, Colin Ware, Francesca Samsel, David H. Rogers Eurovis 2017 1 Y N N NR Women only 100% gender report in MT; Self-reported NR Active NR Similarity - color scale 180 180 180 100% NR NR NR NR NR TurkPrime NR N Y N N N N Y N NR NR NR NR NR NR NR 168 A Crowdsourced Approach to Colormap Assessment Terece L. Turton, Colin Ware, Francesca Samsel, David H. Rogers Eurovis 2017 2 Y N N NR HIT approval rate > 95%; MT works >100; English speaking country residence; Self-reported authors visual literacy test; authors CVD test Active NR Similarity - color scale NR NR NR NR NR NR NR NR NR TurkPrime NR N Y N N N N Y N NR NR NR NR NR NR NR 169 A Crowdsourced Approach to Colormap Assessment Terece L. Turton, Colin Ware, Francesca Samsel, David H. Rogers Eurovis 2017 3 Y N N NR CVD Self-reported and measured NR Active NR Similarity - color scale 298 298 298 NR NR NR NR NR NR TurkPrime NR N Y N N N N Y N NR NR NR NR NR NR NR 170 Finding a Clear Path: Structuring Strategies for Visualization Sequences Jessica Hullman, Robert Kosara, and Heidi Lam Eurovis 2017 https://research.tableau.com/sites/default/files/Hullman-EuroVis-2017.pdf 1 Y N N 19 NR NR NR NR NR order visualization sequence 44 NR 31 56 NR NR NR NR NR AMT NR Y N N N N subjective participant ranking Y N Per study $5.00 raffle for $100 gift-cards NR NR NR NR 171 Finding a Clear Path: Structuring Strategies for Visualization Sequences Jessica Hullman, Robert Kosara, and Heidi Lam Eurovis 2017 https://research.tableau.com/sites/default/files/Hullman-EuroVis-2017.pdf 2 Y N N 7.5 >=95% HIT approval rate NR NR NR NR order visualization sequence 64 NR NR NR NR NR NR NR AMT NR N N N N N subjective preference measure Y N Per study $3.00 N NR NR NR NR 172 Visual Narrative Flow: Exploring Factors Shaping Data Visualization Story Reading Experiences S. McKenna, N. Henry Riche, B. Lee, J. Boy, & M. Meyer Eurovis 2017 https://www.cs.utah.edu/~miriah/publications/narrative-flow.pdf 1 N N Y NR >=98% HIT approval rate, >=100 approved HITs, English-speaking countries only NR NR NR Y fact checking questions 240 240 20 NR NR NR NR NR NR AMT N N Y N N N engagement N Y Per study $2.31 NR Y Y Y N 173 Readability and Precision in Pictorial Bar Charts Drew Skau and Robert Kosara Eurovis 2017 https://research.tableau.com/sites/default/files/Skau-EuroVis-2017.pdf 1 N Y N 16 NR NR NR NR NR Read values (absolute) & Compare two bars (relative) 81 79 79 47 NR NR Y 18- NR AMT NR Y Y N N N N Y N Per study $2.50 NR NR NR Y Y 174 Fauxvea: Crowdsourcing Gaze Location Estimates for Visualization Analysis Tasks TVCG 2016 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7414495 Y N N NR NR NR NR Active NR Chart reading task 400 392 100 NR NR NR NR NR 28 AMT NR N N N N N N N Y Per study $0.15 $0.10 NR NR NR NR 175 Fauxvea: Crowdsourcing Gaze Location Estimates for Visualization Analysis Tasks TVCG 2016 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7414495 Y N N 4.5 NR NR NR Active NR Chart reading task 400 NR 85 NR Y 100:00:00 NR NR 28 AMT NR N N N N N N N Y Per study $0.45 $0.15 NR NR NR NR 176 Towards Perceptual Optimization of the Visual Design of Scatterplots Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf TVCG 2017 http://ieeexplore.ieee.org/abstract/document/7864468/ 1 N Y N 20 not colorblind, >=2 out of 4 easy training/screening questions correct, English-speaking, at least high-school eduction, highest level of performance and reliability on CrowdFlower (this is similar to the 95% HIT approval rate of Amazon MTurk), did not take part in any of the other experiments reported in the paper, used scatter-plots at least occasionally and at least somewhat familiar with scatterplots (scatterplot is the visualization tested in the experiment) Self-reported >=2 out of 4 easy training/screening questions to ensure the participant understood the task and has some level of understanding of scatterplot (i.e., the visualization tested in the experiment); correct answer shown after the participant provided an answer (thus questions used for both screening and training) Active NR correlation estimation 69 69 NA 34.80% N NR Y NR 34 CrowdFlower NR Y Y N N N Success = proportion of correct responses (Error=distance from ground truth) Y N Other - paid 1 USD only if participant answered all ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) and 30% of all other questions correct 1USD N Other - an inattentive participant is one who got >=1 of the ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) incorrect AND/OR <30% of all other questions correct; also the next question page was loaded after one of the buttons was clicked or automatically after 15 seconds, so inattentive participants that took too long to answer would have a less chance of getting >=30% of questions correct Pre study Y Y 177 Towards Perceptual Optimization of the Visual Design of Scatterplots Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf TVCG 2017 http://ieeexplore.ieee.org/abstract/document/7864468/ 2 N Y N 20 not colorblind, >=2 out of 4 easy training/screening questions correct, English-speaking, at least high-school eduction, highest level of performance and reliability on CrowdFlower (this is similar to the 95% HIT approval rate of Amazon MTurk), did not take part in any of the other experiments reported in the paper, used scatter-plots at least occasionally and at least somewhat familiar with scatterplots (scatterplot is the visualization tested in the experiment) Self-reported >=2 out of 4 easy training/screening questions to ensure the participant understood the task and has some level of understanding of scatterplot (i.e., the visualization tested in the experiment); correct answer shown after the participant provided an answer (thus questions used for both screening and training) Active NR correlation estimation 86 86 NA 16.30% N NR Y NR 33 CrowdFlower NR Y Y N N N Success = proportion of correct responses (Error=distance from ground truth) Y N Other - paid 1 USD only if participant answered all ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) and 30% of all other questions correct 1USD N Other - an inattentive participant is one who got >=1 of the ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) incorrect AND/OR <30% of all other questions correct; also the next question page was loaded after one of the buttons was clicked or automatically after 15 seconds, so inattentive participants that took too long to answer would have a less chance of getting >=30% of questions correct Pre study Y Y 178 Towards Perceptual Optimization of the Visual Design of Scatterplots Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf TVCG 2017 http://ieeexplore.ieee.org/abstract/document/7864468/ 3 N Y N 20 not colorblind, >=2 out of 4 easy training/screening questions correct, English-speaking, at least high-school eduction, highest level of performance and reliability on CrowdFlower (this is similar to the 95% HIT approval rate of Amazon MTurk), did not take part in any of the other experiments reported in the paper, used scatter-plots at least occasionally and at least somewhat familiar with scatterplots (scatterplot is the visualization tested in the experiment) Self-reported >=2 out of 4 easy training/screening questions to ensure the participant understood the task and has some level of understanding of scatterplot (i.e., the visualization tested in the experiment); correct answer shown after the participant provided an answer (thus questions used for both screening and training) Active NR class separation 100 100 NA 35.00% N NR Y NR 33.5 CrowdFlower NR Y Y N N N Success = proportion of correct responses (Error=distance from ground truth) Y N Other - paid 1 USD only if participant answered all ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) and 30% of all other questions correct 1USD N Other - an inattentive participant is one who got >=1 of the ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) incorrect AND/OR <30% of all other questions correct; also the next question page was loaded after one of the buttons was clicked or automatically after 15 seconds, so inattentive participants that took too long to answer would have a less chance of getting >=30% of questions correct Pre study Y Y 179 Towards Perceptual Optimization of the Visual Design of Scatterplots Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf TVCG 2017 http://ieeexplore.ieee.org/abstract/document/7864468/ 4 N Y N 20 not colorblind, >=2 out of 4 easy training/screening questions correct, English-speaking, at least high-school eduction, highest level of performance and reliability on CrowdFlower (this is similar to the 95% HIT approval rate of Amazon MTurk), did not take part in any of the other experiments reported in the paper, used scatter-plots at least occasionally and at least somewhat familiar with scatterplots (scatterplot is the visualization tested in the experiment) Self-reported >=2 out of 4 easy training/screening questions to ensure the participant understood the task and has some level of understanding of scatterplot (i.e., the visualization tested in the experiment); correct answer shown after the participant provided an answer (thus questions used for both screening and training) Active NR outlier detection 82 82 NA 12.20% N NR Y NR 35.7 CrowdFlower NR Y Y N N N Success = proportion of correct responses (Error=distance from ground truth) Y N Other - paid 1 USD only if participant answered all ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) and 30% of all other questions correct 1USD N Other - an inattentive participant is one who got >=1 of the ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) incorrect AND/OR <30% of all other questions correct; also the next question page was loaded after one of the buttons was clicked or automatically after 15 seconds, so inattentive participants that took too long to answer would have a less chance of getting >=30% of questions correct Pre study Y Y 180 Towards Perceptual Optimization of the Visual Design of Scatterplots Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf TVCG 2017 http://ieeexplore.ieee.org/abstract/document/7864468/ 5 N Y N 20 not colorblind, >=2 out of 4 easy training/screening questions correct, English-speaking, at least high-school eduction, highest level of performance and reliability on CrowdFlower (this is similar to the 95% HIT approval rate of Amazon MTurk), did not take part in any of the other experiments reported in the paper, used scatter-plots at least occasionally and at least somewhat familiar with scatterplots (scatterplot is the visualization tested in the experiment) Self-reported >=2 out of 4 easy training/screening questions to ensure the participant understood the task and has some level of understanding of scatterplot (i.e., the visualization tested in the experiment); correct answer shown after the participant provided an answer (thus questions used for both screening and training) Active NR correlation estimation 127 127 NA 25.30% N NR Y NR 32.6 CrowdFlower NR Y Y N N N Success = proportion of correct responses (Error=distance from ground truth) Y N Other - paid 1 USD only if participant answered all ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) and 30% of all other questions correct 1USD N Other - an inattentive participant is one who got >=1 of the ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) incorrect AND/OR <30% of all other questions correct; also the next question page was loaded after one of the buttons was clicked or automatically after 15 seconds, so inattentive participants that took too long to answer would have a less chance of getting >=30% of questions correct Pre study Y Y 181 Towards Perceptual Optimization of the Visual Design of Scatterplots Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf TVCG 2017 http://ieeexplore.ieee.org/abstract/document/7864468/ 6 N Y N 20 not colorblind, >=2 out of 4 easy training/screening questions correct, English-speaking, at least high-school eduction, highest level of performance and reliability on CrowdFlower (this is similar to the 95% HIT approval rate of Amazon MTurk), did not take part in any of the other experiments reported in the paper, used scatter-plots at least occasionally and at least somewhat familiar with scatterplots (scatterplot is the visualization tested in the experiment) Self-reported >=2 out of 4 easy training/screening questions to ensure the participant understood the task and has some level of understanding of scatterplot (i.e., the visualization tested in the experiment); correct answer shown after the participant provided an answer (thus questions used for both screening and training) Active NR class separation 107 107 NA 23.40% N NR Y NR 31.7 CrowdFlower NR Y Y N N N Success = proportion of correct responses (Error=distance from ground truth) Y N Other - paid 1 USD only if participant answered all ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) and 30% of all other questions correct 1USD N Other - an inattentive participant is one who got >=1 of the ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) incorrect AND/OR <30% of all other questions correct; also the next question page was loaded after one of the buttons was clicked or automatically after 15 seconds, so inattentive participants that took too long to answer would have a less chance of getting >=30% of questions correct Pre study Y Y 182 Towards Perceptual Optimization of the Visual Design of Scatterplots Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf TVCG 2017 http://ieeexplore.ieee.org/abstract/document/7864468/ 7 N Y N 20 not colorblind, >=2 out of 4 easy training/screening questions correct, English-speaking, at least high-school eduction, highest level of performance and reliability on CrowdFlower (this is similar to the 95% HIT approval rate of Amazon MTurk), did not take part in any of the other experiments reported in the paper, used scatter-plots at least occasionally and at least somewhat familiar with scatterplots (scatterplot is the visualization tested in the experiment) Self-reported >=2 out of 4 easy training/screening questions to ensure the participant understood the task and has some level of understanding of scatterplot (i.e., the visualization tested in the experiment); correct answer shown after the participant provided an answer (thus questions used for both screening and training) Active NR outlier detection 119 119 NA 25.20% N NR Y NR 33.7 CrowdFlower NR Y Y N N N Success = proportion of correct responses (Error=distance from ground truth) Y N Other - paid 1 USD only if participant answered all ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) and 30% of all other questions correct 1USD N Other - an inattentive participant is one who got >=1 of the ~4 catch questions (an easy problem that participants who succeeded in the training phase could be expected to solve) incorrect AND/OR <30% of all other questions correct; also the next question page was loaded after one of the buttons was clicked or automatically after 15 seconds, so inattentive participants that took too long to answer would have a less chance of getting >=30% of questions correct Pre study Y Y 183 Blinded with Science or Informed by Charts? A Replication Study Pierre Dragicevic, Yvonne Jansen TVCG 2017 http://hal.upmc.fr/hal-01580259/document 1 Y N N 1 at least level 3 (the higest on CrowdFlower) NR NR NR Y chart reading NR 123 61 35 N 32 different countries, paper includes percentages: They were from 32 different countries covering Europe (55%), Americas (28%) and Asia (17%). Y NR 35 CrowdFlower NR Y Y Y N N N Y N Per study $0.12 N Y NR Y Y 184 Blinded with Science or Informed by Charts? A Replication Study Pierre Dragicevic, Yvonne Jansen TVCG 2017 http://hal.upmc.fr/hal-01580259/document 2 Y N N 2 at least level 3 (the higest on CrowdFlower) NR NR NR Y chart reading NR 164 83 35 N Y NR 35 CrowdFlower NR Y Y Y N N N Y N Per study $0.20 N Y NR Y Y 185 Blinded with Science or Informed by Charts? A Replication Study Pierre Dragicevic, Yvonne Jansen TVCG 2017 http://hal.upmc.fr/hal-01580259/document 3 Y N N 2.5 at least level 3 (the higest on CrowdFlower) NR NR NR Y chart reading NR 176 88 35 N Y NR 35 CrowdFlower NR Y Y Y N N N Y N Per study $0.15 N Y NR Y Y 186 Blinded with Science or Informed by Charts? A Replication Study Pierre Dragicevic, Yvonne Jansen TVCG 2017 http://hal.upmc.fr/hal-01580259/document 4 Y N N 1.7 job approval rate of 97% NR NR NR Y chart reading NR 160 80 44 Y 100:00:00 Y NR 37 CrowdFlower NR Y Y Y N N N Y N Per study $0.20 N Y NR Y Y 187 Taking Word Clouds Apart: An Empirical Investigation of the Design Space for Keyword Summaries Cristian Felix, Steven Franconeri and Enrico Bertini TVCG 2017 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8017641 1 x NR Task acceptance (HIT approval?) >= 99% NR NR Active NR Select the smaller of two highlighted terms 60 60 20 NR y 100:0:0:0 Y 18-71 NR AMT NR N Y N N N N Y N NR NR NR NR Pre study NR NR 188 Taking Word Clouds Apart: An Empirical Investigation of the Design Space for Keyword Summaries Cristian Felix, Steven Franconeri and Enrico Bertini TVCG 2017 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8017642 2 x NR Task acceptance (HIT approval?) >= 99% NR NR Active NR Select a target tem in a wordcloud 60 60 20 NR y 100:0:0:0 Y 18-71 NR AMT NR N Y N N N N Y N NR NR NR NR Pre study NR NR 189 Taking Word Clouds Apart: An Empirical Investigation of the Design Space for Keyword Summaries Cristian Felix, Steven Franconeri and Enrico Bertini TVCG 2017 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8017643 3 x NR Task acceptance (HIT approval?) >= 99% NR Complete 3 successful training tasks Active NR Select correct topic for keyword summary 150 150 30 NR y 100:0:0:0 Y 18-71 NR AMT NR N Y N N N N Y N NR NR NR NR Pre study NR NR 190 Taking Word Clouds Apart: An Empirical Investigation of the Design Space for Keyword Summaries Cristian Felix, Steven Franconeri and Enrico Bertini TVCG 2017 http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8017644 4 x NR Task acceptance (HIT approval?) >= 99% NR NR Active NR Identify items of interest( issues) using key key word summary 150 150 10 NR y 100:0:0:0 Y 18-71 NR AMT NR N Y N N N Coverage (the percentage of topics identified out of all available topics) Y N NR NR NR NR Pre study NR NR