uncertainty-09-24

Uncertainty Visualization Study Group Notes – 09/24/2012

Discussion

A User Study to Compare Four Uncertainty Visualization Methods for 1D and 2D Datasets Sanyal, J. et al., 2009 link

Slides from presentation by Hoa Nguyen can be found here.

Underlying data similar but not the same for different sub-figures comparing encodings; might have been nice to be able to directly compare.
Deciding on a Participant Pool
- How many participants? What experience level? – no clear easy answer
- Distinguishing: within-subject studies (require fewer participants) vs between-subject studies
- Standard practice now is to report not only the p-value and significance of an effect, but also its effect size.
- Population extremely difficult to control for
  - Population sample size, population sample bias, variability, and effect size all have to be accounted for
  - This is only more difficult online (e.g. automated programs that rely on crowd-sourcing CAPTCHA responses)
- Our grant proposal refers mostly to communicating uncertainty to skilled users–do we need to create a subject pool of skilled users?
- Power analysis: appx. number of subjects to test can be precomputed, built out of variability information, based on effect size
  - Easier if effect already established; but if not, it is possible to do a pilot.
- Mechanical Turk: Worth using?
  - Does the financial incentive degrade data quality?
  - Environmental settings for participants (monitor brightness, surrounding area) cannot be controlled for. To what extent does it really matter?
- Maybe full cognitive studies are excessive, rather sticking to more user study-type investigations would be advisable? - (Miriah)
Possible pattern: Cognitive science concerns adressed during VisWeek keynote driving larger number of projects that try to account for cognitive concerns (i.e. have user studies)
- Spikes over the past 3 years.
- Definitely see general increase in cognitive concerns over the past 5 years or so.
- Might be worth further investigation
Really bad visualization techniques used, so weird results not entirely surprising.
- Error bars were un-readable and indistinguishable
- Color mapping obscured the data
Study tests visual salience rather than uncertainty?
- Looking at the search paradigms, nothing in the study requires that the additional variable being encoded represent uncertainty; users could have been looking for high/low values of any property.
- Study appears to be more about encoding multiple data channels, than examining uncertainty specifically.

Administration

Next Week: Approaches to Uncertainty Visualization. Alex Pang, Craig Wittenbrink, Suresh Lodha, 1997.