http://www.ischool.utexas.

edu

http://bit.ly/ix_lab

Differences in Reading between Word

Search & Information Relevance Decisions
Evidence from Eye-tracking
Jacek Gwizdka
Information eXperience Lab, Co-director
School of Information, University of Texas at Austin
NeuroIS2016@gwizdka.com
http:// j.mp / jacek_research
Gmunden Retreat - NeuroIS2016
June 8, 2016

Talk Outline
Background
Research

and motivation

Questions

Eye-tracking:

modeling reading + pupillometry

Experimental

design

Results
Summary

Jacek Gwizdka

What do Users Want?

Jacek Gwizdka

What do Users Want?

How

can we improve information systems understanding of

user information needs and goals?
for example, can an information system know
when a user is looking at a desired item on an ecommerce site?
when a user has made a decision about finding needed information

How

do we get more information from users?

Obtaining feedback from users
explicit relevance feedback users are typically not willing to do this
implicit relevance feedback
user interaction used to infer their intent and needs
some success, but some signals ambiguous (e.g., dwell time)
looking for other, complementary sources of information

Jacek Gwizdka

Relevance

Something

is relevant to a task if it increases the likelihood of

accomplishing the goal, which is implied by the task.
(Hjrland & Sejer Christensen,2002)

Topical

relevance or aboutness: to what extent an information

object (a document, a web page, an image) matches a users

information need (e.g., Saracevic, 2007)

Jacek Gwizdka

Inferring Relevance from Eye-tracking Data

Eye

regressions and relative duration of the 1st fixation on a

word word relevance (Ajanki et al. ,2009)
Length of coherently read text relevance indicator (Buscher
at el., 2012)

Not

relevant documents were shown to impose lower mental

workload than relevant ones (Gwizdka, 2014; Villa et al, 2013)
Pupil dilates on relevant images (Oliveira et al. 2005) and on short
text documents and web pages (Gwizdka, 2014a; 2014b; 2015).
Relevance typically treated as a binary construct
EEG signals used to detect relevance, including FRPs
fMRI used to investigate differences in brain activations

Jacek Gwizdka

Research Questions
Depth of relevance level of processing required to make
relevance judgment
for example: surface features vs. semantic processing

Operationalization of depth of relevance judgment:

locating a target word (orthographic matching) vs.
finding information needed to answer question (semantic processing)

RQ1.Do people read differently when locating a target word as

compared to when finding needed information?
RQ2. Does pupil dilation differ between the two cases?

Jacek Gwizdka

Eye-tracking:
Modeling reading & Pupillometry

Jacek Gwizdka

Reading Model Origins

Influenced

by E-Z Reader model

Rayner , Pollatsek, Reichle

Serial reading
Words can be identified in parafovial region

2o (70px) foveal region sharp image

parafoveal region

Jacek Gwizdka

Two-State Line-Oriented Reading Model

q
fixation
sequences
in one line of text

Read
1-p

isolated
fixations

Scan
1-q

Filter fixations < 150ms (min time required for lexical processing)
Model states characterized by:
probability of transitions; number of lexical fixations; duration
length of eye-movement trajectory, amount of text covered

Jacek Gwizdka

10

Pupilometry
Pupil

dilation is controlled by the autonomous nervous

system
Pupil dilation is affected by light need to control
Dilation associated with cognitive functions
mental effort and decision making (Kahneman & Beatty, 1966)
Kahneman, D., & Beatty, J. (1966). Pupil Diameter and Load on Memory. Science, 154(3756), 15831585. http://doi.org/10.1126/
science.154.3756.1583

Jacek Gwizdka

11

Experiment
TYPE

controlled lab study

PARTICIPANTS

24 students; 1 search session each

SEARCH TASKS

2 task types: word-search (W) and information finding to

answer question (I); 21 trials each

EXP. DESIGN

within-subjects (more details next slide)

DOC CORPUS

AQUAINT - a corpus of English-language news (TREC Q&A)

DATA COLLECTION

binary relevance judgments, eye-tracking (Tobii T60)

Jacek Gwizdka

12

Experimental Design
Within subject design with 2 task types
W word search: find target word in a short news story press Y/N
I information search: find information that answers question press Y/N
Three types of trials: relevant (R), topical (T), irrelevant (I)
rotations: RTI, RIT, ITR, IRT, TRI, RII, IRI,

Order of questions/stimuli randomized within each task

To minimize memory load information target was repeated before each stimulus

21 x

W
30s

WS task
instructions

6s

6s

6s

target:
word

20s max

6s

xmx ssms nsns snsns

jsdjsd djdjd djdj dkke ekek

kdkddk dkdkdk dkdkdkd

kkdkd d d dd d djdj djdjdj
rjrjr rjr jweje ejejej ejej
kekekek ekeke wej e eej
eje j

21 x

I
30s

IS task
instructions

6s

8s

target:
infoQ

+ 6s

20s max +6s

xmx ssms nsns snsns

jsdjsd ke ekek dkdkdkkd
kdkddk dkdkdk dkdkdkd
kkdkd d rjr jweje ejejej
ejej
kekekek ekeke wee ejej
fjfjf fjfjfjfjf fjfjrjr rreje j

4s

target:
infoQ

+ 6s

20smax +6s

xmx ssms nsns snsns

jsdjsd djdjd djdj dkke ekek
dkdkdkkd
kdkddk dkdkdk dkdkdkd
kkdkd d d dd d djdj djdjdj
rjrjr rjjweje ejejej ejej
kek ekeke wej e ejej eje j

4s

target:
infoQ

Jacek Gwizdka

+6s

20smax +

xmx ssms nsns snsns

jsdjsd ke ekek dkdkdkk
kdkddk dkdkdkdkdkdkd
kkdkd d rjr jweje ejeje
ekeke wej e ejej fjfjf fjfjfjfjf
fjfjrjr rreje j

13

Text Stimuli
Text

stimuli for both tasks came from international news stories

(AQUAINT - a corpus of English-language news, international topics, text only, early 2000s)

Information

Search (I) question and three associated news

stories with controlled degree of relevance:
R- Relevant (contains an answer to the question)
T Topical on the same topic, but only partially relevant (i.e. not
containing an answer to the given question)
I - Irrelevant (on a different topic)

Jacek Gwizdka

14

Example W Trial and Stimulus

Please indicate if the following text contains this word: bubble

Jacek Gwizdka

15

Example I Trial and Stimuli

Does the next news story contain the following information:
Russian submarine Kursk sinks: Which Russian fleet was the submarine part of?
Relevant News Story (R)
Irrelevant Story (I)

Partially Relevant Story (T)

Jacek Gwizdka

16

Independent and Dependent Variables

Independent:
task type: W (word search), I (information search)
perceived document relevance (binary: yes/no)
Dependent:
time to complete each trial
counts of eye fixations
eye fixation durations
reading state (reading, scanning) probability transitions and fixation types
relative change in pupil dilation (RPD) normalized for each person

Jacek Gwizdka

17

Eye-tracking Data Cleanup & Processing

Bad

quality fixations (Tobii validity -4) and those outside screen

were removed (~5% of fixations).
Raw pupil data (@ 60Hz) was:
de-noised (Daubechies D4 level 4 wavelet transform)
blinks detected and missing data interpolated using cubic splines (method
adapted from Matht, 2013)

Relative

change in pupil diameter:

i
RPD t

i
i
(Pt-P baseline)/P baseline

Example of pupil data on the next slide

Jacek Gwizdka

18

Pupil Data Example

Relevant

document judged by a participant as relevant

Jacek Gwizdka

19

Results: Eye-tracking & Reading Model

Differences in RM-Eye variables between tasks and texts
Time to complete [s]
pSS
pRR
Fixation count on words
Count of reading fixations
Count of scanning fixations
Total reading duration [s]
Total scanning duration [s]

(mean(SD))

W Task

I Task - R

I Task - I

K-W 2(2)

8.4(4.4)
0.42(0.25)
0.87(0.11)
26(16)

9.7(4.2)
0.37(0.27)
0.89(0.09)
32(17)

9.6(4.6)
0.50(0.26)
0.81(0.12)
31(17)

17.26****
39.27****
114.38****
20.65****

21.8(14.2)
5.8(4.7)
5.4(3.4)
1.6(1.4)

26.9(15.7)
5.3(3.9)
6.6(3.7)
1.4(1.0)

23.8(16.2)
8.3(5.1)
5.8(3.9)
2.3(1.4)

17.86****
105.13****
18.96****
104.67****

* p<0.05, ** p<0.01, *** p<0.001, **** p<0.0001

Jacek Gwizdka

20

Results: Selected Reading Model Data

Jacek Gwizdka

21

Results: Pupil Dilation

Differences in relative pupil dilation between tasks (mean(SD)) [%]
Epoch (1s)
relevance decision

W Task

I Task

ANOVA

0.63 (5.5)

1.02 (5.4)

F(1,980) = 1.09

Differences in relative pupil dilation for three level factor (mean(SD)) [%]
Epoch (1s)

W Task

I TaskR

I TaskI

relevance decision

0.63 (5.5)

3.09 (4.9)

-0.08 (5.3)

ANOVA
F(2,979) 2 = 27.41****

**** p<0.0001

Differences in relative pupil dilation on relevant words (mean(SD)) [%]

Epoch (1s)

W Task

I TaskR

I TaskI

ANOVA

first relevant word

last relevant word

0.35 (5.4)

0.86 (5.1)

n/a

F(1,393) = 0.92

1.05 (5.6)

2.82 (5.0)

n/a

F(1,393) 1 = 11.05***

*** p<0.001, **** p<0.0001

Jacek Gwizdka

22

Results: Classification new!

Can

we predict task and perceived relevance from pupil data

alone
time segments 500 ms long; 13 variables calculated from pupil RPD:
mean, median, min, max, sd, var, entropy, moment_start_speed,
avg_start_speed, moment_end_speed, avg_end_speed, avg_med_speed

Predicting:
1. W vs. I task
2. W vs. I-I vs. I-R
3. W vs. I-R from

from end of trial segments (decision)

from end of trial segments (decision)
from segments on last fixations on relevant word(s)

Jacek Gwizdka

23

Results: Classification cont.

Classification
feature selection to maximize information gain
resampling to obtain balance class membership
Random Forest algorithm with 10-fold cross-validation (Weka 3.8)
In all cases the best four predictors were: mean, median, min, max RPD
#

Classification

Accuracy Chance Acc.

Precision

Recall

ROC Area

1 W vs. I task, end of trials

0.825

0.825

0.925

82.5%

50%

2 W, I-I, I-R task, end of trials

0.748

0.748

0. 912

74.8%

33%

3 W vs. I-R,

0.863

0.863

0. 948

86.3%

50%

last rel. word

Jacek Gwizdka

24

Summary
RQ1.Do people read differently when locating a target word as
compared to when finding needed information?
yes

RQ2. Does pupil dilation differ between the two cases?

yes

Limitations
one type of texts (news stories)
limited information search scenario

Jacek Gwizdka

25

Emerging Model(s)
Model

Jacek Gwizdka

26

TOC

Acknowledgements
Funding: Google Faculty Research Award to Jacek Gwizdka

Contact: NeuroIS2016@gwizdka.com
http:// j.mp / jacek_research

Thank You
Questions?
Jacek Gwizdka

27