Who I Am

Search Analytics for Fun and Profit Information architecture consultant to Fortune 500s
Publisher and founder, Rosenfeld
An Event Apart Media
Chicago, Illinois Blog at www.louisrosenfeld.com
August 27, 2007 Co-author, Information Architecture
for the World Wide Web (3rd ed.,
Lou Rosenfeld 2006; O’Reilly)
www.rosenfeldmedia.com New book: Search Analytics for Your Site:
Conversations with your customers (2008; Rosenfeld
Media): www.rosenfeldmedia.com/books/searchanalytics

Anatomy of a Search Log The Zipf Curve:

(from Google Search Appliance) Short Head, Middle Torso, Long Tail
Critical elements in pink: IP address, time/date stamp, query, and # of results:

XXX.XXX.X.104 - - [10/Jul/2006:10:25:46 -0800] "GET

/search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD
%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-
8&proxystylesheet=www&q=lincense+plate&ip=XXX.XXX.X.104
HTTP/1.1" 200 971 0 0.02
XXX.XXX.X.104 - - [10/Jul/2006:10:25:48 -0800] "GET
/search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD
%3AL%3Ad1&ie=UTF-
8&client=www&q=license+plate&ud=1&site=AllSites&spell=1&oe=
UTF-8&proxystylesheet=www&ip=XXX.XXX.X.104 HTTP/1.1" 200
8283 146 0.16
XXX.XXX.XX.130 - - [10/Jul/2006:10:24:38 -0800] "GET
/search?access=p&entqr=0&output=xml_no_dtd&sort=date%3AD
%3AL%3Ad1&ud=1&site=AllSites&ie=UTF-8&client=www&oe=UTF-
8&proxystylesheet=www&q=regional+transportation+governance+
commission&ip=XXX.XXX.X.130 HTTP/1.1" 200 9718 62 0.17

Keep It In Proportion What’s the Sweet Spot?

7218 campus map Rank Cumul. % Count Query

5859 map 1 1.40 7218 campus map

14 10.53 2464 housing
5184 im west
4320 library 42 20.18 1351 webenroll
3745 study abroad 98 30.01 650 computer center
3690 schedule of courses 221 40.05 295 msu union
3584 bookstore
3575 spartantrak 500 50.02 124 hotels
3229 angel 7877 80.00 7 department of surgery
3204 cata

1
 Where will you
Topical Patterns and Seasonal Changes
Capture Search Queries?
1. The search logs that your search engine
naturally captures and maintains as searches
take place
2. Search keywords or phrases that your users
execute, that you capture into your own local
database
3. Search keywords or phrases that your
commercial search solution captures,
records, and reports on (Mondosoft, Visual
Sciences, Ultraseek, Google Appliance, etc.)

Querying your Queries: Tune your Questions:

Getting started From generic to specific
1. What are the most frequent unique queries? Netflix asks
2. Are frequent queries retrieving quality results? 1. Which movies most frequently searched?
3. Click-through rates per frequent query? 2. Which of them most frequently clicked through?
4. Most frequently clicked result per query? 3. Which of them least frequently added to queue?

5. Which frequent queries retrieve zero results?

6. What are the referrer pages for frequent queries?
7. Which queries retrieve popular documents?
8. What interesting patterns emerge in general?

Diagnose This: User Research:

Fixing and improving the UX What do they want?…
1. User Research SA is a true expression of users’ information
needs (often surprising: e.g., SKU #s at
2. Content Development clothing retailer; URLs at IBM)
3. Interface Design: search entry interface, Provides context by displaying aspects of
search results single search sessions
4. Retrieval Algorithm Modification
5. Navigation Design
6. Metadata Development

2
 User Research: User Research:
…what else do they want?… …who wants it?…

BBC provides
Specific segments needs as determined by:
reports to determine  Security clearance
other terms  IP address
searched within
 Job function
same session
(tracked by cookies)  Account information
 Alternatively, you may be able to extrapolate
segments directly from SA
Pages they initiate searches from

User Research: User Research:

…who wants it?… …and when do they want it?
Time-based variation (and clustered queries) from
BBC’s top MSU
queries report  By hour, by day,
from children’s by season
section of site  Helps determine
“best bets”
development
 Also can help
tune main page
and other
editorial content

Content Development: Content Development:

Do we have the right content? Are we featuring the right stuff?
Analyze 0 result queries Track clickthroughs to
 Does the content exist? determine which results
 If so, there are titling, should rise to the top
wording, metadata, or (example: SLI Systems)
indexing problems
 If not, why not? Also suggests which “best
bets” to develop to
address common queries

BBC removes navigation

pages from search results
From www.behaviortracking.com

3
Search Entry Interface Design:
“The Box” or something else?
Identify “dead end” points (e.g., 0 hits, 2000 hits)
where assistance could be added
Query syntax helps you select search features to
expose (e.g., use of Boolean operators)
OR
Search Results Interface Design:
How to sort results?
Financial Times has found that users often include
dates in their queries
Obvious but effective improvement: allow users to
sort by date
Navigation:
Any improvements?
Michigan State University builds A-Z index
automatically based on frequent queries
Search Results Interface Design:
Which results where?
#10 result is clicked through more often than #s 6, 7, 8, and
9 (ten results per page)
From SLI Systems (www.sli-systems.com)
Search System:
What to change?
Add functionality: Financial Times added spell
checking
Retrieval algorithm modifications
 Financial Times weights company names higher
 Netflix determines better weighting for unique terms
and phrases
Deloitte, Barnes & Noble, Vanguard demonstrate
that basic improvements (e.g., Best Bets) are
insufficient (and justify increased $$$)
Navigation:
Where does it fail?
Track and study pages (excluding main
page) where search is initiated
 What do they search? (e.g., acronyms, jargon)
 Are there other issues that would cause a
“dead end”? (e.g., tagging and titling
problems)
 Are there user studies that could test/validate
problems on these pages? (e.g., “Where did
you want to go next?)
4
Metadata Development:
How do searchers express their needs?
Tone and jargon (e.g., “cancer” vs.
“oncology,” “lorry” vs. “truck,” acronyms)
Syntax (e.g., Boolean, natural language,
keyword)
Length (e.g., number of terms/query; Long
Tail queries longer and more complex
than Short Head)
Everything we know from analyzing
folksonomic tags applies here, and vice versa
Metadata Development:
Leveraging differences in the curve
Variations in information needs emerge between
Short Head and Long Tail
Example: Deloitte intranet’s “known-item” queries
are common; research topics are infrequent
known-item research
queries queries
Organizational Impact:
Reexamining assumptions
Financial Times learns about breaking stories
from their logs by monitoring spikes in
company names and individuals’ names
and comparing with their current
coverage
Discrepancy = possible breaking story;
reporter is assigned to follow up
Next step? Assign reporters to “beats” that
emerge from SA
Metadata Development:
Which values and attributes?
Uncover hierarchy and
identify
 Metadata values (e.g., mobile vs.
cell)
 Metadata attributes (e.g., genre,
region)
 Content types (e.g., spec, price
sheet)
SA combines with AI tools
for clustering, enabling
concept searching and
thesaurus development
Organizational Impact:
Educational opportunities
“Reverse engineer” performance problems
 Vanguard
 Tests “best” results for common queries
 Determines why these results aren’t retrieved or
clicked-through
 Demonstrates problem and solutions to content
owners/authors benefits
 Sandia Labs does same, only with top results
that are losing rank in search results pages
SA as User Research Method:
Sleeper, but no panacea
Benefits
 Non-intrusive
 Inexpensive and (usually) accessible
 Large volume of “real” data
 Represents actual usage patterns
Drawbacks
 Provides an incomplete picture of usage: was user
satisfied at session’s end?
 Difficult to analyze: where are the commercial tools?
Complements qualitative methods (e.g., persona
development, task analysis, field studies)
5
SA Headaches: Please Share Your SA Knowledge:
What gets in the way? Visit our book in progress site
Problems* Search Analytics for Your Site:
Conversations with your
 Lack of time Customers by Louis
 Few useful tools for parsing logs, generating reports Rosenfeld and Richard
 Tension between those who want to perform SA and Wiggins (Rosenfeld
those who “own” the data (chiefly IT) Media, 2008)
 Ignorance of the method
 Hard work and/or boredom of doing analysis
Most of these are going away…
* From summer 2006 survey (134 responses), available at book site.
Site URL: www.rosenfeldmedia.com/books/searchanalytics/
Feed URL: feeds.rosenfeldmedia.com/searchanalytics/

Contact Information

Louis Rosenfeld
Rosenfeld Media, LLC
705 Carroll Street, #2L
Brooklyn, NY 11215 USA

+1.718.306.9396

lou@louisrosenfeld.com

www.louisrosenfeld.com
www.rosenfeldmedia.com