Vous êtes sur la page 1sur 75

DD3001 - Research: Theory, Method, Practice

Motivation

Hope DD3001 helps you avoid this situation

The Significance of Research

9-Oct-10

Am assuming you already avoid this


Obsolete Scientific Method

Computer Scientific Method

Hypothesis

Hunch

Sequence of experiments.

1 experiment &
change all parameters

Change 1 parameter per


experiment.

Discard if it doesnt support


hunch

Prove/Disprove Hypothesis.
Why waste time?
Document for others to reproduce results.

We know this.

Source: How to have a bad career in research/academia by David Patterson, Feb 2002.

Remember Science/Research is Hard

Why is science hard?


Intrinsic - Our knowledge is imperfect. Makes it difficult to

correctly update this knowledge.


Personal - Scientists have biases and flaws in their approach.
Community - The way we organize ourselves can limit

progress.

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Reason for Difficulty: Intrinsic

All possible theories

True theories

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

All possible theories

Theories we think are true


True theories

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Much of what we consider today is probably wrong


Our successors will be amazed by the
amount of scientific rubbish discussed
today that is, if they have the patience
to trawl through the electronic archives of
obsolete journals.
Sydney Brenner, (1927)
Sydney Brenner is a South African biologist and a 2002 Nobel prize in
Physiology or Medicine laureate.

All possible theories

Theories we
think we have
tested well
Theories we think are true
True theories

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Truth perhaps beyond our current imagination


..my own suspicion is that the Universe
is not only queerer than we suppose, but
queerer than we can suppose
J.B.S. Haldane, (1892-1964)
J.B.S. Haldane was a British-born geneticist and evolutionary biologist.

All possible theories

Theories considered

Theories we
think we have
tested well

Theories we think are true


True theories

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Facts are simple in retrospect


Somebody once observed to the eminent philosopher Wittgenstein
how stupid medieval Europeans living before the time of
Copernicus must have been that they could have looked at the sky
and thought that the sun was circling the earth. Surely a modicum
of astronomical good sense would have told them that the reverse
was true. Wittgenstein is said to have replied:

I agree. But I wonder what it would


have looked like if the sun had been
circling the earth.

Source: James Burke. The Day the Universe Changed. TV Series 1985

Why we need help doing this?


Isnt science just common sense?
Yes, but it is advanced common sense.
Both lay observers and experts tend to...
See patterns in random data.
Generalize from incomplete samples.
See confirmation for favored ideas in ambiguous data.
We tend to believe...
What we already think we know.
What we hear second-hand.
What our community believes.
Source: T. Gilovich. How We Know What Isnt So. 1991.

Reason for Difficulty: Personal

Personal failings: the human condition


Slop - Presenting and doing research so it is not possible to

know what was done or observed


Confused or unclear procedures
Confused or unclear data-collection
Imprecise theorizing
Unexpressed assumptions
Informal derivation of predictions
Sloth - Doing too little!
Important data not obtained or recorded
Partial analysis of the data
Source: Donald D. Jensen. Unpublished lecture notes, University of Nebraska-Lincoln. 1995.

Personal Failings: Self-deception


Precipitance - Jumping to a conclusion

? Premature decision on an issue


? Accepting as established something that deserves further
investigation
Propaganda - Biased presentation of a theory or data

? Special pleading
? Salesmanship rather than science.
Prejudice - Biased evaluation of theory and data

? Expect more of other theories than of ones own


? Tilt the playing field in favour of ones own theory
Preservation - Clinging to a theory despite clear evidence

that it is false.
Source: Donald D. Jensen. Unpublished lecture notes, University of Nebraska-Lincoln. 1995.

Not a new idea (Francis Bacon 1600s)


Idols of the Tribe We see evidence for our favoured

hypotheses in random data.


Idols of the Cave Our personal likes and dislikes affect our

professional judgments.
Idols of the Marketplace Common language is too limited

for accurate scientific communication.


Idols of the Theatre Poor methods and philosophical

systems can doom scientific investigations.

Personal failings: Willful deception


Finagle - Adjusting data so that it fits a favoured theory.

? Minor fraud.
Filch - Stealing ideas or data without giving appropriate credit

? Plagiarism or other unauthorized use of the work of others.


Fraud - Falsifying data and the reports of investigations

performed.

Source: Donald D. Jensen. Unpublished lecture notes, University of Nebraska-Lincoln. 1995.

Reason for Difficulty: Community

Community failings
Science is a human community
Like any other community, it has...

7 Fads and fashions


7 Rigid structures and traditions that may interfere with progress
7 Disagreements over goals and direction that consume resources
7 Outsize personalities who may detrimentally dominate the field
Science progresses in spite of these problems.

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Community benefits
Science is a human community
Like any other community, it can...

X Share data, results and theories


X Quickly explore fads and fashions
X Give independent evaluation of ones own theories and work
X Provide competition and deadlines
Science progresses because of these benefits.

Havent scientists overcome these challenges?


X Yes
With practice
When using tools developed over the past few hundred

years.
7 No when unaided and having little practice.
Most undergraduate education trains you how to learn

what is already known, not how to learn what is currently


unknown or misunderstood.

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

DD3001 - Research: Theory, Method, Practice

Course Overview

Answer to your questions


1. What is this course about?
We will review how research gets done and the important
components of your research career. More details...

2. Do I have to attend all the lectures?


No, but at least 4. But you do have to do all the assigned readings and
responses. Responses for non-attended lectures will be longer as you will
have missed the chance to participate in discussions during the lecture.

3. How can I access the reading material?


I have password protected the website containing the reading
material. I have put the password on the board.

Goals of DD3001
Inform you about the research process help your

burgeoning research career.


Raise your awareness that progress in research is very

non-linear. And requires hard work and perseverance!


Acquaint you with basic strategies for your research career.
? Identify important and solvable research topics, questions and
hypotheses.
? Select between alternative research directions.
? Importance of communication via reading and writing papers.

Make you consider some of the responsibilities of being a

member of the research community.


Teach you some strategies for clear and effective writing.

Outline of the lecture topics


1. Choosing a research topic.
? Strategy and Tactics for Discovery

2. You and your code; Reproducible Research.


3. Communication with other scientists:
Reading & Reviewing papers.
4. Communication with other scientists:
Paper writing.
5. Ethics.
6. Last lecture may be devoted to editing of a paper!
In most of the lectures will devote some time to writing tips.

Papers and texts


Papers & Essays

? General advice to young researchers


? Historical perspective on discoveries
? Stimulating papers for discussion
Selected chapters from these books

? Advice to a Young Scientist by Peter Medawar.


? The Incomplete Guide to the Art of Discovery by Jack E.
Oliver.
? Style - Lessons in Clarity and Grace by Joseph M.
Williams.

Course structure
Standard lectures interspersed with

? discussion of the pre-assigned readings,


? discussion of issues raised during the lectures,
? jotting down instant responses to questions.

What you will do in this course!


Read some interesting articles and essays about science
Think about their content.
Reflect on their relevance to your thoughts about scientific

research.
Summarize and write down these thoughts and reflections.
Write 2 essays to investigate an issue in more depth.

Details of assessment
Course is pass/fail.
To pass you must
Read the Medawar book and write a 400 word response.
Complete and submit the reading responses for the pre-lecture

reading assignments to an acceptable level.


Complete and submit two essay assignment of sufficient

merit. See course website for details.

Details of assessment
Course is pass/fail.
To pass you must
Read the Medawar book and write a 400 word response.
Complete and submit the reading responses for the pre-lecture

reading assignments to an acceptable level.


Complete and submit two essay assignment of sufficient

merit. See course website for details.

Remember you have to pass DD3001 to get your PhD.

Reading responses
At least 3 paragraphs
Upload response to bilda.
Content of response
? One paragraph summarizing main points in assigned reading.
? Make two or more key points that reveal the relevancy of the
paper to you or doing research.
Devote at least one short paragraph to each point.
Focus does not have to be the main point of the paper. But

the ones that are most relevant to you.

This course is a work in progress


Second time Im running this course and it is evolving. So

please be patient !
However, hopefully together we can have an interesting

journey exploring the issues raised.


Dont expect the final word on how to do research.
Expect ideas, partial and potentially conflicting answers
Please engage in discussions. Question what I say. By correct

probing we should closer to the truth!

Selecting Research Projects & Making


Discoveries
DD3001

October 14, 2013

Choose your research problem carefully


What problem to work on?
Most important decision you are going to make as a researcher
Dont choose a problem that is
?

trivial

trivial results

too hard

years of frustration & little accomplishment

in a crowded field

hard to get noticed or be first

Choose your research problem carefully


What problem to work on?
Most important decision you are going to make as a researcher
Dont choose a problem that is
?

trivial

trivial results

too hard

years of frustration & little accomplishment

in a crowded field

hard to get noticed or be first

Are you working on a problem because...

It was the first one that came to mind when you started.
Suggestion made by a professor.
It is related to what colleagues are working on.

Are you working on a problem because...

It was the first one that came to mind when you started.
Suggestion made by a professor.
It is related to what colleagues are working on.

A good basis for making the choice on how youll spend your
time for the next n years?

Careful deliberation is necessary

Select a problem

? whose solution will be important to the flow of science


? that is capable of being solved by you given your skills,
experience and talents.
As a young scientist

? this does not mean you have to limit yourself to problems of


modest significance.

? Remember
Significance not necessarily correlated with difficulty.

Tips for selecting a basic research project

1. Spend time and think very hard about the matter.


2. Do not settle for what happens to be at hand.
3. Choose the most fundamental significant and most
fundamental problem that may yield to your capabilities and
endeavour.

Tips for selecting a basic research project

1. Spend time and think very hard about the matter.


2. Do not settle for what happens to be at hand.
3. Choose the most fundamental significant and most
fundamental problem that may yield to your capabilities and
endeavour.

Do you agree? How did you make your choice?

!"#$%#&'(')$*+,#

63@"B("4
!#%%&AA

CDE&%5&8(F3-#&

!"#$%&'()*+,-)'.++(&$)/&,01(23(

6$"=3=?-?5@(
"4(!#%%&AA

!"#$%&'()*(+"&,-&(./0012(345&$(6*(7&8393$(.:;,&(<$5("4(5,&(!"-#=-&>2

Source: C. Loehle (1990) after P. Medawar (The Art of the Soluble)

!"#$%#&'(')$*+,#

63@"B("4
!#%%&AA

CDE&%5&8(F3-#&

!"#$%&'()*+,-)'.++(&$)/&,01(23(

6$"=3=?-?5@(
"4(!#%%&AA

How would you draw this curve?!"#$%&'()*(+"&,-&(./0012(345&$(6*(7&8393$(.:;,&(<$5("4(5,&(!"-#=-&>2


Source: C. Loehle (1990) after P. Medawar (The Art of the Soluble)

Remember

Competent research results can be ignored.

Why? 7 Worked on a trivial problem.


Modest work can produce a paper of great consequence.

Why? X Brilliance is in the choice of problem and the


approach of the solution.

A successful person isnt necessarily better than her less


successful peers at solving problems; her
pattern-recognition facilities have just learned what
problems are worth solving.
Ray Kurzweil

Interesting CS research problems


Explore or explain the behavior of algorithms, systems,

protocols, and other computational artifacts


Typical form of research questions:

? Why...
? How...
? Under what circumstances...
? What are the necessary and sufficient conditions for...
Answers

? 7 not yes/no
? Xparagraphs, mathematical or statistical models, simulations,
etc.
Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Factors to consider
Importance

? How important is the research topic within the larger research


and application community?
State of Knowledge

? What do we know already?


? What is the position of the research with respect to the
frontier?
Unique Competence

? Are you uniquely qualified to address this research?


? What is your secret weapon?
Interest

? How much does this research problem interest you personally?


? Do you have a passion for this problem?
Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Assessing importance
Audience

Who will care about the answer?


Impact

Will different answers change...


? ...what research gets done next?
? ...what is done by practitioners?
Longevity

How long will the answer be relevant and important?

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

It is fundamentally the confusion between effectiveness


and efficiency that stands between doing the right things
and doing things right. There is surely nothing quite so
useless as doing with great efficiency what should not be
done at all.
Peter Drucker

How many CS talks begin


Graph of rapid growth in...

? Processing power
? Heat generation
? Relative size or cost of caches
? Size of the Web
? Installed base of specific devices
? ....
Why?

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

The Frontier
Important work typically takes place at the frontier of a field.
You want your research findings to be relevant when they are

published, not just now.


The frontier in CS has two components:

? Changing face of computer science knowledge - a problem


common to all fields of science

? Changing environment of computer technology and


applications - a particularly challenging element of computer
science

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

I skate to where the puck is going to be, not where it has


been.
Wayne Gretzky

Methods for identifying frontiers


New observations or findings that need explanation.
New opportunities provided by instruments, methods of study,

or theoretical frameworks
Converging lines of research that combine to provide new

opportunities.
Emerging needs from outside the field (e.g., Web search,

bioinformatics, ecological monitoring, autonomous vehicles)


New questions or conjectures by researchers with good track

records.
Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Is the frontier special in CS?


Research groups often focus on very specific algorithms or

systems.
Frequently one system per research group.
But...

? ....Underlying concepts and fundamentals are the goal rather


than incidental algorithmic features.

? ...Exploratory work often converges.


? ...Core algorithms and principles emerge.
? ...Leaders in the field often agree on what work is central and
at the frontier .
The frontier is always difficult to identify early on.
Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Is the frontier special in CS?


Research groups often focus on very specific algorithms or

systems.
Frequently one system per research group.
But...

? ....Underlying concepts and fundamentals are the goal rather


than incidental algorithmic features.

? ...Exploratory work often converges.


? ...Core algorithms and principles emerge.
? ...Leaders in the field often agree on what work is central and
at the frontier .
The frontier is always difficult to identify early on.
Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Is the frontier special in CS?


Research groups often focus on very specific algorithms or

systems.
Frequently one system per research group.
But...

? ....Underlying concepts and fundamentals are the goal rather


than incidental algorithmic features.

? ...Exploratory work often converges.


? ...Core algorithms and principles emerge.
? ...Leaders in the field often agree on what work is central and
at the frontier .
The frontier is always difficult to identify early on.
Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Good research is done with a shovel, not with tweezers...


You should find an area where you can get a lot out of it
fast.
Roger Needham

Unique competence
What is it that makes you better able to address this research

question than others?


What is your secret weapon?
Poor answers

? I am smarter than other people


It is rarely true.

? I will work harder, longer, or faster


There is always someone who can do those things better than
you, and that weapon can cause unacceptable losses (harm
to family, friends, and personal happiness).

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Unique competence
What is it that makes you better able to address this research

question than others?


What is your secret weapon?
Poor answers

? I am smarter than other people


It is rarely true.

? I will work harder, longer, or faster


There is always someone who can do those things better than
you, and that weapon can cause unacceptable losses (harm
to family, friends, and personal happiness).

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Always have a secret weapon the biggest computer, a


problem imported from another field that others havent
heard of yet, a fact you stumbled on by being curious
about everything, a friend who is smarter than you are.
Anything, in short, that will give you an unfair advantage
in getting there first.
Herbert Simon

Write down what your or your groups secret weapon?

Always have a secret weapon the biggest computer, a


problem imported from another field that others havent
heard of yet, a fact you stumbled on by being curious
about everything, a friend who is smarter than you are.
Anything, in short, that will give you an unfair advantage
in getting there first.
Herbert Simon

Write down what your or your groups secret weapon?

Personal interest
Focus your work on areas of personal interest.
You will work harder, smarter, and more creatively.
Others will sense your strong interest and want to work with

you, support you work, and credit you with innovations.


Personal interest can be fostered and destroyed.

Track what affects your interest and use that knowledge


One widely shared factor: Speed of progress.

Slide Source: 2009 David Jensen, CMPSCI 691DD - Research Methods for Empirical Computer Science, Univ. of Mass.

Review & Discussion of Tips in


The Incomplete Guide to the Art of Discovery,
Jack E. Oliver

Before we start
Book assumes that researchers want to make discoveries.
What are your motivations for completing a PhD?

? want to learn how to do research?


? want to make discoveries?
? want to understand something?
? want to tackle a certain problem?
? a fun job ?
? .....
Whatever they are, if you remain in research then youll have

a more successful and satisfying career if you are making


discoveries.

Strategies for Discovery


1. Dont Follow the Crowd
2. Rebel, but Wisely
3. Strive to Enhance Serendipity
4. Avoid Science Eddies
5. Study the Earth, and the Science of Geology
6. Seek the Non-questions
7. See Your Era in Long-Term Perspective
8. Go with Intuition
9. Avoid Sidetracking to Trivia
10. Be Competitive, Be a Winner, Be First
11. Argue by Analogy
12. Vision, Hypotheses, and Objective Testing
13. The Strategy of Exploration for Understanding

Strategies for Discovery


Pick and write down what your top 5
1. Dont Follow the Crowd
2. Rebel, but Wisely
3. Strive to Enhance Serendipity
4. Avoid Science Eddies
5. Study the Earth, and the Science of Geology
6. Seek the Non-questions
7. See Your Era in Long-Term Perspective
8. Go with Intuition
9. Avoid Sidetracking to Trivia
10. Be Competitive, Be a Winner, Be First
11. Argue by Analogy
12. Vision, Hypotheses, and Objective Testing
13. The Strategy of Exploration for Understanding

Questions
What strategies are missing from this list?
Which ones would you remove?
Why?
Do they apply to you as a PhD student? Why?

Questions
Have you seen these strategies in action by senior scientists in

your research field?


Which ones?

Questions
Have you got a long-term perspective of your field?
What were people doing a decade or 2 decades ago in your

field?
What was the stimulus for a big discovery in your field?

Tactics for Discovery


1. Adapt and Adopt Instruments and Technique
2. Skim the Cream
3. Minimize Jargon
4. Speak (Listen) to the Earth, and It Shall Teach Thee
5. Go for the Spatial Pattern
6. Skim the Rest of the Volume, Any Volume
7. Do It Yourself
8. The Knack of the Fresh Perspective
9. Choose Your Problem Very Carefully
10. The Curve of Discovery
11. Overcoming the Terminal Paper

Tactics for Discovery


Which of these tactics do you follow?

1. Adapt and Adopt Instruments and Technique


2. Skim the Cream
3. Minimize Jargon
4. Speak (Listen) to the Earth, and It Shall Teach Thee
5. Go for the Spatial Pattern
6. Skim the Rest of the Volume, Any Volume
7. Do It Yourself
8. The Knack of the Fresh Perspective
9. Choose Your Problem Very Carefully
10. The Curve of Discovery
11. Overcoming the Terminal Paper

Questions
What tactics do you or your research group follow that are

not on the list?


Write down examples where Instruments and Technique have

been adapted and adopted by those in your field.


What are the roots of the techniques you use in your research

field?

flattened. Judging whether a given set of observations is significant or important is not always easy. Sometimes, however, the
decision is obvious simply because of the large scale or broadQuestion
scope of the feature under study or because of its role in a larger
context.

t
Q

IME
Where do you think your research topic is on this curve?

Overcoming the 'Terminal Paper"

Questions
How do you keep a fresh perspective on your research?
Do you think it is beneficial to read broadly?
Do It Yourself . In your opinion, what does this mean w.r.t

your field.