ISTE Workshop

Research Methods in Educational Technology

IIT Bombay February 2-9, 2013

Session 3:
What is the referee looking for in your paper?

What do referees look for

Referees look for
Novelty Positioning

Your paper must have

Analysis of prior work to show that your idea is unique Analysis to show that your work is required, how your work advances the state of the art

Soundness Steps to show that you have of procedure implemented solution carefully Evidence to Data to show that your solution support claim works as claimed Overall coherence Consistency between parts of your paper treatment should address problem, results should give answer to problem

What do referees look for

Referees look for
Novelty Positioning

Your paper must have

Analysis of prior work to show that your idea is unique Analysis to show that your work is required, how your work advances the state of the art Steps to show that you have implemented solution carefully

We discussed this in
Is this an acceptable paper, features of published papers this session

Soundness of procedure

Analysis of strong paper Features of published papers, analysis of strong/ weak paper activity Analysis of strong / weak paper activity

Evidence to Data to show that your solution support claim works as claimed Overall coherence Consistency between parts of your paper treatment should address problem, results should give answer to problem

Is the paper positioned well?

1) Has your paper analyzed related prior work?
a) Have you referred to other papers that have addressed a problem similar to yours? b) Have you referred to other papers that have a solution approach similar to yours? c) Have you clearly brought out the gaps?

This will bring out the need for your research study. 2) Does your solution address any of the gaps above? This will bring out the novelty of your solution. 3) Is your solution based on the application of appropriate educational theory? This will bring out the soundness of your solution.

Have you referred to other papers that have addressed a problem similar to yours
Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories (Section II A) Several simulations have been formulated for electronics labs. Circuit Tutor, an electronic simulation for building circuits at the University of Illinois, is embedded in a virtual classroom that includes the simulation along with online discussions.
Brief Description of what they do

The Electronic Workbench [2] is a popular circuit capture and simulation system that is frequently used for education and training. A simulation with a long history and positive results, it can be used with circuit design software to create and try out various circuits.
Brief Description of their results

Have you referred to other papers that have a solution approach similar to yours
Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories (Section II A) The system (Circuit Tutor) continues to use lectures to teach theory. The Electronic Workbench package includes tutorials and reference information, but without the detailed coaching of the ELS.
Identification of their drawbacks

Both Circuit Tutor and Electronic Workbench may reduce the cost and time of laboratory experiences, but the efficacy of these simulations compared with that of physical equipment labs is not well understood. The studies reported here (ELS) investigate the extent to which laboratory simulations of electronics circuits that add realistic graphic representations of equipment, with immediate computer coaching, may replace some physical electronics laboratories.

Have you clearly brought out the gaps in related work?

Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories(Section II A) Both Circuit Tutor and Electronic Workbench may reduce the cost and time of laboratory experiences, but the efficacy of these simulations compared with that of physical equipment labs is not well understood.

Gap

Does your solution address the gaps?

Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories (Section II A) Gap Both Circuit Tutor and Electronic Workbench may reduce the cost and time of laboratory experiences, but the efficacy of these simulations compared with that of physical equipment labs is not well understood. The studies reported here investigate the extent to which laboratory simulations of electronics circuits that add realistic graphic representations of equipment may replace some physical electronics laboratories. How authors solution addresses gap

Is your solution based on application of appropriate educational theory?

Example from PaperA - The Effectiveness of Learning Simulations for Electronic Laboratories (Section II B) Active Learner Construction and Use of Knowledge and Skills Learning simulations typically require job-like performance by learners; thus, active learning is inherent in the methodology ([6], [7]). Campbell and Gibbons [8] also describe progressively challenging evaluative simulations that integrate assessment, learning and performance support. Why this theory is Theory appropriate to apply in this case

Recap: Positioning your paper

1. First, have good research to write about
Claims must match your results Results must be a significant improvement over known results Results are more convincing if they are confirmed in different ways

2. Focus on presenting one key idea, for a conference paper

What precisely was your research question? What exactly is your new result? Use verbs that show results and achievement, not just effort and activity

3. Explain the relation to other work clearly (See next slide)

Explain the relation to other work clearly

Awful The galumphing problem has attracted much attention [3,8,10,18,26,32,37] Smith [36] and Jones [27] worked on galumphing. Bad

Poor
Good

Smith [36] addressed galumphing byblitzing, whereas Jones [27] took a flitzing approach Smith's blitzing approach to galumphing[36] achieved 60% coverage [39]. Jones [27] achieved 80% by flitzing, but only for pointer-free cases [16]. (Good Above) + We modified the blitzing approach to use the kernel representation of flitzing and achieved 90% coverage while relaxing the restriction so that only cyclic data structures are prohibited.

Better

Source: Mary Shaw, Writing good Software Engineering Research Papers, ICSE 2003

Positioning your paper

Idea Proposal Form Next Assignment Let us walk through the Assignment

Home-works
1. Go through all the slides posted for todays session 2. Watch the videos posted for
1. How to read a paper 2. How to do literature survey

3. Find three papers related to your idea 4. Fill out the Idea Proposal Form
Posted on Moodle

Concluding remarks for the day

What is next. Next hour Reflection, Q & A Next week Feb 3-8 Complete Idea Proposal Assignment, think how to apply what you learned today to go from ET Practitioner to ET researcher Next Saturday How to set up ET research study, methods to evaluate your solution, more activities! Week after next Saturday Feb 10-16 Final Assignment And after that