Vous êtes sur la page 1sur 7

ASME District F - Early Career Technical Conference Proceedings

ASME District F - Early Career Technical Conference, ASME District F ECTC 2013
November 2 3, 2013 - Birmingham, Alabama USA

COGNITIVE EVIDENCE IN ENGINEERING DESIGN DOCUMENTATION


Sophoria Westmoreland
United States Naval Academy
Annapolis, Maryland, USA

ABSTRACT
In order to process knowledge during the engineering
design process certain cognitive tools are necessary. At a
surface level those tools are creativity, scientific, and process
knowledge. While some progress has been made recently in
exploring cognitive processes, reading between the lines, and
thinking about design thinking much more work is yet to be
done in this expansive field.
The purpose of this paper is to present a method of
extracting cognitive evidence from engineering design
documentations-specifically capstone design journals from
undergraduate students-and the results from its application.
Attempting to reveal cognitive processes is a complex science,
as such methods and tools should be created to explore the
unknown realms of the engineers mind. Using different types
of engineering design documentation is one path to retrieving
cognitive information.
Capstone design journals are examined as part of a larger
study that partially fulfilled the requirements for the authors
dissertation research. A Cognitive Coding Scheme was created
by the author to explore evidence of design thinking and
behavior. This paper seeks to identify patterns of behavior
found in a capstone design team using hand written design
journals.
INTRODUCTION
A wide array of engineering design studies on cognition
exist in literature that combine research from the engineering
and psychology domains [1-5]. This present work effort is a
part of a larger work which includes the authors dissertation
[6]. The goal of this work is to contribute to the understanding
of cognitive processes during engineering design.
The mixture of art and engineering is what design is all
about. Designers use what they know to create some new
artifact. Executing this skill requires the use of cognitive
activities that are the evidence of the process of a designers
thinking. Some examples of cognitive activities are analogical
thinking, questioning, and inquiry. The minds arrangement of
this information is used to energize the art of innovation.

ASME District F - ECTC 2013 Proceedings - Vol. 12

Linda Schmidt
University of Maryland College Park
College Park, Maryland, USA

This study proposes a method for understanding those


cognitive methods and seeing how they are organized. The
results would support good design education and training
needed to produce quality engineers prepared to lead a global
society. According to one source promoting innovative thinking
can drive future economic growth and continue to lead on the
global stage [7].
Many things can be revealed by studying written
documentation. Thomas Jeffersons letters, Albert Einsteins
paper, and Leonardo da Vincis mirror writings are examples of
famous written documentation [8-10]. Visualizing the design
process can result in many forms of design documentation such
as design journals, final reports, presentations, notes, and
sketches. Using cognitive research techniques (i.e. a cognitive
coding scheme on students engineering design journals), this
study seeks to understand what happens in the mind during the
design process.
LITERATURE REVIEW: COGNITIVE PSYCHOLOGY
AND ENGINEERING DESIGN
According to Cross et al. design activities are among those
occurring at the highest possible human cognitive levels [11].
One definition of design from the engineers perspective is to
pull together something new or arrange existing things in a new
way to satisfy a recognized need of society [12]. A general
understanding of cognition and engineering design activities is
needed in the creation of a cognitive coding scheme.
Cognition
Research begins with starting with what we know. William
James says it clearly the first fact is that thinking of some sort
goes on [13]. Cognitive processes are a part of everyday life,
from the smallest tasks to the larger ones. Studying cognitive
processes reveals how people organize and use knowledge in
daily life and work situations. Many researchers have
successfully studied the mysteries of the mind, knowledge, and
thought processes [14-16]. These studies increase our
understanding of human thinking processes and maximize
usefulness of the tools created to promote learning methods and
learning metacognitive strategies. Common researcher
questions are, what is knowledge and where does it come from?

236

According to Alexander et al., knowledge is an individuals


personal stock of information, skills, experiences, beliefs, and
memories[17].
It is important to note the interdisciplinary nature of the
fields of study that examine cognition. They include cognitive
science, cognitive psychology, and educational psychology.
These fields all have their foundation in psychology. Studies in
these fields can be complementary and are all focused on the
central theme of understanding the mind. Studying the mind
creates a path to understanding human behaviors.
Cognition in Design
The original way to learn about cognitive activities in
design was to study designer behavior. Studying designer
behavior can be done using a variety of methods such as verbal
protocol analysis [18], design prompts [19], direct observation
[20], coding design journal content [21, 22], and interviewing
designers [23]. Different methodologies were used in each
study based, at least partially, on the anticipated results. It is
useful to note that studying design does not solely belong to the
field of engineering. Of the studies mentioned above some are
from architecture, industrial design, and other engineering
disciplines that have design as a central tenant. Some of these
studies have focused on designer behavior and in the process
uncovered cognitive findings. The studies detailed below focus
on students design activities because they are the subjects used
in this current paper.
ABET requires students in engineering undergraduate
programs to take a course in capstone design towards the end of
their course studies[24]. Each engineering discipline is different
in how they present the course, but the goal is generally for
students (working in a team) to create an original design
product using the culmination of engineering subject
knowledge acquired during the previous years of study. This
course provides the perfect opportunity to study student design
behavior in a natural design setting. A deeper understanding of
team behavior is often also a result of these types of studies.
Grenier et al. analyzed design journal sketches and
notations of capstone design students to learn how students are
learning and practicing design [25]. Twelve student design
journals were used from a senior design course. The results
showed that positive links exist between sketching and
cognitive processes. Two cognitive operators were displayed in
the student sketches, generation and exploration; both are
critical for solving complex problems. Greniers study presents
promise for learning more about the cognitive processes of
engineering design students.
Another important study was done on team communication
by Stempfle [26]. The goal was to examine the thinking
processes of student design teams. Teams of mechanical
engineering design students were recorded (4-6 students per
team) designing a sun planetarium for six consecutive hours. A
coding system was created to analyze the team communication
in order to create a model of design team activity. Four basic
operations of design thinking were identified: generation,
exploration, comparison, and selection.

ASME District F - ECTC 2013 Proceedings - Vol. 12

A study by Shah fully integrated traditional engineering


based design with traditional psychology based labs [27-29].
The goal was to study the cognitive processes that happen
during the engineering design process through an ideation
experiment to create a cognitive model of the design process.
Models were pulled from cognitive psychology related to
information processing such as human problem solving, mental
imagery, and visual thinking. The resulting preliminary
cognitive model included six ideation components: provocative
stimuli, suspended judgment, flexible representation, frame of
reference shifting, incubation, and example exposure. The
design teams in the study were exposed to the six ideation
components, and the results were recorded. The study
concluded that introducing ideation components has a positive
effect on divergent thinking during idea generation. This
created a basis for linking cognitive psychology with
engineering design studies through a cross-disciplinary study
using terms from both fields.
Sobek implemented design journals for a senior capstone
design course at Montana State University in order to find the
correlation between thoughts and written documentation during
the design process[21, 22]. The students kept a design journal
and received a portion of their course grade for the contents.
Each member of the design team was required to keep a journal
documenting the process. A coding scheme was created and
applied to the design journals to find out what design process
variables affect the design outcome. It was concluded that
design process models do not suit novice designers as well as
they do expert designers. Sobek noted the importance of the
potential cognitive benefits students gain from using a design
journal.
This current study utilizes written documentation to create
a cognitive coding scheme for engineering design
documentation.
A COGNITIVE CODING SCHEME
A cognitive coding scheme is defined as a system
developed for the classification of design documentation
content for quantitative analysis. Many researchers have
created cognitive coding schemes for application to other types
of design documentation such as verbal protocol analysis and
design interviews [20, 30, 31]. A good cognitive coding scheme
is one suitable for extracting evidence to imply cognitive
activities from design documentation. Application of such a
coding scheme can benefit design researchers progress towards
developing design competency tools and help clarify the
differences between novice and professional design engineers.
The development of the coding scheme began with a
detailed literature search (both cognitive psychology and
engineering design) using an iterative process. A series of 4
design journal studies were conducted with students in senior
design courses at the University of Maryland- College Park to
apply and refine the coding scheme. The cognitive codes were
validated against similar coding schemes found in literature
[20, 32, 33]. We have confidence that our final version of the
coding scheme, as shown in Table 1, is useful for revealing the

237

cognitive activities that occur during the design process. The


individual cognitive codes are grouped into larger classes.
Table 1: Cognitive Coding Scheme Class and Related
Codes
Cognitive Codes
INFORMATION SEEKING AND NOTING
Search (1), References (2), Questioning (3), Price
Quotes (4), and Definitions (5)
PROBLEM UNDERSTANDING
Customer Requirements (6), Problem Statement
Clarification (7), Criteria Lists (8), and Engineering
Characteristics (9)
IDEA GENERATION
Project Ideas(10), Analogical Reasoning (11), and
Material Options (12)
ANALYSIS
Estimate (13), Assumptions (14), Calculations (15),
Testing Procedures (16), Variables (17), and
Explanations (18)
DECISIONS
Recommendations (19), Conclusions (20), and Design
Changes (21)
PROJECT MANAGEMENT
To Do Lists (22), Meeting Notes (23), Task Assignment
(24), Inventory (25), Task Completion (26), Project
Milestones (27), and Field Trip Notes (28)
REFLECTION
Personal Notes(29), Design Process Notes (30),
Revelations (31), Mistakes (32), and Cross References
(33)
OTHER
Illegible Entries (34), Designer Signature (35), and No
Evidence of Cognitive Activity (36)
METHODOLOGY: STUDENT DESIGN TEAMS STUDY
A design journal study was conducted in fall 2011 with a
team of students in Mechanical Engineering at the University of
Maryland. This study is one out of a larger group of similar
studies done for a larger work. The team presented in this paper
consisted of 5 members with only N=4 participating in the
study. The students were given a design journal at the
beginning of the course to capture the complete design process
experienced by each student (1 semester lasted 15 weeks). A
one page Design Journal Guidelines gave a brief overview of
the journaling process in the event that a team member was not
familiar with the process. A short presentation was given on the
first day of class introducing the study and giving details about
the journaling process and expectations. The students
participating in this study were volunteers but were
compensated for their time with gift cards to the local college
bookstore.
A journal is defined for this study as a bound notebook
with lined pages used as a permanent record of what happened

ASME District F - ECTC 2013 Proceedings - Vol. 12

during the design process. A sample design journal page is


shown in the Appendix as Figure 6.
The journals were reviewed weekly, and feedback was
given to the study participants. Dates of entries in the journal
are important for correlating them with course due dates and
team meeting times. Hence when the journals were checked
this was a priority and feedback was given to the students if
dates were missing from entries. The content of the journals
was left entirely up to the students in order to give them control
over how they utilize their journals to benefit them in the
design course. Students who participated in this study
completed an exit survey through e-mail.
Coding the Design Journals
Coding the design journals produces a design string
(Figure 1) which is an order of numbers that relate to
components in the coding process. The cognitive code
presented previously represents only one component in the
design string. For breadth and depth this study all captures
other important information from the students design journals.

Figure 1: Elements of the design string assigned to


each segment of a coded journal page
The design session includes all the written records found to
have occurred on a single date or during a single period of
concentration. The design segment is a section of work within
each session in the journal that is focused on the same design
thought, which can be described by a single cognitive code. The
design phases are conceptual design, embodiment design,
detailed design, and re-design in accordance with the course
text [12]. The cognitive codes are from Table 1. The concept
code indicated the concept (if any) to which the segment is
referring. The visual types classify a visual representation
found in the design segment such as sketches and free body
diagrams. A sample coded example is shown in Figure 2.
The main coder is the author of this paper and a trained
undergraduate assistant also coded a representative set of the
design journals in order to perform an inter-coder reliability
study. The Cohens Kappa was calculated at 64.7% at the
cognitive code level, which is a good strength of agreement.
RESULTS AND DISCUSSION
The students reported on in this study are all from the same
capstone design team. The team was made up of five students,
and four out of five of the students participated in the study.
Their project was to design a dynamic coring system, which is
a drill bit stand, over a 15 week period. Detailed information
about the students on this team is given below in Table 2.

238

members. A deeper understanding of the benefits of using a


design journal for a project like this would likely even out these
Conceptual Design
Detail Design
100%

Embodiment
Redesign

75%
50%
Figure 2: Coding Example

25%

Table 2: Detailed Team Information

0%

ID

Number of
Journal
Pages
Recorded

Design
Sessions

Design
Segments

Activity
Density

3
5
6
7

32
28
20
20

20
11
20
9

69
123
101
119

3.45
11.18
5.05
13.22

Variation between the number of design sessions and the


number of design segments is expected. Activity density is
defined as a measure of the amount of journaling activity
within a design session. A lower activity density indicates that
students were writing about a specific topic or had a narrower
focus during some of their design sessions. A higher activity
density indicates a wider range of cognitive activities each time
they sat for a design session.
The design phase coding results are shown in Figure 3. The
students in this study found the design journals to be most
useful during the conceptual design and embodiment design
phases of the design process. These stages involve activities
such as gathering information, conceptualization, concept
development, product architecture, and parametric design.
The cognitive coding results from the study are shown in
Table 3.
For spacing purposes cognitive codes that were not found
in this teams design journals were omitted. These codes are
Definition, Estimates, Analogical Reasoning, Assumptions,
Variables, Meeting Notes, Task Assignments, Inventory, Field
Trip Notes, Task Completion, Design Changes, Mistakes, and
Designer Signature. These results present a good representation
of the cognitive codes across the students design journals.
Student 3 produced the lowest number of different types of
cognitive activities with only 9 (out of a possible 36). Student 7
produced the highest number of different types of cognitive
activities with 15 (out of a possible 36).
All 4 students in this study used their design journals for
Project Ideas more than anything else. It is clear that the journal
provided a convenient space to document their creativity and
share ideas for solutions to the problem amongst the group

ASME District F - ECTC 2013 Proceedings - Vol. 12

Student 3 Student 5 Student 6 Student 7


Figure 3: Design Phase Results
Table 3: Cognitive Code Results as a Percent
Students
Cognitive Code
3
5
6
7
Search
0%
0%
0%
0.84%
References
0%
13.82%
0%
0%
Questioning
4.35%
2.44% 3.88%
0%
Price Quotes
0%
0%
0%
0.84%
Customer
1.45%
3.25% 2.91% 3.36%
Requirements
PS Clarification
1.45%
1.63%
0%
2.52%
Criteria List
0%
0.81%
0%
7.56%
Engineering
0%
4.88%
0%
0%
Characteristics
Project Ideas
65.2%
29.2% 61.1% 53.78%
Material Options
7.25%
0%
1.94%
0%
Calculations
1.45%
2.44% 0.97% 0.84%
Testing
5.80%
9.76% 5.83% 1.68%
Procedures
Explanations
0%
5.69% 5.83%
0%
Recommendations
0%
0%
4.85%
0%
Conclusions
0%
6.50% 5.83% 5.04%
To Do Lists
10.14% 7.32% 0.97% 1.68%
Project
0%
0%
0%
0.84%
Milestones
Personal Notes
0%
0%
2.91% 1.68%
Design
Process
0%
0%
0.97%
0%
Notes
Revelations
0%
0%
0.97%
0%
Cross References
0%
0%
0%
2.52%
Illegible Entries
0%
1.63%
0%
2.52%
None
2.90%
8.94% 0.97% 14.29%
results. It would have been expected to see Engineering
Characteristics appear more in these design journals because of
the course requirement to make a House of Quality that

239

includes an engineering characteristics room. Only 1 student


had engineering characteristics entries in their design journal.

Figure 4: Cognitive Codes by Class


Figure 4 is suggestive of the behavior of the students, in
that they mostly used the journals the same. With the exception
of student 5, they all show a peak during idea generation. The
cognitive codes corresponding to each of the classes can be
found in Table 1. Understanding time constraints for the design
project, other course requirements, and the voluntary nature of
this study these are definitely promising results.
1

100%

100%
80%

60%

40%

40%

20%

20%
7

80%

60%

0%

0%

Student 5

1
100%
8

1
80%

80%

2
8

60%
40%

REFERENCES

40%

20%
7

60%

FUTURE WORK
An important next step is combining the cognitive coding
scheme results with a design performance measure to
understand the relationship between good design and
cognitive activities. It would be essential to involve
professional design experts in this process to identify student
projects that are of high quality. Creating a design performance
measure can effectively codify innovation early in the design
process. This path will lead to an increased understanding of
innovative design thinking.
ACKNOWLEDGEMENT
This material is based in part on work done for the authors
dissertation. The authors are grateful for the student volunteers,
coding assistants, and the Mechanical Engineering department
at the University of Maryland for their support and assistance
with this work.

Student 3

code classes found in Table 1. The conceptual design phase is


mirrored for 3 of the 4 students participating in this study.
Students use of visual representations in the design
journals is important for this work because visuals are tools
used for understanding, explaining, modeling, and creating
during the design process. All the visuals found in the 4
students design journals were sketches, which is not surprising.
Sketching in engineering design is vital to innovation and the
outpouring of ideas from the mind. Many of these sketched
ideas become the final design or at least a subsystem.
The comprehensive cognitive coding scheme presented
here is shown to be a prescriptive method for extracting
cognitive evidence from engineering design journals. The use
of the design string coding method allows for quantitative
analysis and the collection of a rich data set. One of the
benefits to having students journal in a non-prescriptive manner
is to be able to see the differences in their behavior.
This research reveals the patterns of journaling behavior in
different phases of the design process. The larger study
generated a large amount of data that will provide research
results into the future. Working with educational psychologist
these results can be used to create tools for teaching innovative
strategies in engineering design courses.

20%
3

0%

4
5

0%

4
5

Student 6

Student 7
Figure 5: Cognitive Codes during Conceptual Design
Taking a closer look at the students design journal
behavior during the conceptual design phase is shown in Figure
5. The numbers in the figure correspond with the cognitive

ASME District F - ECTC 2013 Proceedings - Vol. 12

[1] Atman, C.J., Kilgore,D., And Mckenna, A., Characterizing


Design Learning: A Mixed-Methods Study Of Engineering
Designers' Use Of Language Journal Of Engineering
Education, 2008. 97(3): P. 309-326.
[2] Ball, L.J., J.S.B.T. Evans, And I. Dennis, Cognitive
Processes In Engineering Design: A Longitudinal Study
Ergonomics, 1994. 37(11): P. 1753-1786.
[3] Cross, N. The Expertise Of Exceptional Designers In
Creativity And Cognition Conference: Expertise In Design
2003. Sydney, Australia
[4] Liang, J., Et Al. A Meta-Cognition Modeling Of
Engineering Product Designer In The Process Of Product

240

Design In International Conference On Human-Computer


Interaction Proceedings Part 1 2007. Beijing, China.
[5] Williams, C.B., Et Al. Exploring The Effect Of Design
Education On The Design Cognition Of Mechanical
Engineering Students In Asme 2011 International Design
Engineering Technical Conferences & Computers And
Information In Engineering Conference 2011. Washington, Dc,
Usa
[6] Westmoreland, S., Design Thinking: Cognitive Patterns In
Engineering Design Documentation
In Mechanical
Engineering2012, University Of Maryland - College Park:
University Of Maryland Digital Repository In College Park,
Maryland.
[7] Obama, B. A Strategy For American Innovation: Securing
Our Economic Growth And Prosperity 2011 [Cited 2011
October 12].
[8] Da Vinci, L., Leonardo's Notebooks Ed. H.A. Suh2005,
New York.: Black Dog & Leventhal Publishers, Inc. .
[9] Gorman, M. And W.B. Carlson, Interpreting Invention As A
Cognitive Process: The Case Of Alexander Graham Bell,
Thomas Edison, And The Telephone. Science, Technology, &
Human Values, 1990. 15(2): P. 131-164.
[10] Gorman, M., Confirmation, Disconfirmation, And
Invention: The Case Of Alexander Graham Bell And The
Telephone. Thinking And Reasoning, 1995. 1(1): P. 31-53.
[11] Cross, N., Christiaans, H., And Dorst, K. , Analysing
Design Activity 1996, West Sussex, England: John Wiley &
Sons Ltd.
[12] Dieter, G.E. And L. Schmidt, Engineering Design 5th
Edition. 4 Ed. Vol. 5. 2012, New York, New York.: McgrawHill Higher Education
[13] James, W., The Principles Of Psychology: Edited By
Christopher Green, Ed. C. Green1890, Toronto.: Self Published.
[14] Vygotsky, L.S., Thought And Language 1986, Boston: The
Mit Press.
[15] Lave, J., Cognition In Practice 1988, Cambridge.:
Cambridge University Press.
[16] American Psychological Association Division 15, A.E.P.,
Educational Psychology: A Century Of Contributions, Ed.
B.J.Z.A.D.H. Schunk2003, Mahwah, New Jersey Lawrence
Erlbaum Associates, Inc. .
[17] Alexander, P.A., D.L. Schallert, And V.C. Hare, Coming To
Terms: How Researchers In Learning And Literacy Talk About
Knowledge Review Of Educational Research, 1991. 61(3): P.
315-343.
[18] Gero, J.S. And T. Mcneill, An Approach To The Analysis
Of Design Protocols Design Studies, 1998. 19(1): P. 21-61.
[19] Bogusch, L.L., Turns, J., And Atman, C.J. . Engineering
Design Factors: How Broadly Do Students Define Problems. In
30th Asee/Ieee Frontiers In Education Conference. 2000.
Kansas City, Mo.
[20] Suwa, M., Purcell, T., And Gero, J. , Macroscopic Analysis
Of Design Processes Based On A Scheme For Coding
Designers' Cognitive Actions Design Studies, 1998. 19(4): P.
455-483.

ASME District F - ECTC 2013 Proceedings - Vol. 12

[21] Sobek, D.K. Use Of Journals To Evaluate Student Design


Processes In Asee Annual Conference. 2002. Montreal, Qe.
[22] Sobek, D.K. Preliminary Findings From Coding Student
Design Journals In Asee Annual Conference 2002. Montreal,
Qe.
[23] Chimka, J.R., Cynthia, J.A., And Bursic, K.M. .
Describing Student Design Behavior In Asee Annual
Conference. 1997. Honolulu, Hi.
[24] ABET, Criteria For Accrediting Engineering Programs In
General Criteria For Baccalaureate Level Programs 2009-2010,
Engineering Accredidation Commission Baltimore, Maryland
[25] Grenier, A. And L. Schmidt. Analysis Of Engineering
Design Journal Sketches And Notations In Asme International
Design Engineering Technical Conferences And Computers
And Information In Engineering Conference 2007. Las Vegas,
Nv.
[26] Stempfle, J. And P. Badke Schaub, Thinking In Design
Teams- An Analysis Of Team Communication Design Studies,
2002. 23(5): P. 473-496.
[27] Shah, J., Kulkarni, S.V., And Vargas-Hernandez, N. ,
Evaluation Of Idea Generation Methods For Conceptual
Design: Effectiveness Metrics And Design Of Experiments
Journal Of Mechanical Design, 2000. 122(4): P. 377-384.
[28] Shah, J., Vargas-Hernandez, H., And Smith, S., Metrics
For Measuring Ideation Effectiveness. Design Studies, 2003.
24(2): P. 111-134.
[29] Shah, J., Smith, S., Vargas-Hernandez, H., Gerkins, D. R.,
And Wulan, M. . Empirical Studies Of Design Ideation:
Alignment Of Design Experiments With Lab Experiments In
Asme Idetc/Cie Design Theory And Methodology Conference.
2003. Chicago, Il: Asme
[30] Atman, C.J., And Turns, J. , Studying Engineering Design
Learning: Four Verbal Protocol Studies, In Design Knowing
And Learning: Cognition In Design Education C. Eastman,
Mccracken, M., And Newstetter, W., Editor 2001, Elsevier:
Kidlington, Oxford
[31] Adams, R.S., Cognitive Processes In Iterative Design
Bahavior In College Of Education 2001, University Of
Washington Seattle, Wa.
[32] Atman, C.J., Et Al., Engineering Design Processes: A
Comparison Of Students And Expert Practitioners. Journal Of
Engineering Education, 2007. 96(4): P. 359-379.
[33] Jain, V.K. And D.K. Sobek, Linking Design Process To
Customer Satisfaction Through Virtual Design Of Experiments
Research In Engineering Design, 2006. 17(2): P. 59-71.

241

APPENDIX

Figure 6: Design Journal Page Example

ASME District F - ECTC 2013 Proceedings - Vol. 12

242