CMSC 2011 Measurement Study Report

2011 CMSC Measurement Study Report

How Behavior Impacts Your Measurement

27th Annual Coordinate Metrology Systems Conference Phoenix, Arizona July 2011

CMSC 2011 Measurement Study Report

TABLE OF CONTENTS
Introduction and Scope .......................................................................................Page 4 Project Overview .................................................................................................Page 4 2010 Measurement Study....................................................................................Page 4 2011 Measurement Study....................................................................................Page 5 Measurement Workshop Activity ........................................................................Page 6 Measurement Tasks and Participant Profiles .......................................................Page 8 Observations ........................................................................................................Page 16 Results .................................................................................................................Page 35 Conclusion ...........................................................................................................Page 42 Appendix A ..........................................................................................................Page 44 Appendix B...........................................................................................................Page 48 Appendix C ...........................................................................................................Page 57 Acknowledgements .............................................................................................Page 58

Cover Photography: Courtesy of the Coordinate Metrology Society, Photographer: Dirk Dursharme

Page 2

CMSC 2011 Measurement Study Report

COORDINATE METROLOGY SOCIETY REPORT: How Behavior Impacts your Measurement REPORT AUTHORS: Keith Bevan, National Physical Laboratory (NPL) Trevor Toman, Coventry University

Keith Bevan serves on the Certification Committee of the Coordinate Metrology Society. He is the Training Product Development Manager, Knowledge Services, at the National Physical Laboratory. NPL is the UKs National Measurement Institute, and one of the worlds leading science and research facilities developing and applying the most accurate standards, science and technology available. Hampton Rd | Teddington | Middlesex | UK | TW11 0LW Tel: +44 20 8977 3222 | Email: keith.bevan@npl.co.uk | Web: www.npl.co.uk

Trevor Toman serves as the Academic Committee Chair for the Coordinate Metrology Society. He is the Manager of Dimensional Metrology, Faculty of Engineering and Computing, at Coventry University. Tel: +44 2476 236327 Email: ttoman@cad.coventry.ac.uk | Web: www.cad.coventry.ac.uk

CERTIFICATION COMMITTEE, COORDINATE METROLOGY SOCIETY:

Randy Gruver serves as the Certification Committee Chair for the Coordinate Metrology Society, and has served on the Committee for three years. Randy is an Employee Development Specialist at The Boeing Company in Seattle, WA. Email: mrandell.l.gruver@boeing.com

Page 3

CMSC 2011 Measurement Study Report

INTRODUCTION AND PROJECT OVERVIEW In 2009, the Coordinate Metrology Society (CMS) formed a Certification Committee to explore the need for personnel certification in portable metrology. Their first actions were to research existing certifications, survey the CMS membership at the 2009 Conference, and develop a preliminary Body of Knowledge (BOK). An Established Charter was developed to define the committee membership, establish the reporting responsibility, and update the CMS bylaws to make the Certification Committee a standing committee. Their original objectives were to: 1) justify the need for certification, 2) determine if equipment operators and data processors were the target audience, 3) investigate a partnership in administering certification, and 4) determine if training would be provided by 3rd party organizations such as manufacturers, service providers, academia, and national institutes. CONCLUSIONS DRAWN FROM 2009 CERTIFICATION COMMITTEE WORK Based upon marketing and internal research, the Certification Committee determined a properly structured certification program would be of definite value for the metrology community. Currently, equipment must be calibrated, but the operator, the greatest potential source of error, is not required to be certified. The research also established: Certification should be multi-level to delineate degree of capability and responsibility. Certification should indicate mastery of a core body of knowledge with additional certifications for equipment/software. Hardware/software certification should demonstrate appropriate technical knowledge as well as proficiency. There should be certified examiners for each hardware group. There may be areas where certification would be application specific. CMSC 2010 STATISTICAL STUDY ON SKILLS DEVELOPMENT In 2010, the CMS Certification Committee performed a statistical study at the annual CMSC conference. The study was developed to identify skill gaps in the general metrology community, and was held as an open measurement workshop for conference delegates. In addition to this event, other workshops held at CMSC would relate content-to-data developed in the study. The basis of this first principle measurement study was focused on a variety of hand tools used in dimensional measurement. The objective of the study was to observe participant behavior when acquiring measurements, and examine the importance of: Core measurement principles Implementing the right measurement strategy A questioning culture

Page 4

CMSC 2011 Measurement Study Report

The measurement study was conducted over two-day period in two separate areas using a range of defined first principle tasks. The criteria of the tasks were modified to allow for various training and assessment methods to be undertaken, such as questioning techniques, practical task monitoring and demonstration, and assessment of prior learning and experience. More than 200 sets of measurements were gathered, and many people participated in all four tasks. Please reference Appendix A for an overview of the data. Upon completion of the study, the following eight areas were identified as core knowledge and understanding vital to making informed decisions during the measurement process. 1. 2. 3. 4. Standards Traceabilty Calibration Measurement Uncertainty 5. 6. 7. 8. Understanding Design Requirements Repeatability and Reproducibility Questioning Attitudes and Behavior

2011 CMSC MEASUREMENT STUDY Inspired by the success of the 2010 measurement study, the CMS commissioned a follow-up study for CMSC 2011 based on coordinate metrology equipment. The research project would collect measurement results and focus on core knowledge, understanding, and behavioral requirements. The event was held over a 3-day period and conference delegates were encouraged to participate in the study entitled How Behavior Impacts Your Measurements. On days 1 and 2, attendees performed 3D coordinate measurements while being observed. On the third day, the participants were invited to the measurement workshop where the study group presented their findings.

Figure 1: Photorealistic Rendering of the 2011 Measurement Study Work Area.

Page 5

CMSC 2011 Measurement Study Report

2011 SCHEDULE OF MEASUREMENT WORKSHOP ACTIVITY Day 1 (Tuesday) On the first day, 3D measurement tasks were carried out with little or no instruction from the study group. Without guidance, participants determined the requirements to take measurements or instructed the operator as to which measurements were needed. The resultant data was stored for later analysis. Day 2 (Wednesday) On the second day, 3D measurement tasks were conducted with instructions or procedures, and the participant was required to take measurements or instruct the operator with a method to collect the needed measurements. The resultant data was stored for later analysis. Day 3 (Thursday) In the afternoon of the third day, a formal Workshop was held to present the observations and data from the measurement study to conference delegates. MEASUREMENT STUDY CATEGORIES For the measurement study, key characteristics were identified and observed during the data collection period. An operator and observer were available to monitor those key characteristics for every participant on every task in these categories: 1. 2. 3. 4. Pre-measurement Measurement planning Measurement Post-measurement

STUDY P ARTICIPANTS The experience level of each participant was recorded during the study. Their metrology skills fell into one of the following categories: A beginner to measurement A beginner to portable measurement Some experience in portable measurement Experienced in portable measurement

Page 6

CMSC 2011 Measurement Study Report

As stated, no formal procedure or planning was devised or available to the participants on the first day of the measurement study. The following table was used by the operator and observer to assist in the data acquisition process. TABLE 1: Observer/Operator Guidelines Beginner to Measurement Measurement Study Categories Pre-Measurement Measurement planning Measurement Post Measurement Beginner to Portable Coordinate Measurement Measurement Study Categories Pre-Measurement Measurement planning Measurement Post Measurement

Give the participant the titles above and ask their Observe if the participant asks the general questions opinion of each area, and its relevance to the task as per the check sheet they are performing. Guide the participant, but let them take the Guide the participant by explaining the task, but measurements. let them take measurements at their own speed using their own method. If they are probing, inform them of the number of points in/on the feature, but do not be specific. Provide help if needed! Some experience in Portable Coordinate Measurement Measurement Study Categories Pre-Measurement Measurement planning Measurement Post Measurement Ask participant if they are comfortable in taking measurements, but urge them to explain the software operative, and articulate what they want to achieve from the software. Let participants take measurements and notice if they ask about results. They may have asked the question at the pre-measurement stage. Experienced in Portable Coordinate Measurement Measurement Study Categories Pre-Measurement Measurement planning Measurement Post Measurement No guidance let the participant tell the observer what they want from the software and allow them to use the equipment. Do not interrupt unless there is a problem or they are not using a device properly. Check to see which categories above are mentioned by the participant.

Page 7

CMSC 2011 Measurement Study Report

On day two, formal procedures were introduced for each task encompassing the four categories in Table 1 on page 7. The operator and observer noted if participants questioned the procedures associated with the measurement study categories and demonstrate the relevant behavior linked to good practices. Were best practices and the right behavior instilled? Did the participant question and plan the measurement process around the task requirements? MEASUREMENT TASKS DEFINED In preparation for the study, three distinct measurement tasks were devised and incorporated coordinate metrology equipment as follows:

TASK 1 - An automobile door panel measured using a combination of a Laser Tracker, retro reflectors, and inspection software

TASK 2 - An engine compartment measured using a combination of an articulated arm, probe and inspection software

TASK 3 - A vehicle measured using a Laser Tracker, combined with probing and laser scanning systems, and inspection software

Page 8

CMSC 2011 Measurement Study Report

P ARTICIPANT PROFILES Each participant of the study was asked a series of questions about their industrial sector, job role, department, experience, frequency of taking measurements, if they participated in the 2010 measurement study, and if they use or operate metrology equipment. This information was used to generate a participant profile. Once the Q & A session was completed, the study group would monitor the participants measurement behavior and its application to each task performed. This demographic information will be presented in the following section. Profile data and observations have been assessed and separated into the graphical representations showing the actual number of participants. Industries Represented: Aerospace, Nuclear, Automotive and Other

Page 9

CMSC 2011 Measurement Study Report Job Roles in Industry: Engineer, Metrologist, Scientist, Management and Other

Page 10

CMSC 2011 Measurement Study Report Company Department: Design, Quality, Inspection, Manufacturing, Calibration, and Other

Page 11

CMSC 2011 Measurement Study Report Measurement Experience: 0 3 Years, 3 7 Years, More than 7 Years

Page 12

CMSC 2011 Measurement Study Report Measurement Frequency: Daily, Weekly, Occasionally or Never

Page 13

CMSC 2011 Measurement Study Report Participated in the CMSC 2010 Measurement Study: Yes or No

Page 14

CMSC 2011 Measurement Study Report Uses/Operates Metrology Equipment

Page 15

CMSC 2011 Measurement Study Report

OBSERVATIONS In the following section, behavioral and good practice observations were collated and categorized in the graphical representations. Each measurement task is identified and separated into day 1 and 2. It is important to note the values in the following graphs are percentages in relation to the observations listed in the graph. Each observation title was identified, questioned, or commented upon relative to the measurement category. For example, in the Pre-measurement category of the articulated arm, 6% of the participants asked about the temperature in the measurement area, 19% asked about the calibration status, 6% mentioned the product was located on a carpet. Day 1: Articulated Arm / Engine Compartment - Pre-measurement Category

Page 16

CMSC 2011 Measurement Study Report Measurement Planning

Page 17

CMSC 2011 Measurement Study Report Measurement Process

Page 18

CMSC 2011 Measurement Study Report Analysis

Page 19

CMSC 2011 Measurement Study Report Day 1: Laser Tracker / Door - Pre-measurement Category

Page 20

CMSC 2011 Measurement Study Report Laser Tracker / Door - Pre-measurement Category (Continued)

Page 21

CMSC 2011 Measurement Study Report Laser Tracker / Door - Pre-measurement Category (Continued)

Page 22

CMSC 2011 Measurement Study Report Analysis

Page 23

CMSC 2011 Measurement Study Report Analysis (Continued)

Page 24

CMSC 2011 Measurement Study Report Day 1: Laser Tracker / Vehicle - Pre-Measurement Category

Page 25

CMSC 2011 Measurement Study Report Day One: Laser Tracker / Vehicle Pre-Measurement Category (Continued)

Page 26

CMSC 2011 Measurement Study Report Day 2: Articulated Arm / Engine Compartment Pre-Measurement Category

Page 27

CMSC 2011 Measurement Study Report Day 2: Articulated Arm Measuring / Engine Compartment Pre-Measurement Category (Continued)

Page 28

CMSC 2011 Measurement Study Report Day 2: Articulated Arm Measuring / Engine Compartment Pre-Measurement Category (Continued)

Page 29

CMSC 2011 Measurement Study Report Day 2: Articulated Arm / Engine Compartment Pre-Measurement Category (Continued)

Page 30

CMSC 2011 Measurement Study Report Day 2: Laser Tracker / Automobile Door Pre-Measurement Category (Continued)

Page 31

CMSC 2011 Measurement Study Report Day 2: Laser Tracker / Automobile Door Pre-Measurement Category (Continued)

Page 32

CMSC 2011 Measurement Study Report Day 2: Laser Tracker / Vehicle Pre-Measurement Category

Page 33

CMSC 2011 Measurement Study Report Day 2: Laser Tracker / Automobile Door Pre-Measurement Category (Continued)

Page 34

CMSC 2011 Measurement Study Report

MEASUREMENT RESULTS During the two days of observations and data collection for all three tasks, the number of sets of measurements collected were as follows: Day 1 - 54 Day 2 94 The split of participants across each task was very similar, although it was noted that nearly double the amount of sets of measurements were taken on Day 2. The results for each task were obtained as follows: On Day 1, each participant was required to tell the equipment operator the measurement strategy for both the alignment and measured features of the part. Guidance would be given if requested. On Day 2, each participant was required to follow set procedures developed by the equipment operator to determine the alignment and the measured feature data.

Page 35

CMSC 2011 Measurement Study Report

GENERAL OBSERVATIONS OF THE RESULTANT DATA Task 1: Door The following graphs are a sample of data taken from numerous features measured. The measurement units are in inches. The variability in the results on Day 1 ranges from 1.3799 to 1.7007 inches, and Day 2 varies from 0.0051 to 0.0093 inches. The graphs identify outliers but consistency seemed to be more apparent from Day 1 to Day 2 when incorporating good practice, asking questions of participants, encouraging questions from the participants to the operator about the pre-measurement, measurement planning, measurement and post-measurement requirements. Day 1 Day 2

Page 36

CMSC 2011 Measurement Study Report Task 2 - Engine Compartment The following results show that the consistency in results from Day 1 to Day 2. On Day 1, it was observed that the measurement strategy chosen by the participants varied thus giving deviation in the results. Whereas on Day 2 with a specific procedure and measurement strategy being in place, the variation was greatly reduced. Additional equipment such as the use of adaptors were made available as an option on Day 1, but were required on Day 2. The following two graphs show the variability in the alignment routines between Day 1 and 2. Day 1 In this graph, both the Z and Y values are not zero as they should be, and the Y values are not balanced. The ranges of the values are very similar because on Day 1 almost all users used the best fit alignment method.

Page 37

CMSC 2011 Measurement Study Report Day 2 In this graph, you will notice the Z values for all 3 reference points show a zero deviation, the X values for points 1 and 3 also show a zero deviation, and the Y values for points 1 and 3 are mirrored and equally opposite as they should be. This proves a much more controlled and functional alignment, thereby making the subsequent measurements more reliable. This was due to a controlled alignment being implemented in the program.

Page 38

CMSC 2011 Measurement Study Report The following 2 graphs give a snap shot of the typical observation between the results on Day 1 and Day 2. Even though the number of reading vary, the range on Day 1 varies from 0.0223 to 0.1503 inches, and on Day 2 from 0.00589 to 0.0357 inches. Day 1

Page 39

CMSC 2011 Measurement Study Report Day 2

See Appendix B for further results

Page 40

CMSC 2011 Measurement Study Report

Task 3 - Vehicle The following graphs are a sample of data taken from numerous features measured. The measurement units are in millimeters and the variability in the results on Day 1 ranges from 5.36 to 8.198 millimeters, and Day 2 varies from 0.926 to 1.59 millimeters. The graphs identify outliers, but consistency seemed to be more apparent from Day 1 to Day 2 when incorporating good practices, asking questions of participants, encouraging questions from the participants to the operator about the pre-measurement, measurement planning, and post-measurement requirements. See Appendix B for further results. Day 1 Day 2

Page 41

CMSC 2011 Measurement Study Report

CONCLUSION After completion of the measurement study, the same key areas identified in the 2010 study were also acknowledged as important during the 2011 study irrespective of the equipment. The importance of the requirements for pre-measurement, measurement planning, obtaining the measurement, and the post measurement criteria was emphasized on both days. Knowledge and understanding of the following factors is essential for a sound measurement strategy incorporating best measurement practices and using the right behavior to reduce the variation and uncertainty about the results. Standards Traceabilty Calibration Measurement uncertainty Understanding design requirements Repeatability and reproducibility Analysis and output requirements Functionality Questioning Planning Attitudes and behavior In conclusion, whether the metrologist uses hand tools or the most sophisticated coordinate measuring system, questioning and planning all the requirements of the measurement help to reduce the possibility of generating poor measurements. The keys to informed measurement results are: 1) not taking things for granted, and 2) implementing behavioral questions and techniques for the measurement process.

Page 42

CMSC 2011 Measurement Study Report

SIX GUIDING PRINCIPLES FOR ACHIEVING GOOD MEASUREMENT RESULTS 1. Make the Right Measurement Define and understand the measurement to be made. Routinely repeated measurements should follow a pilot study. 2. Choose the Right Instruments Appropriate instrument should be in a good state of repair and calibrated. Use according to instructions of owner or manufacturer 3. The Right People Human error is major source of poor measurement quality. An operator needs to receive instructions and training. Where a group of operators is involved, their individual roles and responsibilities need to be understood. 4. Regular Review Follow a written schedule to check performance of instruments at regular intervals. 5. Demonstrable Consistency A measurement result is only valid if it can be reproduced consistently. Local factors need to be taken into account. Important or difficult measurements should be compared with the same measurements acquired by other laboratories/operators. 6. The Right Measurement Procedures Follow appropriate written measurement procedures, including health and safety guidelines. Review the procedures regularly.

Page 43

CMSC 2011 Measurement Study Report

APPENDIX A 2010 Measurement Study The 2010 The Coordinate Metrology Society hosted an open measurement workshop inviting conference delegates to participate in a first principle measurement study based on a variety of hand tools used in dimensional measurement. The objectives were to examine the importance of: o Core measurement principles o Instilling the right measurement strategy o Observing behavior when dealing with measurements o Instilling questioning The measurement studies were undertaken in a two day period in two separate areas using various defined first principle tasks. The criteria of the tasks were modified to allow for various training and assessment techniques to be undertaken such as: o Review of prior learning and experience o Questioning techniques o Practical task monitoring o Demonstration Participant Profile Diverse Industries o Aerospace o Nuclear o Automotive o Science o Woodworking Varied Job Roles o Quality o Scientist o Management o Measurement o Supplier Experience Measurement Experience o Ranging from newcomers to veterans with over of 15 years of experience

Page 44

CMSC 2011 Measurement Study Report

Observations During Measurement Study Checked jaws for parallelism Checked calibration status Used gauge block to check vernier Cleaned before using Made multiple measurements Made multiple re-zeros Checked vernier zero Cleaned jaws Misunderstanding of scale and units Check for damage Locked jaws using thumbscrew Measured with jaws not parallel to the workpiece Used internal jaws for depth measurement Used depth bar and checked zero Spent effort to find minimum value Used gauge block as a comparator Poor lighting didnt help reading scale Cleaned the artefact Checked the zero Felt for dirt on the surface table Used without zeroing difference between 2 values Base upwards / part downwards Cleaned surface plate Checked using gauge block Used direct comparison against gauge block

During the study, key characteristics were identified and observed during the data collection period. An operator and observer monitored those characteristics for every participant on every task in these categories: pre-measurement, measurement planning, measurement, and post-measurement.

Page 45

CMSC 2011 Measurement Study Report

Resulting Data Four measurement tasks were executed with and without instruction that covered diameters, length, depths and gauge block wringing. Task 1: Without procedures 55 readings ranged from 0,155 to 0,191mm Task 2: Without procedures 43 readings ranged from 0.0015 to 1.21 inches Task 3: With procedures 39 readings ranged from 0,009 to 0,057mm Task 4: Without procedures 28 readings varied by 0.0045 inches Key observations and behavior across a wide range of expertise:

Page 46

CMSC 2011 Measurement Study Report

Resulting Data (Continued)

Page 47

CMSC 2011 Measurement Study Report

APPENDIX B Appendix B shows further results for data obtained during the study across all 3 measurement tasks. Day 1 Door: Examples of data obtained on Day 1 that were not toleranced features.

Page 48

CMSC 2011 Measurement Study Report

Day 2 Door: Examples of data obtained on Day 2 that were not toleranced features.

Page 49

CMSC 2011 Measurement Study Report

Day 1 Door: Examples of data obtained on Day 1 that were toleranced features.

Page 50

CMSC 2011 Measurement Study Report

Day 1 Door: Examples of data obtained on Day 1 that were tolerance features.

Page 51

CMSC 2011 Measurement Study Report

Day 1 Door: Examples of data obtained on Day 1 that were out of specification.

Page 52

CMSC 2011 Measurement Study Report

Day 2 - Door: Examples of data obtained on Day 1 that were out of specification.

Page 53

CMSC 2011 Measurement Study Report

Day 1 Engine compartment: The capability index example for the engine compartment. The ideal CP is >1.

Page 54

CMSC 2011 Measurement Study Report

Day 2 Engine Compartment: The CP of 99.0 is due to there being no deviation of the data.

Page 55

CMSC 2011 Measurement Study Report

Day 1 Vehicle: Examples of data obtained on Day 1 that were tolerance features.

Day 2 Vehicle: Examples of data obtained on Day 2 that were toleranced features.

Page 56

CMSC 2011 Measurement Study Report

APPENDIX C Flyer Example

Inviting you to the CMSC 2011 Interactive Measurement Study How Behavior Impact Your Measurement After the success of the Gage R&R study in 2010 the measurement study returns for 2011... We invite you to booth 506 to participate in a study that will be coordinated by the National Physical Laboratory (UK), and assisted by members of the CMS Certification Committee. All attendees are encouraged to participate in this daily data collection activity, which will provide hands-on experience with large volume measurement systems. The purpose of the workshop is to explore the measurement strategies and behavior of coordinate metrologists. Each station in Booth 506 will allow the participant to take measurements using a Laser Tracker, Portable Arm and Optical System. You will have the option to either perform a series of prescribed measurements with assistance or alternatively instruct the appropriate operator on how to take the required measurements. We will ask you to share your level of experience and background in metrology. Data collected during the first two days (Tuesday and Wednesday) will be automatically stored and compiled. The anonymous information collected will then be examined for measurement variability. The Measurement study outcomes will be presented by the National Physical Laboratory on Thursday in the FLW Ballroom I-J at 2.30pm. The criteria of the measurement tasks will enable various training and assessments techniques to be analyzed, such as the evaluation of prior learning and experience, questioning techniques, and practical task monitoring and demonstration. Your cordial invitation to Booth 506 Tuesday & Wednesday

CMSC 2011 Measurement Study Report

Figure 2: Photorealistic rendering of the 2011 Measurement Study Work Area.

ACKNOWLEDGEMENTS The success of the measurement study would not have been possible without the support of the CMS Certification Committee, the booth sponsors and volunteers, the metrology equipment manufacturers, and lastly, all of the participants. A special thanks goes to the following individuals for their dedication to the study and contributions to this report: Bertrand Gili Metrologic Nathalie Blanco Metrologic Mark Denham Metrologic Dennis Martin Metrologic Ben Hughes National Physical Laboratory

Page 58