More myths and realities of psychometric testing Selecting the psychometric tool which best suits your organisation and purposes can be a difficult task. In this second article in a two-part series, Dr Paul Englert, Sarah Burke and Teresa MacGregor uncover more myths and realities around testing JUDGING THE QUALITY OF A PSYCHOMETRIC TEST, AND -poosng hobesttest for your oganisation from the grossing amber ‘of providers inthe market, is a tough cll, particularly if you don’t have specialist training in the area of psychometris. Inthe astissue of Enptoynet Taday, weoutline the frst seven of 1Smyths surrounding the assessment testing industy. In this issue we ake a look at the remaining eight myths and what you need 10 knowin ore to make the best purchasing decisions, Myth 6: People have a ‘work personaly Reality: A mainstream personality questonnaite Is usta valid as work personality’ profile, The conoept of personality transcends such astifical Boundaries as our work oF home life. The fesibility to adapt one’s behaviour in different situations or not i merely ‘one aspect of personality, and it isnot necessary to create a whole neve personality questionnaire to cover i. That is dane more for plausibility rather than reliably. Meta-analysis isthe standard in independent research activity and meta-analysis has found that 2 test ehich provides goo! measures ofthe big ive! personality traits ‘does predict performance egardles of the setting. Tis finding is ‘base onthe analysis of patemsin the published esearch ina whole field rather than on the research of one entity. Myth 9:18 doese't mater how a tools eonstructed Reality: The effectiveness ofa too! depends primarily om hove sell ‘hasbeen constructs. Factor analysis is generally regarded as one ‘oF the most bust statis processes for ensuring the rigour of a psychometric tol. In particular, allows us to find dimensions of| personality that are distinct from each other and to identily ques tionnaire items that do not overlap. A fundamental law with tests ‘which are not factor analysed is that they tend to be excessively long and claim to measure multiple, independent, dimensions of| personalty—sometimes as many’ a 200 traits. In reality, there is Tikly tobe a igh degree of overlap betveen the scales inside these assessment fools, We believe tests should be built aud broadly accepted psychometric theory. This means that thee isa strong fell ‘ance on the underlying factor structure that supports the to "Myth 10: Research materia should oly be given fo cursent test users, Reality: Research material shoukd be mae available to everyone to both further the worldwide knowledge base and to allow for informed consumer decisions. One ofthe strongest indicators of «| test publisher’ lack of commitment to psychology i a closed-door policy with mgards to accessing technical manuals and/or inde pendent research that supports ther tools If hiss made available at all it tends to be to cients who have aleady baught into thei rmarkoting hype. In this way, echnical information isnot avilable for people to make informed decisions, but rather i used to preach to the converted. Moreover, it contravenes the international call for open-access to research, You can read mare about this topic at hi ww soros ong openaccess [Myth 1: Ipsative tess are good for making selection decisions Reality: Ipsative tests have been ertcsed by psychometricians as beng inappropriatefor use in selection. Usinga forced choiceeption, they require a person to select from a range of tats the one thats most an least ke thernselves, Nonpsative tests equi a person to respond toa questionby indicating their proference on say, ive-of| three-point scale. Put simpy, ipsative tests allow the comparison of| relative strengths of personality tats within an individual but do otal comparison between peopl. The dilemma here, ofcourse, is that selection is all about making comparisons between people Ipsatve tests have numerous other problems including 1+ Thesetests cannot be normed and therefore comparisons agaist groups of people are nonsense; 4 The results cannot be factor analysed and therefore the personal ity dimensions being measured must be questioned 4+ They are subject to input response biases. As a result of these and other issues, the use of ipsative tools for selection has been Aiscouragey the British Psychological Society Instead psative tests are of more benoit in the area of individual counselling such as career guidance “Thesecone problemisthatipsative tests fore the spread of aperson's Scoresand while thismay make interpretation eaieritdoes nat make it any more valid tis a clasic case of people being misled by fase logic and simplistic marketing, Indeed a discusses above, its the lack of validity with respect to making comparative jdgement that isthe key problem with psative tools For those seriously interested In exploring the misuse of ipsative personality tols we strongly recommend an atcleby SF Blinkhor, etal ented “Spuriouser and ‘Spuriouser: The use of ipsative personality tess’ which appeared in ‘the Joural of Oscipational Psychology (61, 153162), ecenee2scanuaty200 employment today 97Myth 12: Psychometric tools should only be imerpreted by a psy chologist Reality: Psychometric oolscan beinterpretedby anyone ho hashad the relevant training, Psychometric tls are built to be interpreted ina standardised way. Thsis why computer-based reporting isnot only valid but may be less prone to error than human generated repos. A computer report will always provide a similar reading based on a person's sooes and is nt affects by any potential exors such as political bias or stereotyping. ‘Computer programmes ae not, however, sophisticated enough to make all the requized integrations between indivi scales bat ‘provide afar fist draft of a report, Human input is required in ‘order to make some ofthe more sublle sale interactions. The irony is that psychometric tools are easier to interpret than an employ- ment interview. Moreover, just because someone isa psychologist it does not mean that they have had the taining in psyehometric tools required to utilise them to their potential “Myth 18: If tests are objective anyone can interpret them and thene= fore taining is unnecessary Reality: You need tobe trained to make psychometric tools really tasefal, The rationale for training or ensuring certain skills is sum- mmarised as follows: 1 Ethical: Psychometric tools are used on real people to make ecisions that effect their ives. People who compte these tests provide a lot of information about themselves and they are en= tiled to recive feedback on theis results. Skills are needed to 4o this constructively 2 Standardisation: As discussed earlier, the usefulness of psy= chometrc tools is that they are aclminstered in a standardised ‘manner so that valid comparisons can be made. This requites| some education and training. Legal Slecton can boa itigious activity Its therfore vital that best practice protocol isrigorously followedin the adminstation andl interpretation of psychological tests as well asthe delivery ‘of candidate feedback 44 Uilty Training teaches people hove to maximise the usefulness ‘of test data, In particular, training teaches people how to relate the information frm each too to competency models selection decisions and people development. 5. Psychological and HR guidelines: Given points I+, it under standable why the New Zealand Psychological Society and the Human Resources Institute of New Zealand stipulate that tests should only be made available to trained users While training is of value it should not be training simply for tesining’s sake, nor mechanism for extracting excessive revenue It is a matter of providing sulficient training to us the tols provided properly, without overkill Myth I The size of a norm group is often promoted as the most important norm evteria. 38 employment today oxceunesxesanuany 207 Reality: The rlevance and distribution is often the most important norm criteria. When a person is compared to a norm group, the size ofthat group has little relevance to the comparisons that can ‘be made, Instead, i isthe relevance of a norm group to Your own onganisation and the distribution (or spread of scores inside that norm that sof critical importance. Ifa norm group isnot rflctive ofthe people you ave testing and dows not differentiate people well, there is no benefit in knowing that atest publisher has collected rrr geoups of several thousand people. Ifa norm group is well built and is found to be reflective ofthe population of interest, a norm grouping, of 200 people may be suficient! (One of the key issues with norm grouping is that we must be comparing like with like. Thus, the people inside a norm group must have sat the fst in the same conditions and come fom like ethnic and educational backgrounds, Only ifthe conditions of in- termet testing are sufficiently controlled do they achieve suficent standardisation to allow the building of robust norm categories. Many internet tools do not, Beyond these criteria, atest publisher should also be abe to confirm the gender mix of patiipants and what roles people were tested for. Allo these factors wll help define the suitability of one paticulae comparison group over another. Asa final note, the most relevant ‘orn comparison group will often be a representative cros section of inchouse staff who perform each ofthe role(s) under review: “Myth 15: There needs to be an additional chaege fo reporting. Reality: You need only be charged once for testing. Test produc- ers ave looked at various means of extracting additonal money {om client organisations, This incldes everyting from having an ‘organisation pay each time a candidate sits atest to everytime @ report is generated or both. In reality, once test data is input into scoring system, no additional time commitment is required for a ‘report tobe automaticaly generated. If clients do not wish to incur {he often sigeticant reporting charges, they will opt to rely solely on verbal feedback toa candidate or recruiting manager using ust profile chart) or they may prepare short-form, less comprehensive reports by hal for distsbution In conclusion: Clearly not all tests are equal, Those who use assessments as part of their selection process must therefore be educated on the myths and reality of testing to be discerning pur- chasers. While far from a panacea for selection dificities, if used cormectly psychometric testing can be a powerful addition to any selection process, providing valuable data on the high and low performers. Too often however testing is sold by a sleight of hand ‘and on faulty’ logie. Only by being aware of the myths behind the testing industry can consumers differentiate testing providers and identity those providers that moet their needs a Paul Englert and Sarah Burke ae arecters and Trsa MacGregor Is the Southland manager fr OpraConsuting Group (wu opr.o 2).