Once again here on Hacker News we are talking about hiring procedures for technical companies. Many people find this topic interesting, because most of us have applied for a job at least once, and many of us have been in a position to recommend someone else for a job, or to hire someone for a job. From participants in earlier discussions I have learned about many useful references on the subject, which I have gathered here in a FAQ file. The review article by Frank L. Schmidt and John E. Hunter, "The Validity and Utility of Selection Models in Personnel Psychology: Practical and Theoretical Implications of 85 Years of Research Findings," Psychological Bulletin, Vol. 124, No. 2, 262-274
sums up, current to 1998, a meta-analysis of much of the HUGE peer-reviewed professional literature on the industrial and organizational psychology devoted to business hiring procedures. There are many kinds of hiring criteria, such as in-person interviews, telephone interviews, resume reviews for job experience, checks for academic credentials, personality tests, and so on. There is much published study research on how job applicants perform after they are hired in a wide variety of occupations.
EXECUTIVE SUMMARY: If you are hiring for any kind of job in the United States, prefer a work-sample test as your hiring procedure. If you are hiring in most other parts of the world, use a work-sample test in combination with a general mental ability test.
The overall summary of the industrial psychology research in reliable secondary sources is that two kinds of job screening procedures work reasonably well. One is a general mental ability (GMA) test (an IQ-like test, such as the Wonderlic personnel screening test). Another is a work-sample test, where the applicant does an actual task or group of tasks like what the applicant will do on the job if hired. (But the calculated validity of each of the two best kinds of procedures, standing alone is only 0.54 for work sample tests and 0.51 for general mental ability tests.) Each of these kinds of tests has about the same validity in screening applicants for jobs, with the general mental ability test better predicting success for applicants who will be trained into a new job. Neither is perfect (both miss some good performers on the job, and select some bad performers on the job), but both are better than any other single-factor hiring procedure that has been tested in rigorous research, across a wide variety of occupations. So if you are hiring for your company, it's a good idea to think about how to build a work-sample test into all of your hiring processes.
Because of a Supreme Court decision in the United States (the decision does not apply in other countries, which have different statutes about employment), it is legally risky to give job applicants general mental ability tests such as a straight-up IQ test (as was commonplace in my parents' generation) as a routine part of hiring procedures. The Griggs v. Duke Power, 401 U.S. 424 (1971) case
interpreted a federal statute about employment discrimination and held that a general intelligence test used in hiring that could have a "disparate impact" on applicants of some protected classes must "bear a demonstrable relationship to successful performance of the jobs for which it was used." In other words, a company that wants to use a test like the Wonderlic, or like the SAT, or like the current WAIS or Stanford-Binet IQ tests, in a hiring procedure had best conduct a specific validation study of the test related to performance on the job in question. Some companies do the validation study, and use IQ-like tests in hiring. Other companies use IQ-like tests in hiring and hope that no one sues (which is not what I would advise any company). Note that a brain-teaser-type test used in a hiring procedure could be challenged as illegal if it can be shown to have disparate impact on some job applicants. A company defending a brain-teaser test for hiring would have to defend it by showing it is supported by a validation study demonstrating that the test is related to successful performance on the job. Such validation studies can be quite expensive. (Companies outside the United States are regulated by different laws. One other big difference between the United States and other countries is the relative ease with which workers may be fired in the United States, allowing companies to correct hiring mistakes by terminating the employment of the workers they hired mistakenly. The more legal protections a worker has from being fired, the more reluctant companies will be about hiring in the first place.)
The social background to the legal environment in the United States is explained in many books about hiring procedures
Previous discussion on HN pointed out that the Schmidt & Hunter (1998) article showed that multi-factor procedures work better than single-factor procedures, a summary of that article we can find in the current professional literature, for example "Reasons for being selective when choosing personnel selection procedures" (2010) by Cornelius J. König, Ute-Christine Klehe, Matthias Berchtold, and Martin Kleinmann:
"Choosing personnel selection procedures could be so simple: Grab your copy of Schmidt and Hunter (1998) and read their Table 1 (again). This should remind you to use a general mental ability (GMA) test in combination with an integrity test, a structured interview, a work sample test, and/or a conscientiousness measure."
But the 2010 article notes, looking at actual practice of companies around the world, "However, this idea does not seem to capture what is actually happening in organizations, as practitioners worldwide often use procedures with low predictive validity and regularly ignore procedures that are more valid (e.g., Di Milia, 2004; Lievens & De Paepe, 2004; Ryan, McFarland, Baron, & Page, 1999; Scholarios & Lockyer, 1999; Schuler, Hell, Trapmann, Schaar, & Boramir, 2007; Taylor, Keelty, & McDonnell, 2002). For example, the highly valid work sample tests are hardly used in the US, and the potentially rather useless procedure of graphology (Dean, 1992; Neter & Ben-Shakhar, 1989) is applied somewhere between occasionally and often in France (Ryan et al., 1999). In Germany, the use of GMA tests is reported to be low and to be decreasing (i.e., only 30% of the companies surveyed by Schuler et al., 2007, now use them)."
Integrity tests have limited validity standing alone, but appear to have significant incremental validity when added to a general mental ability test or work-sample test.
Bottom line: if someone is hiring for a company that produces technical documentation, a company like iFixit.com, and one feature of the product is grammatically correct writing, it's a reasonable subpart of a work-sample test to include testing for revising English prose. If someone is hiring for managing a jewelry store (a local example I know) or for building wood-frame houses, it's quite possible that a work-sample test would completely disregard the issue of correct spelling and grammar. I know a very successful owner of a jewelry store (I know him as a fellow soccer dad who once coached one of my children) who has quite dodgy spelling and grammar and punctuation, but who can communicate in written English for emailing people. I'm aware of multiple local carpenters and other people in construction businesses, including managing construction businesses, who have varying degrees of punctilious correctness in English writing, but all of them making their reputations and their livings by how they construct buildings, not by how they construct sentences. If writing is part of the work (even just for exchanging ideas with colleagues in memos or emails), sure, test it. If writing is not particularly part of the work, don't worry about it.
Great post. Thought this was interesting from the Schmidt & Hunter article:
"This meta-analysis found that the validity of GMA for predicting job performance was .58 for professional managerial jobs, .56 for high level complex technical
jobs, .51 for medium complexity jobs, .40 for semi-skilled jobs, and .23 for completely unskilled jobs."
Apparently having a high IQ is a better predictor of being a good business guy than a good hacker.
Take any "meta-analysis" with a large grain of salt (or better yet, and industrial size vat of salt).
Meta-analysis suggests that men under the age of 26 have a 60% chance of being sex offenders. This same type of meta-analysis is also the reason that all sex crime offenders (including drunk people pissing in a park and amorous couples getting it on in a dark parkin glot) are lumped together for sex offender registration purposes, because the meta-analysis suggests that the recidivism rate for "sex offenders" (regardless of actual offense) is greater than 90% (without regard to the actual recidivist offense).
TLDR: Meta-analysis can be used to support any claim.
Would you care to elaborate on what, specifically, is flawed about meta analysis? I'd definitely be interested in a source on that 60% number and how that source arrived at a number that seems, on it's face, so incorrect.
"I'd definitely be interested in a source on that 60% number and how that source arrived at a number that seems, on it's face, so incorrect."
I'd be curious to see if anyone has actually done a study on this, but to me this number actually seems quite low. I would guess that it would easily be 90+%.
META: Karl that's a tremendously massive comment. Wonder if it might be more useful as an online FAQ that you could link to? I know I lost interest about 3 sentences in, but it looks like some great material in there.
I agree on the length. Ironic how, in an article on the importance of grammar, and presumably communication, the parent while most likely using perfect grammar, fails to convince [1] simply because of the amount of the material and the way it is presented.
I've done consulting for busy people who don't want to spend time reading a long email. They want to know the bottom line, however I also find it good practice to provide backup information in case they or someone they forward the email to want to know how I came to the conclusions.
I start the email with a summary and a list of actions they can take or questions that I have. Typically no longer than a few sentences. Below that is a demarcation point to all the backup, links and further details (should they want to forward or in all honesty to CYA as well for the conclusions).
Edit: [1] Because if people don't read they won't be convinced and they won't learn anything. One of the reasons I typically avoid books by academics: to much difficult verbosity.
> If writing is not particularly part of the work, don't worry about it.
I suspect the author is (perhaps unknowingly) using his grammar test as a proxy for general IQ. General IQ correlates _very_ well with performance across a broad spectrum of tasks. (The notion that there are different IQs for different areas of life, though sentimentally appealing, doesn't correspond to reality.) It's no surprise, then, that someone with a high IQ (as measured by this proxy) would do well in a position that requires solving other complex problems, even if these problems have nothing to do with writing per se.
I was wondering about this as well. Is this a correlation number, and should I translate it as "the best tests give no better than a coin flip of selecting a good candidate"?
I know the NFL gives the Wonderlic to potential draft choices. I guess they must have done some tests showing Wonderlic is useful to get around disparate impact.
Oddly, Wonderlic scores have never been proven to have any positive link with future on-field performance. In fact, one study[1] actually found a negative correlation at some positions.
But as with many silly personnel evaluation techniques that are in common use, general managers will never get punished for doing what everyone else is doing.
http://mavweb.mnsu.edu/howard/Schmidt%20and%20Hunter%201998%...
sums up, current to 1998, a meta-analysis of much of the HUGE peer-reviewed professional literature on the industrial and organizational psychology devoted to business hiring procedures. There are many kinds of hiring criteria, such as in-person interviews, telephone interviews, resume reviews for job experience, checks for academic credentials, personality tests, and so on. There is much published study research on how job applicants perform after they are hired in a wide variety of occupations.
http://www.siop.org/workplace/employment%20testing/testtypes...
EXECUTIVE SUMMARY: If you are hiring for any kind of job in the United States, prefer a work-sample test as your hiring procedure. If you are hiring in most other parts of the world, use a work-sample test in combination with a general mental ability test.
The overall summary of the industrial psychology research in reliable secondary sources is that two kinds of job screening procedures work reasonably well. One is a general mental ability (GMA) test (an IQ-like test, such as the Wonderlic personnel screening test). Another is a work-sample test, where the applicant does an actual task or group of tasks like what the applicant will do on the job if hired. (But the calculated validity of each of the two best kinds of procedures, standing alone is only 0.54 for work sample tests and 0.51 for general mental ability tests.) Each of these kinds of tests has about the same validity in screening applicants for jobs, with the general mental ability test better predicting success for applicants who will be trained into a new job. Neither is perfect (both miss some good performers on the job, and select some bad performers on the job), but both are better than any other single-factor hiring procedure that has been tested in rigorous research, across a wide variety of occupations. So if you are hiring for your company, it's a good idea to think about how to build a work-sample test into all of your hiring processes.
Because of a Supreme Court decision in the United States (the decision does not apply in other countries, which have different statutes about employment), it is legally risky to give job applicants general mental ability tests such as a straight-up IQ test (as was commonplace in my parents' generation) as a routine part of hiring procedures. The Griggs v. Duke Power, 401 U.S. 424 (1971) case
http://scholar.google.com/scholar_case?case=8655598674229196...
interpreted a federal statute about employment discrimination and held that a general intelligence test used in hiring that could have a "disparate impact" on applicants of some protected classes must "bear a demonstrable relationship to successful performance of the jobs for which it was used." In other words, a company that wants to use a test like the Wonderlic, or like the SAT, or like the current WAIS or Stanford-Binet IQ tests, in a hiring procedure had best conduct a specific validation study of the test related to performance on the job in question. Some companies do the validation study, and use IQ-like tests in hiring. Other companies use IQ-like tests in hiring and hope that no one sues (which is not what I would advise any company). Note that a brain-teaser-type test used in a hiring procedure could be challenged as illegal if it can be shown to have disparate impact on some job applicants. A company defending a brain-teaser test for hiring would have to defend it by showing it is supported by a validation study demonstrating that the test is related to successful performance on the job. Such validation studies can be quite expensive. (Companies outside the United States are regulated by different laws. One other big difference between the United States and other countries is the relative ease with which workers may be fired in the United States, allowing companies to correct hiring mistakes by terminating the employment of the workers they hired mistakenly. The more legal protections a worker has from being fired, the more reluctant companies will be about hiring in the first place.)
The social background to the legal environment in the United States is explained in many books about hiring procedures
http://books.google.com/books?hl=en&lr=&id=SRv-GZkw6...
http://books.google.com/books?hl=en&lr=&id=SRv-GZkw6...
Some of the social background appears to be changing in the most recent few decades, with the prospect for further changes.
http://intl-pss.sagepub.com/content/17/10/913.full
http://www.economics.harvard.edu/faculty/fryer/files/Fryer_R...
http://books.google.com/books?hl=en&lr=&id=frfUB3GWl...
Previous discussion on HN pointed out that the Schmidt & Hunter (1998) article showed that multi-factor procedures work better than single-factor procedures, a summary of that article we can find in the current professional literature, for example "Reasons for being selective when choosing personnel selection procedures" (2010) by Cornelius J. König, Ute-Christine Klehe, Matthias Berchtold, and Martin Kleinmann:
"Choosing personnel selection procedures could be so simple: Grab your copy of Schmidt and Hunter (1998) and read their Table 1 (again). This should remind you to use a general mental ability (GMA) test in combination with an integrity test, a structured interview, a work sample test, and/or a conscientiousness measure."
http://geb.uni-giessen.de/geb/volltexte/2012/8532/pdf/prepri...
But the 2010 article notes, looking at actual practice of companies around the world, "However, this idea does not seem to capture what is actually happening in organizations, as practitioners worldwide often use procedures with low predictive validity and regularly ignore procedures that are more valid (e.g., Di Milia, 2004; Lievens & De Paepe, 2004; Ryan, McFarland, Baron, & Page, 1999; Scholarios & Lockyer, 1999; Schuler, Hell, Trapmann, Schaar, & Boramir, 2007; Taylor, Keelty, & McDonnell, 2002). For example, the highly valid work sample tests are hardly used in the US, and the potentially rather useless procedure of graphology (Dean, 1992; Neter & Ben-Shakhar, 1989) is applied somewhere between occasionally and often in France (Ryan et al., 1999). In Germany, the use of GMA tests is reported to be low and to be decreasing (i.e., only 30% of the companies surveyed by Schuler et al., 2007, now use them)."
Integrity tests have limited validity standing alone, but appear to have significant incremental validity when added to a general mental ability test or work-sample test.
http://en.wikipedia.org/wiki/Employment_integrity_testing
http://apps.opm.gov/ADT/Content.aspx?page=3-06&JScript=1
http://www.princeton.edu/~ota/disk2/1990/9042/9042.PDF
http://www.hotelschool.cornell.edu/research/chr/pubs/reports...
Bottom line: if someone is hiring for a company that produces technical documentation, a company like iFixit.com, and one feature of the product is grammatically correct writing, it's a reasonable subpart of a work-sample test to include testing for revising English prose. If someone is hiring for managing a jewelry store (a local example I know) or for building wood-frame houses, it's quite possible that a work-sample test would completely disregard the issue of correct spelling and grammar. I know a very successful owner of a jewelry store (I know him as a fellow soccer dad who once coached one of my children) who has quite dodgy spelling and grammar and punctuation, but who can communicate in written English for emailing people. I'm aware of multiple local carpenters and other people in construction businesses, including managing construction businesses, who have varying degrees of punctilious correctness in English writing, but all of them making their reputations and their livings by how they construct buildings, not by how they construct sentences. If writing is part of the work (even just for exchanging ideas with colleagues in memos or emails), sure, test it. If writing is not particularly part of the work, don't worry about it.