Skip Page Banner  
Skip Navigation

Everything You Think You Know About Lawyer Recruiting Is Wrong: The New Science of Evidence-Based Hiring Practices

Monday, October 10, 2011

Contributed by Bill Henderson, Caren Ulrich Stacy and Steve Gibson, Lawyer Metrics, LLC

For a typical senior partner in today’s law firm, the changes that have occurred since law school are substantial. From the commonplace miracle of global 24/7 connectivity to the emergence of offshore e-discovery―the world has evolved with the legal industry in tow.

Yet, the lawyer hiring process would be as familiar to today’s partner as it was to the partner who first interviewed him or her 25 years ago. Anchored by the one-on-one interview and an unwavering bias towards a cookie cutter educational pedigree, law firms make million dollar hiring decisions using little or no analytics. This “we know it when we see it” approach has contributed to astronomical attrition rates in most law firms with 50 percent of new lawyers departing after three to four years with their firm.1 Even worse, the convergence of so many firms vying for the same top candidates produced a salary scale―typically $160,000+ in major markets―that caused many clients to question the value of having junior associates work on their matters.

Although a relatively simple hiring model has enabled many law firms to become national or global powerhouses and enriched their partners in the process, the competitive landscape is changing dramatically. Until recently, there was a shortage of sophisticated business lawyers relative to demand. For several decades, firms with a solid base of corporate clients could prosper by adding lawyers as the U.S. economy boomed and clients grappled with more regulation and legal complexity. Between 1979 and 2008, the average size of the 250 largest law firms grew over 500 percent (from 102 attorneys to 535).2 The average number of offices also increased from 2.5 offices per firm to 10.2, with a sizeable proportion now in foreign locations.

With so much sophisticated capacity available at so many brand-named law firms, the general counsel of major corporations have a surplus of options for their legal work. As a result, law firms are now in the unfamiliar position of competing over market share. One way to win market share―and possibly the only way―is to hire and develop better talent.

Going forward, law firms must employ more sophisticated and reliable hiring and development tools commensurate with the challenge of making million dollar “talent” decisions. Unfortunately, the current most popular hiring tool―the one-on-one interview―ranks only slightly above a coin-toss as research reveals (see inset chart). Simply hiring someone because of a shared favorite sports team works just as well. With the profit difference between a mid-level A-player associate and their C-player counterpart ranging from $100,000 to $250,000 per associate each year, firms are putting themselves in a vulnerable position by persisting with the one-on-one interview as their primary selection tool.

Validity of Selection Methods

PredictorsValidityHit Rate
Perfect Prediction Scheme1.00100%
Cognitive Ability Tests.5377%
Job Tryouts.4472%
Biographical Inventories.3769%
Structured Board Interviews (Consensus Scored).3568%
Reference Checks.2663%
Grade Point Averages.2161%
Job-Related Experience.1859%
One-on-One Interviews.1457%
Ratings of Training.1357%
Years of Education.1055%
Interests (Preferences).1055%
Random Selection Scheme.0050%

Sources: Hunter and Schmidt, “Quantifying the Effects of Psychological Interventions on Employee Job Performance and Work-Force Productivity,” Vol. 38, American Psychologist, 473, 474 (1982); Wiesner and Crowshaw, “A Meta-Analytic Investigation of the Impact of Interview Format and Degree of Structure on the Validity of the Employment Interview,” Journal of Occupational Psychology, Vol. 61, 275, 282 table 1 (1988).

By the same token, having interview data absent a rigorous analysis regarding what it takes to be successful as a lawyer would position the firm―as the cliché goes―to know more and more about less and less until they know everything about nothing. Only when matched to the specific culture, practices, and people of a firm can the data become evidence. And, through careful analysis over time, the resulting evidence becomes knowledge and eventually a competitive advantage. At some firms, the next stage of evolution has begun, as they tap what was essentially an unknown wealth of information about managing their talent―the single biggest thing a law firm can do to improve its future―using a variety of evidence-based practices.

THE FOUNDATION―DEFINING SUCCESS

The basis for evidence-based hiring practices begins with the firm. “Success” is firm-specific. Through an Organizational Success Study (OSS), it’s possible to effectively reverse-engineer what makes for a successful associate and partner at a given firm. The notion is simple: figure out who is more likely to be successful now and in the future, and then go find more of them. The execution, of course, is more complex. But the alternative is the stark and increasingly costly reality of hiring A-recruits who become C-players, and too many A-players who leave the firm too soon.

An OSS goes to the heart of a firm-specific approach to hiring by revealing the traits and attributes associated with success, and then codifies a set of associated behavior-based competencies. This three-part scientific process proceeds with analytic rigor using a simple ranking of several dozen attributes―teamwork, initiative, commitment, etc.―by the firm’s lawyers. Then, through a methodical discussion with high performers at the firm (know technically as behavioral event interviews), a firm-specific list of behaviors related to each attribute is generated.

As a third step, once the competencies and behaviors are identified and agreed upon by the firm, a variety of workplace personality assessment tools can be brought to bear to further explore and validate the behaviors that actually lend to high performance at the firm. These assessment tools are given to current, high-performing partners and associates to create a detailed, data-driven profile of the varying types of successful lawyers (i.e. service lawyers, managers, technicians, mentors, rainmakers, etc.).

EVIDENCE-BASED SELECTION TOOLS

Armed with the knowledge of the particular success traits or competencies that exemplify a high performing lawyer, the firm has the ability to employ evidence-based and data-driven tools for lawyer selection. There are three primary methodologies that are being introduced in law firms with great success―Biographical Inventories, Structured Panel Interviews and Workplace Personality Assessments.

— BIOGRAPHICAL INVENTORIES

From Wall Street to Major League Baseball, the use of biographical inventories is commonplace outside of the tradition-bound realm of law. As seen in the recent big screen adaptation of Michael Lewis’ book, “Moneyball,” biographical inventories and sophisticated quantitative analysis transformed the scouting and recruiting practices of baseball by uncovering what really mattered for winning.

Applied to legal recruiting, our studies to date have shown increases of 10-33 percent in the ability to predict A-players. Even more valuable are the increases of 50-60 percent in the firm’s ability to predict―and thus avoid hiring―C-players. The analysis is based upon a rigorous, firm-specific analysis of high/low performance associated with the traits found on candidates’ résumés and transcripts to determine which of the traditional (i.e. grades, law review) and non-traditional (i.e. blue collar work, team sports, military service) hiring criteria are predictors of success.

The data is hidden in plain sight on candidates’ résumés and transcripts. With analytic insight into what’s important―is prior work experience a strong positive predictor of success at the firm, or a strong negative?―a firm can pre-populate its interview schedule with candidates who have a higher likelihood of success.

— STRUCTURED PANEL INTERVIEWS

The traditional one-on-one interview is good at judging one thing: likability. Likeability matters, to be sure, and the recruiting role of the one-on-one interview remains vital for a business that requires a cultural fit. After all, the supply of top candidates is always limited and the choices that these topmost candidates have as far as where to work are real regardless of the economic climate. But when it comes to selection―versus recruiting―a coin-toss is not the right way to allocate valuable and scarce partner time.

The best way to reduce bias and increase accuracy of the interview process is to structure it as a panel interview. In a Structured Panel Interview (SPI), also known as Structured Board Interview, all of the candidates are asked the same behavioral-based questions that are mapped onto the firm’s most essential success factors. The SPI questions are focused on past behaviors as the best indicators of future performance―not hypothetical or situational questions, which only reveal what the interviewer wants to hear. All of the panel interviewers (typically four) simultaneously see and hear the candidates’ responses, thus enabling them to score the candidate on what matters using a reliable, evidence-based process.

Again, the data are clear: a consistent and effective use of panel or board interviews is three times more effective than their one-on-one alternative.

— WORKPLACE PERSONALITY ASSESSMENTS

The same assessment tools used to explore the firm’s competencies are also useful in selection. The body of knowledge around personality traits effecting workplace success is well developed and validated, and such assessments are a familiar part of the hiring process in Fortune 500 companies as well as other professions.

These tools are fairly new to law firms. However, data are beginning to amass in two key areas. First, there are now data regarding how successful lawyers score on these tests, including a core set of attributes crossing all firms―business awareness, client focus, innovation, problem solving, decision making, etc. And second, a data set of equally important firm-specific attributes such as teamwork, adaptability, developing others, and building relationships. One advantage of the workplace personality assessments is the ability to quickly and cost-effectively administer these tests such that the results can be compiled and available even for a compressed on-campus interviewing timeline.

It’s not just that the tests work―several decades of use have proven that―it’s that there is simply no other effective way to screen for a wide variety of pay-to-play attributes that every successful associate must have. When paired with Structured Panel Interviews, a firm can dramatically reduce their incidences of C-player hires.

DEVELOPMENT & RETENTION IMPLICATIONS

Less analytically complex, but no less data-driven, firms armed with the full breadth and depth of best-practice tools available for recruiting and selection have an asset―a fully-rendered view of who they are hiring and why. This investment has returns that extend well beyond a new associate’s start date.

By knowing, in advance, what an associate’s strengths and weaknesses are, and which of those weaknesses are developable or intrinsic parts of his/her personality, the firm can craft a much more effective professional development plan. Even more powerful, the developmental goals are based upon the firm’s own unique key competencies as revealed in the Organizational Success Study, and not a one-size-fits-all prescription from the consultant flavor of the week.

From the associate’s perspective, the process is also much more engaging. Unique to them, it has an even greater resonance for today’s generation of young professionals, raised on a steady chorus of adults singing their praises. Engagement, in turn, has been shown to be the major determinant of retention.

SURVIVING CREATIVE DESTRUCTION

As the business of law continues to evolve, the practices of the law firm must change as well. Followers of the history of the business of law know that today’s law firm itself is the outcome of previous evolutionary jumps in law firm structure and composition. Ironic as it may be, the rise of data-driven and technology-enabled hiring practices reaffirms, and even strengthens, the role of the lawyer, not technology, in the practice of law.

Bill Henderson is Principal of Lawyer Metrics LLC. Bill is a Professor of Law at Indiana University Maurer School of Law, where he specializes in the empirical analysis of law firms and legal education. He is recognized as one of the foremost experts on the market trends affecting the business of law. To study law firm management and private practice, Bill founded the Law Firms Working Group, co-sponsored by the American Bar Foundation, which draws members from business, economics, law, sociology, political science and psychology. Because of his unwavering commitment to innovation in the legal field, the ABA Journal named Bill a “Legal Rebel.”  

Caren Ulrich Stacy is President of Lawyer Metrics LLC. Caren has 20 years of experience in talent management with law firms across the country, including Arnold & Porter, Cooley, Weil Gotshal and McGuireWoods. As a noted expert in the field of lawyer selection and development, Caren was awarded the National Association of Legal Professionals (NALP) Mark of Distinction in 2009. She was also elected as a Fellow of the College of Law Practice Management in 2010―an honor awarded to less than 200 individuals in the country. Caren is the author of several highly-regarded talent management books and an adjunct professor for the Denver University Sturm College of Law.  

Steve Gibson is Director of Lawyer Metrics LLC. After graduating from Princeton University with a degree in economics, Steve begun his career as a research assistant for Alan Greenspan and as an Associate Economist at Bear, Stearns. He has over two decades of experience in the design, use and communication of quantitative measures. Most recently, he was the Chief Operating Officer for the National Institute for Trial Advocacy, the nation’s oldest and largest provider of experiential education for lawyers. Steve has been quoted in national publications including The Wall Street Journal and Business Week, and was the editor of NITA’s whitepaper, “The Future of Legal Education.”  

Disclaimer  

This document and any discussions set forth herein are for informational purposes only, and should not be construed as legal advice, which has to be addressed to particular facts and circumstances involved in any given situation. Review or use of the document and any discussions does not create an attorney-client relationship with the author or publisher. To the extent that this document may contain suggested provisions, they will require modification to suit a particular transaction, jurisdiction or situation. Please consult with an attorney with the appropriate level of experience if you have any questions. Any tax information contained in the document or discussions is not intended to be used, and cannot be used, for purposes of avoiding penalties imposed under the United States Internal Revenue Code. Any opinions expressed are those of the author. The Bureau of National Affairs, Inc. and its affiliated entities do not take responsibility for the content in this document or discussions and do not make any representation or warranty as to their completeness or accuracy.  

©2014 The Bureau of National Affairs, Inc. All rights reserved. Bloomberg Law Reports ® is a registered trademark and service mark of The Bureau of National Affairs, Inc.  

To view additional stories from Bloomberg Law® request a demo now