The Use of Predictive Coding in E-Discovery Document Review - Demonstrated Cost Savings and Increased Efficiency - But Is It Right For Your Company?

Bloomberg Law®, an integrated legal research and business intelligence solution, combines trusted news and analysis with cutting-edge technology to provide legal professionals tools to be...

Contributed by Kevin F. Brady, Connolly Bove Lodge

When it comes to electronic discovery, there are three new “uses” that are keeping in-house counsel up at night — the use of social media, cloud computing and advanced technology-assisted document review. The use of social media by employees has not only increased the volume of information that might be subject to discovery, it has also increased the risk of intentional or unintentional disclosure of the company’s confidential, proprietary, trade secret or privileged information. Cost↑ Risk↑ While cloud computing has the potential to provide significant costs savings to the company by outsourcing hardware and software requirements, there is also the possibility of significant increase in risk of the intentional or unintentional disclosure of the company’s information because it is in the cloud and beyond the direct control of the company. Cost↓ Risk↑ However, the use of advanced technology-assisted document review software could reduce significantly the costs associated with document review, and at the same time significantly increase the efficiency and effectiveness of the document review process. Cost↓ Risk↓ The question is whether such advanced technology-assisted document review is right for your company.


Any in-house counsel can tell you that the problems with traditional document review are longstanding and obvious — lawyers spend tremendous amounts of time reviewing and re-reviewing information to get to the relevant documents and the privileged documents. It is time-consuming, very expensive and it still has a significant amount of risk that privileged or confidential, proprietary or trade secret information might be inadvertently produced. There is no method or grouping with respect to the documents in a particular batch and there is inconsistency between reviewers looking at the same information so multiple passes do not necessarily improve the results. At the heart of the review is the search for relevant and privileged documents. Of particular concern to in-house counsel and by necessity any outside counsel, is the preservation of any attorney-client privilege or work product immunity. Counsel must take great care to make sure that there is no inadvertent production of any privileged information because the penalty can be quite severe.


Two major decisions by Chief Magistrate Judge Paul Grimm of the U.S. District Court for the District of Maryland were instrumental in moving the needle forward on the issue of privilege review of electronic information and waiver. In the 2005 decision, Hopson v. The Mayor and City Council of Baltimore, Judge Grimm held that protective orders entered into by the parties could provide protection from the waiver of privilege for documents that were inadvertently disclosed. However, Judge Grimm also noted that these agreements were not self-selecting in that they would not obviate the need for parties to undertake “reasonable pre-production efforts” to avoid inadvertent disclosure in the first instance, when doing so would not be unduly burdensome or expensive.1

In the May 2008 decision, Victor Stanley, Inc. v. Creative Pipe, Inc., Judge Grimm discussed the standard of reasonableness that must be adhered to in order to avoid a waiver situation. Judge Grimm, in evaluating the “reasonableness” of the producing party’s actions to avoid waiver, looked at five balancing factors: (1) the reasonableness of the precautions taken to prevent inadvertent disclosure; (2) the number of inadvertent disclosures; (3) the extent of the disclosures; (4) any delay in measures taken to rectify the disclosure; and (5) the overriding interests in justice.2

Not long after Judge Grimm’s decision in Victor Stanley, the President signed into law a new Federal Rule of Evidence Rule 502 which was created intentionally to reduce the costs and risks associated with privilege review and to protect parties from the severe penalty of a broad subject matter waiver through the inadvertent disclosure of privileged communications. In particular, Rule 502(b) provides that:

[d]isclosure of a communication or information covered by the attorney client privilege or work product protection . . . does not operate as a waiver in a Federal or State proceeding if (1) the disclosure is inadvertent; (2) the holder of the privilege or protection took reasonable steps to prevent disclosure; and (3) the holder promptly took reasonable steps to rectify the error, including (if applicable) following Federal Rule of Civil Procedure 26(b)(5)(B)3.

While the thrust of Rule 502 is reasonableness, the courts have been inconsistent in addressing the issue of what is reasonable. However, the Advisory (Explanatory) Notes to Federal Rule of Evidence 502(b) specifically state that:

[d]epending on the circumstances, a party that uses advanced analytical software applications and linguistic tools in screening for privilege and work product may be found to have taken ‘reasonable steps’ to prevent inadvertent disclosure. The implementation of an efficient system of records management before litigation may also be helpful.


In today’s world of electronically stored information (“ESI”), the explosion in the volume and complexity of the information that is potentially subject to review in any litigation has shown that the traditional manual review by lawyers is unacceptable. However, the transition to technology-assisted document review has been slow and the acceptance of non-traditional methods for document review, more appropriately information review, by the courts has been fleeting.

A well-known 1985 study by Blair and Maron showed that even skilled reviewers making multiple passes on a batch of documents were only identifying 20 percent of the relevant documents.4 However, a recent article by Maura Grossman and Gordon Cormack5 demonstrates that an e-discovery process that uses automated tools to prioritize and select documents for review is more efficient resulting in superior results at a significant cost savings. If technology created this problem, will technology be the solution?

Companies with sophisticated record retention programs know that the weakest link in the record retention protocol is the human element. If the author or custodian of the record fails to follow the record retention protocol for whatever reason, the system breaks down and the record is not properly retained. The more automated the record retention system, the greater the likelihood for compliance, reliability and defensibility. If the same is true for document review then the answer is simple — a fully automated document review with no human (this is meant to refer to lawyers) contact. In-house counsel would probably prefer a system where the cost is minimal, the results are reliable, there is no waste of time for human review and the process is completely defensible. While we might not be ready a world as portrayed in iRobot, the 2004 film starring Will Smith, where anthropomorphic robots were viewed as inherently safe and dependable and could be used as servants for various functions, some combination of human and computer collaboration to identify relevant documents kept in electronic form should be an option.

We are already using technology-assisted review processes in terms of keyword or Boolean searches but those have resulted in limited success in terms of reliability. The next step — predictive coding — involves the more sophisticated use of concept searching to identify information that is conceptually similar to the search terms provided. A concept search will identify the query or search term and any additional words with similar meanings. Indeed, a 2010 recent survey by Anne Kershaw and Joe Howie showed that users of predictive coding software reported average savings of 45 percent with some respondents reporting savings of up to 100 percent.6 But predictive coding may not be right for every organization.


Predictive coding involves the use of sophisticated technology to automate part but not all of the document review process. It involves a significant amount of pre-review planning by lawyers knowledgeable about the topics and the issues, who will organize and prioritize sets of information based on their relation to the issues in the case — relevance, privilege, confidential, proprietary or trade secret information, etc. It starts with experienced attorneys identifying information, documents or data from the information identified and collected by the company as potentially relevant (the “collection set”). The documents selected from the collection set as relevant and privileged (the “sample set”) are then analyzed by the predictive coding software to determine a base set of relevant information — names, groups, dates, places, concepts, etc., to set parameters for reviewing the remainder of the collection set.

Critics of predictive coding cite defensibility issues noting that courts are reluctant to accept new technology that deviates from the norm or standard technical solutions without proof of authenticity. They also note that there is a general reluctance among lawyers (and clients) to be the “test case” for any new technology for fear that the court will reject the technology and the time and expense employed to implement the new system will be wasted. However, it is highly unlikely that a court will issue an opinion stating that predictive coding is approved. Rather, it is more likely that a court will issue a decision that finds that the use of predictive coding was reasonable based upon the process that was designed and employed given the facts of the case.


In The Sedona Conference® Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery,7 the Sedona Conference highlights a couple of the issues associated with the use of technology: (i) whether automated search and information retrieval methods are reliable and accurate, and (ii) whether the legal profession has developed the skills, know-how and processes to use such automated search and retrieval methods intelligently, when applied to huge data sets, in ways that are defensible under the rules governing discovery. Whether the technology is reliable, accurate and defensible will depend in large part on the design and implementation of the process for implementing the technology-assisted document review software. Just like the key to a defensible preservation protocol starts with a well-designed records retention program, the key to defensibility of a technology-assisted document review like predictive coding starts with the design of the process. Was the approach the result of a collaborative design that was reviewed, analyzed, and audited to verify the results? Is the process transparent such that a third party could repeat the procedure and get the same results? What alternatives were considered or should have been considered? If the answers to these questions show that the company has acted in good faith in implementing the use of technology to reduce the costs of litigation, and at the same time increase the efficiency of the document review process, that should be enough. Perfection at any cost is not the goal. Indeed, Rule 1 of the Federal Rules of Civil Procedure states that the Rules of Civil Procedure “should be construed and administered to secure the just, speedy, and inexpensive determination of every action.”


While it is difficult to answer this question in the abstract, it would seem that for certain companies, predictive coding would make a great deal of sense and cents. Based upon Judge Grimm’s discussion of “pre-production efforts” in Hopson and the Advisory Notes to Rule 502(b) in essence encouraging the use of analytical software along with a proven records management program, it would appear that the “success” table has been set for an experienced company which does its due diligence and implements predictive coding. The cost savings could be substantial and while there is still risk that the process will not be accepted by the court, there are some additional steps that a company can take to maximize its chances for success. As noted, a good starting place would be a defensible and reliable records management system. It would then be helpful to incorporate the company’s experience in the management of electronic information as well as electronic discovery into a strategic business process as opposed to an event-driven (such as a specific litigation) response. This approach should put the company in a better position to defend the use of advanced technology-assisted document review such as predictive coding. Moreover, to the extent that a company is prepared to use predictive coding, it would be helpful to engage the opposition in a discussion about the advanced technology-assisted software the company is planning to use. If this transparency of process results in an agreement between the parties, while not binding on the court, it could avoid significant problems later in the litigation. If an agreement cannot be reached with the other side about predictive coding, it might be helpful to raise the issue with the court to see if it has any initial problems or concerns that could be addressed before the process is implemented. While there are no guarantees in litigation, transparency and cooperation will be particularly helpful in this cutting-edge area of the law of e-discovery.

Kevin F. Brady is a Litigation Partner and Chair of the Business Law Group of the Wilmington, Delaware office of Connolly Bove Lodge & Hutz LLP. Kevin’s practice includes corporate and complex commercial litigation in the Delaware state and federal courts. Prior to joining Connolly Bove, Kevin was Counsel in the Complex Mass Torts and Insurance Group of Skadden Arps Slate Meagher & Flom LLP in Wilmington, Delaware, for almost 20 years. He is a frequent author in the areas of Delaware corporate law, electronic discovery, records management, and data security. Kevin can be reached at  


This document and any discussions set forth herein are for informational purposes only, and should not be construed as legal advice, which has to be addressed to particular facts and circumstances involved in any given situation. Review or use of the document and any discussions does not create an attorney-client relationship with the author or publisher. To the extent that this document may contain suggested provisions, they will require modification to suit a particular transaction, jurisdiction or situation. Please consult with an attorney with the appropriate level of experience if you have any questions. Any tax information contained in the document or discussions is not intended to be used, and cannot be used, for purposes of avoiding penalties imposed under the United States Internal Revenue Code. Any opinions expressed are those of the author. The Bureau of National Affairs, Inc. and its affiliated entities do not take responsibility for the content in this document or discussions and do not make any representation or warranty as to their completeness or accuracy.  

©2014 The Bureau of National Affairs, Inc. All rights reserved. Bloomberg Law Reports ® is a registered trademark and service mark of The Bureau of National Affairs, Inc.  

Request Bloomberg Law®