Maura Grossman Talks ‘Virtuous Cycle' in TAR Research

Bloomberg Law’s combination of innovative analytics, research tools and practical guidance provides you with everything you need to be a successful litigator.

By Tera Brostoff

Aug. 15 — The development of technology-assisted review is about to get a boost from the person who knows it best. Maura Grossman just left her 17-year career at Wachtell, Lipton, Rosen & Katz to devote her time to researching and evaluating information retrieval tools—and then applying her research to real-world practice at her own law and consulting firm.

She's joining the David R. Cheriton School of Computer Science, University of Waterloo in Ontario, Canada as a research professor, where she plans to advance the information retrieval tools available, making them more efficient and accessible.

Her new career is springboarding off her work at Wachtell, where she spent almost two decades in the eDiscovery field as a corporate attorney, contributing to the growth and recognition of many of the processes viewed today as critical to the culling, preserving and production process in litigation.

Making Moves

Grossman will be collaborating with Gordon Cormack and other faculty and graduate students at the School of Computer Science to advance the state of the art of information retrieval tools.

The goal of their study is to develop more effective and efficient methods for dealing with “big data.”

“In order to advance the state of the art in any field, it is necessary to conduct controlled experiments and to measure outcomes; to determine what works and what does not, or what is an improvement and what is not,” she told Bloomberg BNA.

In her own words, Grossman will serve as an “incubator” for inventing and evaluating new tools.

But once the laboratory experiments suggest which methods are most promising, it's necessary to test them with actual users.

The “user studies” will take place in the lab first, and then subsequently in the field, with real matters, real datasets, real information needs and real users.

“Along the way, we discover new challenges and research problems, and the cycle repeats itself: First, we identify and evaluate new approaches in the lab through simulation; then we test them in the lab with real users; and, finally, we try them out in the field,” Grossman said. “Elsewhere, we have referred to this process as a ‘virtuous cycle.'”

Grossman has also opened an eDiscovery law and consulting practice in New York to facilitate technology transfer from the laboratory to practice.

She'll be deploying the new tools through the practice, and bring back any issues or questions to the lab for further study.

Defining Technology-Assisted Review

Technology-assisted review is a collection of software-based tools that allow an individual to cull and prioritize a large data set by training the tool on what type of documents are being sought.

Understanding TAR

Grossman hopes her research and applications will clarify some of the mislabeling and misunderstanding of the tools that has happened along the way. She said she was loath to refer to TAR in the singular, as one “tool.”

She and Cormack are pleased that practitioners and the courts have taken notice of their study showing that certain TAR methods compare favorably with human review, she said.

“At the same time, we are concerned that the term ‘TAR' has been diluted nearly beyond recognition, and that our work has been cited as testament to the efficacy of methods that bear no resemblance to those we evaluated, and that we do not endorse,” she said.

Since the publication of their seminal article in the 2011 Richmond Journal of Law and Technology, many vendors have marketed a variety of offerings as TAR, regardless of whether they employ supervised learning methods.

Supervised learning methods extrapolate from and apply the results of human coding for responsiveness on a subset of a collection, in order to rank or categorize the remaining documents in the collection.

Unsupervised learning methods, like clustering, concept search, latent semantic indexing, e-mail threading, and near-duplicate detection do not rank or categorize documents on this basis. In Grossman and Cormack's view, those tools are not TAR.

“That does not mean that they cannot be helpful in eDiscovery, it just means that they do not meet our definition of TAR, and were not the methods shown to work in our Richmond study,” Grossman said. “Nevertheless, some vendors have advanced these methods—and other very weak methods derived from them—as TAR.”

Grossman provided an application of TAR to Bloomberg BNA when she shared a compilation of Democratic vice presidential nominee Tim Kaine's e-mails, which she and the The Library of Virginia loaded into an “AutoTAR Continuous Active Learning (CAL)” system, accessible at

Essentially, TAR allows a user to search the e-mails by telling the system whether or not a specific e-mail (based on a search term or key word) is useful or not. The system then learns what is useful based on the answers, and searches for a new selection that the system determines is closer to what the user wants. With continuous training, the system is able to get closer and closer to the desired content.

TAR's Future

She also agrees with Judge Andrew Peck's recent ruling in Hyles v. New York City, 2016 BL 248010 (S.D.N.Y. Aug. 1, 2016), in which he refused to force a producing party to use a TAR tool for eDiscovery production (16 DDEE 344, 8/4/16).

“Right now, TAR is not so well understood, so readily available at an affordable price, so consistently reliable, and so commonly used, that it is unreasonable not to use it,” she said. “But Judge Peck also stated in Hyles—and I agree—that that day may still come.”

She said she doesn't know how long it will take but she believes that at some future point, it will be considered unreasonable not to use TAR.

“I think the adoption of TAR won’t necessarily be prescribed by the court per se, but it may very well be ‘mandated' through discovery schedules—or client budgets—that are impossible to meet without using state-of-the art methods such as TAR,” she predicted.

Looking Forward

Recently, retired Magistrate Judge John M. Facciola talked to Bloomberg BNA about the changing landscape of eDiscovery as more judges who have issued landmark opinions in the field retire, making way for a new cohort of attorneys and judges to take up the flag (16 DDEE 280, 6/23/16).

Grossman agreed with Facciola that the “passing of the torch” is a necessary and healthy occurrence.

“There is no question that the first generation of rock-star judges blazed the trail in eDiscovery,” she told Bloomberg BNA. “They were forced to deal with the rapidly changing paper-to-digital landscape that I described earlier.”

But the newer generation of judges will have grown up in a digital world, so addressing the challenges of electronically-stored information may come more naturally to them, she said.

To contact the reporter on this story: Tera Brostoff in Washington at

To contact the editors responsible for this story: Jessie Kokrda Kamens at, Carol Eoannou at

Copyright © 2016 The Bureau of National Affairs, Inc. All Rights Reserved.

Request Litigation on Bloomberg Law