The Benefits Of Greater Transparency In Reporting Of Law Graduate Employment Outcomes

Ten years ago, there was no standardized reporting format law schools had to follow for reporting graduate employment outcomes.  As a result, it was very difficult to make comparisons across law schools because the data were presented by law schools in a way that did not facilitate apples to apples comparisons.  At that time, many law schools presented employment outcomes data in a manner that reflected as favorably as possible on the institution.

Following the Great Recession, the ABA mandated greater transparency in the way law schools report employment outcomes, requiring each law school to report disaggregated results for all law school graduates using a common template — the Employment Summary Report.  This mandated reporting format has fostered much greater transparency with respect to graduate employment outcomes by increasing both the amount and detail of employment outcomes data and by fostering comparisons across law schools due to a consistent reporting framework. 

Interestingly, however, throughout this period of time, there has been some variability in how entities compiling employment outcomes data report such data, particularly as it relates to the "top line" statistic — what proportion of each class of new law graduates is working nine (now ten) months after graduation — an interval long enough to take and get results on a bar exam.  While NALP has reported its results on a consistent basis throughout this period of time, US News and World Report has significantly changed its methodology for reporting employment outcomes, largely in response to the ABA’s requirement of increased transparency with respect to employment outcomes data.

To understand the extent to which the ABA’s mandate for increased transparency in reporting of employment outcomes data has mattered, one need only compare aggregated employment outcomes data as reported by NALP and USNews for the Class of 2004 and the Class of 2014 (five years before and five years after the Great Recession).  As shown below, the ABA’s requirement for greater transparency in reporting of employment outcomes data has resulted in public dissemination of much more realistic understandings of employment outcomes of law graduates. 

Table 1 shows the comparison of reported employment outcomes for the class of 2004 and for the class of 2014 as presented by NALP (reporting data on all graduates with reported employment outcomes) and by USNews (reporting the average of employment outcomes for the top 100 law schools in each year).

TABLE 1 – Class of 2004 and Class of 2014 Employment Outcomes Percentages as Reported by NALP (All Graduates with Reported Employment Outcomes) and USNews (Avg. of Top-100 Law Schools)

	Class of 2004	Class of 2014	Change 2004 to 2014
NALP	88.9	86.7	-2.2
USNews (Avg. of Top-100 Law Schools)	95.8	74.7	-21.1
Difference	+6.9	-12

The NALP Calculations

NALP reported employment rates for the Class of 2004 and the Class of 2014 as being 88.9% and 86.7%, respectively.  This highlights that the post-recession employment market had not rebounded completely to the pre-recession levels for law school graduates.  (Notably, employment percentages have improved in recent years, with the Class of 2018 at 89.4% according to NALP, although this increased percentage employed is largely due to the decrease in the number of graduates between 2014 and 2018.)

In calculating these percentages, NALP used the same formula in both reporting periods.  It counted in the numerator anyone who is employed and counted in the denominator anyone whose employment status was known.

The numerator (number employed) and denominator (number whose status is known) are defined below.

• A graduate who has a job as of March 15 (February 15 in 2004 and 2014) is employed. The job may be full-time, part-time, temporary, permanent, law-related or not.
• Graduates whose status is known include, in addition to those who are employed, those pursuing an advanced degree full-time, those not working and seeking a job, and those neither working nor seeking a job.

This is a pretty generous definition, as it includes any job, law-related or not, whether part-time or temporary.  But it is also a consistent definition in both reporting periods (and remains consistent today).  (Note that throughout this period, NALP also has promoted transparency by providing much more disaggregated information regarding the types of employment outcomes graduates had obtained in its annual Jobs and JDs publication.)

The USNews Calculations

By contrast, employment outcomes as reported by USNews were much more generous than NALP’s for the class of 2004 and much less generous than NALP’s for the class of 2014.

In the published data released by USNews in its 2007 rankings (which were published in March 2006 and encompassed employment outcomes data for the class of 2004 (which had been reported in spring 2005)), the average of the employment rates for the top-100 law schools for the class of 2004 was 95.8%, nearly seven percentage points higher than the NALP percentage employed for the class of 2004. 

In 2004, before the focus on greater transparency in reporting of employment outcomes, the published employment rates in USNews were calculated as follows:

Employment rates include graduates reported as working or pursuing graduate degrees; for the nine month rate only, 25 percent of those whose status is unknown are also counted as working. Those not seeking jobs are excluded.

This explains how the USNews average employment percentage could be nearly seven percentage points higher than the NALP percentage for the class of 2004.  USNews increased the numerator by including as working those pursuing a graduate degree and one-quarter of those whose status was unknown. USNews also decreased the denominator by excluding those not seeking jobs.

Ten years later, however, USNews reports a very different employment picture than NALP — a much less rosy employment picture.           

In the published data released by USNews in its 2017 rankings (which was published in March 2016 and encompassed employment outcomes data for the class of 2014 (which had been reported in spring 2015)), the average of the employment rates for the top-100 law schools for the class of 2014 was 74.7%, 12 percentage points lower than the NALP percentage employed for the class of 2014. 

Thus, even though the NALP data showed only a 2.2 percentage point decline in graduates employed between 2004 and 2014, the USNews data showed a 21.1 percentage point decline in the average percentage of graduates reported as being employed among law schools ranked in the USNews top-100 between the class of 2004 and the class of 2014.

What Explains the Difference?  Much Greater Transparency

The much greater difference in the USNews average reported employment rates for the top-100 ranked law schools for the class of 2004 and the class of 2014 (compared to the NALP tallies for all law schools in those years) is attributable to a different methodology the USNews used for the class of 2014 that reflects a much greater emphasis on transparency.

By the time the class of 2014 graduated, the ABA had instituted its new set of reporting requirements to foster greater transparency with regard to employment outcomes, including the Employment Summary Report each school must post.  In response, USNews changed how it reported the employment rates such that for the Class of 2014:

Employment rates published in the tables reflect all full-time jobs lasting at least a year for which bar passage was required or a J.D. degree was an advantage. 

US News thus excluded from the published employment rates for the class of 2014 a number of jobs that had been included in 2004 (any part-time or short-term jobs and any jobs that were not bar passage required or JD advantage positions).  It also appears that US News did not add to the numerator a factor for those whose employment status was unknown or for those pursuing a graduate degree and did not exclude from the denominator those not seeking employment. (Notably, however, according to an email from Robert Morse, the US News published employment rates for the class of 2014 did include law school/university funded jobs.)

In reporting employment outcomes for the class of 2004 during the era of limited transparency, US News (and law schools) reported employment outcomes even more robust than those reported by NALP.

In the new era of increased transparency, US News reports employment outcomes focused only on what some might consider “good” law or law-related jobs — full-time, long-term positions that are bar passage required or JD-advantage positions.  This provides a much more realistic picture of employment prospects for prospective law students than what they would have seen and understood ten to fifteen years ago.     

Looking at Differences across Different Categories of Law Schools

These differences between the class of 2004 and class of 2014 US News employment rates are even more pronounced when one looks at disaggregated results focused on law schools in different rankings categories.

As shown in Table 2, the difference in employment rates between the class of 2004 and the class of 2014 increases from modest declines among top-10 law schools to significantly larger declines as one goes down the rankings.  (Again, as I noted above, the results for the class of 2014 include law-school-funded jobs.  In a blog I posted in May 2016, I highlighted that law school funded positions in 2014 were much more likely to be found among law schools in the top-25 than among lower-ranked law schools.  Thus, some of the difference across ranking categories is attributable to the disproportionate presence of law-school-funded positions among higher-ranked law schools.)

TABLE 2 – Average Percentage Employed across Categories of Law Schools in USNews Rankings 2007 and 2017 (Class of 2004 and Class of 2014)

	Class of 2004	Class of 2014	Change 2004 to 2014
Top 10 Avg.	99.3	93.4	-5.9
11-25 Avg.	98.1	82.1	-16
26-50 Avg.	96	73.6	-22.4
51-100 Avg.	94.3	69.2	-25.1
Top 100 Overall Avg.	95.8	74.7	-21.1
Diff. between Top 10 Avg. and 51-100 Avg.	-5.0	-24.2

During the ABA Questionnaire Committee’s hearing on employment reporting in December 2010, several law students or recent graduates commented that they had not really considered employment outcomes in weighing decisions about where to go to law school.  The Class of 2004 column in Table 2 demonstrates why people were not concerned about employment outcomes prior to the Great Recession.  When all the law schools you are considering appear to be reporting employment outcomes of 94% or better, it suggests that employment outcomes are not a meaningful basis on which to make distinctions among law schools. 

With the reporting for the class of 2014, however, the range of employment outcomes does provide a justification for prospective law students to look more closely at employment outcomes in deciding whether and where to go to law school.  Moreover, the disaggregated detail contained in the Employment Summary Report allows prospective law students to understand better the difference in “types” of jobs graduates obtain at various law schools — small firm, large firm, government, business, judicial clerkships — which allows prospective students with an interest in a specific practice context to take that into account when choosing among law schools.

Finally, it is worth noting that one great consequence of increased transparency in reporting of employment outcomes is a greater alignment between the interests of law schools and the interests of their graduates.  With much greater attention being paid to employment outcomes of graduates, and with consistent reporting obligations now mandated through the Employment Summary Reports, law schools have a much greater interest in supporting their law graduates in their efforts to seek meaningful employment following graduation.  In addition, as Kellye Testy recently noted in providing an enrollment update for fall 2019, this desire to promote better employment outcomes for graduates may be functioning to constrain law schools from admitting too many students even as the applicant pool rebounds.

(I am very grateful to Bernie Burk and Scott Norberg for very helpful comments on an earlier draft of this blog.)