National Conference of Bar Examiners, Testing Task Force Phase 2 Report: 2019 Practice Analysis (March 2020):
In 2018, the National Conference of Bar Examiners (NCBE) created a Testing Task Force (TTF) to undertake a comprehensive three-year study to ensure that the bar examination continues to test the knowledge, skills, and abilities needed for competent entry-level legal practice in a changing profession. The TTF’s study consists of three phases. Phase 1 was a series of listening sessions with stakeholders to solicit their impressions about the current bar examination and ideas for the next generation of the bar examination. Phase 2 consisted of a national practice analysis to provide empirical data on the job activities of newly licensed lawyers (NLLs), which are defined in the practice analysis survey as lawyers who have been licensed for three years or less. Phase 3, which will be completed in 2020, will translate the results from Phase 1 and Phase 2 into a recommended test blueprint and design for the next generation of the bar examination. This Executive Summary provides a high-level synthesis of the 2019 Practice Analysis Report.
The 77 knowledge areas were rated in terms of their importance to the practice of all NLLs. The overall means for all knowledge areas as rated by NLLs and non-NLLs were nearly identical, and the correlation between the two sets of ratings was very high; thus, data for the two groups were combined for the analyses in this report.
The knowledge areas with the highest mean importance ratings included the following: Rules of Professional Responsibility and Ethical Obligations, Civil Procedure, Contract Law, Rules of Evidence, and Legal Research Methodology. Among the lowest-rated knowledge areas were topics such as Transportation Law, Bioethics, Public Utility Law, Sports and Entertainment Law, and Admiralty Law.
Knowledge importance ratings were remarkably consistent across demographic groups; that is, mean ratings did not vary much based on the demographic backgrounds of respondents. However, mean knowledge area ratings did vary by practice area. For example, the mean importance rating for Business Organizations Law by those respondents who practice Criminal Law was lower than that from respondents who practice primarily Real Estate Law. Decisions about whether to include a knowledge area in the test blueprint should include evaluation of the extent to which it is relevant to multiple practice areas; therefore, results based on practice clusters will be taken into consideration.
The Knowledge Areas section of the survey has direct implications for the test blueprint because most licensure tests include an assessment of the subject matter knowledge required for competent practice. ...
Based on the systematic process of developing the practice analysis survey, and of gathering information from a representative sample of lawyers, stakeholders should have confidence that the practice analysis results provide meaningful guidance for the TTF’s comprehensive study of the bar examination. Next, the TTF will appoint an independent panel of subject matter experts (SMEs) to translate the results of the practice analysis survey into a test blueprint and test design. The test blueprint will identify the knowledge and skill domains to be assessed by the bar examination and the emphasis to be allocated to each domain. After that, the TTF will appoint a test design committee composed of external stakeholders, such as bar administrators, bar examiners, justices, and legal educators. The test design committee will focus on methods of assessment (e.g., multiple-choice or essay questions), the timing and sequencing of those assessments, procedures for scoring, and other important features of test delivery. Recommendations regarding the test blueprint and test design will be reviewed by NCBE’s Technical Advisory Panel, and the TTF will seek input more broadly from the stakeholder community before deciding on the blueprint and design recommendations to submit to the NCBE Board of Trustees at the end of 2020.