I recently saw a fascinating preprint of a paper testing GPT-3.5, currently the best public large language model, against the bar exam, the two-day test required to become a lawyer in the United States. As someone who has passed the bar exam and who is very interesting in programming, this intrigued me. I knew language processing had come a long way, but the bar exam requires months of study after years of law school to pass. Surely many seeing the rapid progress of GPT envision AI reaching the intelligence of humanity and robot lawyers taking over the profession.
Looking at the paper however, I was disappointed that they are only testing it against the MBE, the multiple-choice section of the exam. Randomly picking answers will score you a 25% on this and most lawyers scored about 66% when they passed. GPT got a 50%, which is still impressive, but as computers do have faulty memory, it is much less impressive when considering that it can "look up" any rote information perfectly, while humans are unable to Google during the exam.
I was hoping to see it matched against the MEE, one of the essay portions of the exam which is just as important. Since GPT is a language model, I thought that would make sense for it, while obviously being much harder to program a computer to do. However, since I have access to ChatGPT and old sample bar questions, I do not have to wait for somebody to do this. I can do it myself.
The Question
To test it, I fed it this actual former bar exam question which I used during my bar review and asked it to write "a long, detailed answer". It is a pretty typical bar question with three main issues. First, Abe gets a deed which is invalid because of an obscure rule and which he forgets to record. Then he has the potential for adverse possession, which would fail if he had actual ownership because of the unrecorded deed, but since the deed is invalid, he actually manages to do it. Then finally, Buyer purchases it, which would have precedence over the unrecorded deed but does not over the adverse possession. It is a nuanced question which can easily trip you up but provides lots of opportunity to explain different rules. ChatGPT's answer was as follows (interspersed with my comments):
In this case, it appears that Abe has the strongest claim to possession of Whiteacre.
Firstly, it is worth noting that state B has a statute stating that actions to recover possession of real property must be brought within 10 years of the cause of action accruing. As Abe has only recently returned to state B and discovered Buyer in possession of Whiteacre, it seems likely that he would be able to bring a successful action for possession of the property.
Adverse possession is one of the big, obvious issues in this problem, and it has a nice five-point requirement bar takers should go through each point of. GPT does not mention any of these points, and just says it applies. This is very bad form, as the rule and application thereof are the most important parts of the bar. The conclusion does not matter nearly as much as how you got there. In fact, if you missed the part about the deed being invalid (which GPT did) you're actually supposed to give the wrong answer, since it would be the logical answer in that case.
Furthermore, Abe has a deed from Owen conveying Whiteacre to him, which was signed and notarized but never recorded. While state B has a statute stating that conveyances of real property must be recorded in order to be valid against subsequent purchasers for value and without notice, the fact that the deed was never recorded does not necessarily invalidate it. Instead, the lack of recording would likely only affect the notice that subsequent purchasers have of the conveyance, rather than the conveyance itself. In this case, both Doris and Buyer were unaware of the Owen-to-Abe deed and believed that Owen was the owner of Whiteacre at the time of his death, so they would be considered subsequent purchasers without notice of the earlier conveyance.
Here you can see it missing the issue with the deed. Since it lacked the grantee's name, it was actually invalid. GPT correctly repeats the given rule about subsequent purchasers without notice of earlier conveyances, but it reaches the exact opposite of the logical conclusion. It repeats the rule. It says the rule would invalidate the deed. But it implies that the deed still strengthens Abe's claim instead of acknowledging that Buyer would prevail if relying on the deed.
Additionally, Abe has been in possession of Whiteacre for the past 20 years, during which time he used the land for various purposes such as planting vegetables, cutting timber, and allowing others to park vehicles there. This long period of possession, combined with the fact that he paid the real property taxes on the land and took other actions to assert ownership, could be considered evidence of Abe's ownership of the property.
Abe was only in possession for 11 years, so it's misstated the facts. More troubling is that it says his assertion of ownership is evidence of Abe's ownership though. This seems to be just a fabrication of a new rule. These could be seen as parts of actual possession or hostile possession perhaps if one is generous, but it does not make this connection. It again merely repeats the facts and sheepishly states a conclusion.
Overall, it seems likely that a court would rule in favor of Abe's claim to possession of Whiteacre. However, it is worth noting that the specifics of the case could potentially affect the outcome, and it would ultimately be up to a court to determine the final decision.
It repeats its conclusion here, while saying that ultimately it is up to a court based on the specifics. But the question has all of the specifics. It acts like it is impossible to know the law without asking a judge.
Results
My own answer to this practice question admittedly missed the mark as well. I got both the main rules, but I, too, missed the requirement for the deed to have the grantee's name. I analyzed the remainder fairly well I thought despite that, but my essay was still "significantly below passing". ChatGPT's essay possibly would score zero points. It lacks any correct rules and any logical analysis.
Of course the "AI" doesn't actually understand the logic of the sentences. It's only operating on the statistics of the words, but its—for lack of a better word—charisma in delivering the wrong answer is quite good at fooling people into believing it is right one. While such a problem may not be unique to AI (indeed, Reddit and other social media is full of it), it does mask the massive shortcomings of GPT.
It is impressive for what it is, but as a lawyer, I see no use in an answer that sounds good when I can't be certain it's right. Relying on GPT would obviously be malpractice, and I am not worried about that changing soon. I believe, at least for the near future, my job is very secure and will likely remain LLM free.
The above is my understanding of the law. I am a lawyer, but I am not your lawyer. This post does not constitute legal advice. I make no warranty as to its accuracy or applicability to you. If you need a lawyer, get a lawyer. If you want me specifically, hire me.