Better Call GPT is not the latest spin-off of Breaking Bad, but the title of a research by a team of New Zealand researchers from the company Onit, who analyzed how large language tools (LLM)like ChatGPT, beat lawyers when it comes to comparing and reviewing contracts.
ChatGPT (almost) outperforms lawyers in contract review
The research report, titled “Better Call GPT, Comparing Large Language Models Against Lawyers” (via Punto Informatico), examined the performance of AI LLM models, including GPT-4 of OpenAI, in the contractual review. It appears that the GPT-4 model is able to compete with and sometimes outperform contract law professionals, including experienced lawyers. Although it must be said that Orbit does provide software and AI services in the legal sector.
Researchers have tested various LLMs, including Google Bard (now Gemini) and that of Amazon, applying them to real procurement contracts. GPT-4 demonstrated accuracy comparable to that of lawyers, especially in detecting contractual problems. Although lawyers showed a slight advantage in detection, the differences were negligible. And LLMs far outpaced lawyers in speed: seconds versus hours.
In addition to performance, the research evaluated the costs of services. Using LLMs has proven to be significantly cheaper, costing only 2 cents per contract respect to $74 from a junior lawyer. Which suggests that several studios could use them in the future.
While further research is needed to validate these findings, the research offers interesting perspectives. Human control, however, remains essential: AI can make even colossal errors. And when the risk is of losing an important case or signing an incorrect contract, the eyes of a professional can (and perhaps must) make the difference.
