Eye on AI Weekly Research Watch
Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System
· 2 min · Seizoen 1
Beschrijving
LLMs are increasingly used in legal work, yet their reliability outside the United States is poorly documented. This expert-validated benchmark has 1,042 items across ten areas of law and three formats, and was used to test 15 models. Closed-question accuracy reaches 0.905, but free-text correctness never exceeds 0.45, and only about half of cited norms are correct. Models often sound responsive while being wrong, which is risky for non-experts.
Authors: Rubén Manrique, Michelle Castellanos, Jorge Morales, Juan David Gutiérrez, Antonio Barreto Rozo, Joaquín Vélez Navarro
Paper…
Website: share.transistor.fm
