  Recent strides in Large Language Models (LLMs) have saturated many NLP
benchmarks (even professional domain-specific ones), emphasizing the need for
novel, more challenging novel ones to properly assess LLM capabilities. In this
paper, we introduce a novel NLP benchmark that poses challenges to current LLMs
across four key dimensions: processing long documents (up to 50K tokens),
utilizing domain specific knowledge (embodied in legal texts), multilingual
understanding (covering five languages), and multitasking (comprising legal
document to document Information Retrieval, Court View Generation, Leading
Decision Summarization, Citation Extraction, and eight challenging Text
Classification tasks). Our benchmark comprises diverse legal NLP datasets from
the Swiss legal system, allowing for a comprehensive study of the underlying
Non-English, inherently multilingual, federal legal system. Despite recent
advances, efficiently processing long documents for intense review/analysis
tasks remains an open challenge for language models. Also, comprehensive,
domain-specific benchmarks requiring high expertise to develop are rare, as are
multilingual benchmarks. This scarcity underscores our contribution's value,
considering most public models are trained predominantly on English corpora,
while other languages remain understudied, particularly for practical
domain-specific NLP tasks. Our benchmark allows for testing and advancing the
state-of-the-art LLMs. As part of our study, we evaluate several pre-trained
multilingual language models on our benchmark to establish strong baselines as
a point of reference. Despite the large size of our datasets (tens to hundreds
of thousands of examples), existing publicly available models struggle with
most tasks, even after in-domain pretraining. We publish all resources
(benchmark suite, pre-trained models, code) under a fully permissive open CC
BY-SA license.
