  Recent advancements in Generative Language Models (GLMs) have transformed
Natural Language Processing (NLP) by showcasing the effectiveness of the
"pre-train, prompt, and predict" paradigm in utilizing pre-trained GLM
knowledge for diverse applications. Despite their potential, these capabilities
lack adequate quantitative characterization due to the absence of comprehensive
benchmarks, particularly for low-resource languages. Existing low-resource
benchmarks focus on discriminative language models like BERT, neglecting the
evaluation of generative language models. Moreover, current benchmarks often
overlook measuring generalization performance across multiple tasks, a crucial
metric for GLMs.
  To bridge these gaps, we introduce NLEBench, a comprehensive benchmark
tailored for evaluating natural language generation capabilities in Norwegian,
a low-resource language. We use Norwegian as a case study to explore whether
current GLMs and benchmarks in mainstream languages like English can reveal the
unique characteristics of underrepresented languages. NLEBench encompasses a
suite of real-world NLP tasks ranging from news storytelling, summarization,
open-domain conversation, natural language understanding, instruction
fine-tuning, toxicity and bias evaluation, to self-curated Chain-of-Thought
investigation. It features two high-quality, human-annotated datasets: an
instruction dataset covering traditional Norwegian cultures, idioms, slang, and
special expressions, and a document-grounded multi-label dataset for topic
classification, question answering, and summarization. This paper also
introduces foundational Norwegian Generative Language Models (NorGLMs)
developed with diverse parameter scales and Transformer-based architectures.
Systematic evaluations on the proposed benchmark suite provide insights into
the capabilities and scalability of NorGLMs across various downstream tasks.
