  Automated logging statement generation techniques facilitate developers in
writing appropriate logging statements that document software behaviors.
Current retrieval-based and learning-based logging methods fail to provide
accurate logging statements in complex software. Although existing large
language models (LLMs) might be a good fit for the task due to their great
success in natural language generation and programming language comprehension,
their effectiveness and generalization capabilities have not been explored. To
this end, this paper performs the first extensive study on applying LLMs for
logging statement generation. We build LogBench, the first logging statement
generation dataset. On LogBench, we evaluate the effectiveness and
generalization capabilities of eight state-of-the-art LLMs, which include
general-purpose and code-specific models ranging from 60M to 175B in size.
Specifically, we evaluate LLM's logging effectiveness by studying 1) their
ability to decide logging ingredients, 2) the impact of the internal
characteristics of LLMs, and 3) the influence of external factors. We further
evaluate LLM's logging generalization capabilities using unseen data derived
from code transformation techniques. Our study demonstrates that existing LLMs
fall short of practical requirements for generating proper logging statement
texts. We also disclose the impact of internal characteristics and external
factors for LLMs in automated logging. In addition, we observe that existing
LLMs cannot generalize to logging unseen code, revealing their unsatisfactory
generalization capabilities. Based on our findings, we further discuss three
implications that can enhance logging statement generation in the future, such
as developing a unified metric for logging quality, incorporating shareable
code knowledge into LLMs, and devising suitable prompts.
