  Ever since the launch of ChatGPT in 2022, a rising concern is whether ChatGPT
will replace programmers and kill jobs. Motivated by this widespread concern,
we conducted an empirical study to systematically compare ChatGPT against
programmers in question-answering and software-maintaining. We reused a dataset
introduced by prior work, which includes 130 StackOverflow (SO) discussion
threads referred to by the Java developers of 357 GitHub projects. We mainly
investigated three research questions (RQs). First, how does ChatGPT compare
with programmers when answering technical questions? Second, how do developers
perceive the differences between ChatGPT's answers and SO answers? Third, how
does ChatGPT compare with humans when revising code for maintenance requests?
  For RQ1, we provided the 130 SO questions to ChatGPT, and manually compared
ChatGPT answers with the accepted/most popular SO answers in terms of
relevance, readability, informativeness, comprehensiveness, and reusability.
For RQ2, we conducted a user study with 30 developers, asking each developer to
assess and compare 10 pairs of answers, without knowing the information source
(i.e., ChatGPT or SO). For RQ3, we distilled 48 software maintenance tasks from
48 GitHub projects citing the studied SO threads. We queried ChatGPT to revise
a given Java file, and to incorporate the code implementation for any
prescribed maintenance requirement. Our study reveals interesting phenomena:
For the majority of SO questions (97/130), ChatGPT provided better answers; in
203 of 300 ratings, developers preferred ChatGPT answers to SO answers; ChatGPT
revised code correctly for 22 of the 48 tasks. Our research will expand
people's knowledge of ChatGPT capabilities, and shed light on future adoption
of ChatGPT by the software industry.
