  A great part of software development involves conceptualizing or
communicating the underlying procedures and logic that needs to be expressed in
programs. One major difficulty of programming is turning concept into code,
especially when dealing with the APIs of unfamiliar libraries. Recently, there
has been a proliferation of machine learning methods for code generation and
retrieval from natural language queries, but these have primarily been
evaluated purely based on retrieval accuracy or overlap of generated code with
developer-written code, and the actual effect of these methods on the developer
workflow is surprisingly unattested. We perform the first comprehensive
investigation of the promise and challenges of using such technology inside the
IDE, asking "at the current state of technology does it improve developer
productivity or accuracy, how does it affect the developer experience, and what
are the remaining gaps and challenges?" We first develop a plugin for the IDE
that implements a hybrid of code generation and code retrieval functionality,
and orchestrate virtual environments to enable collection of many user events.
We ask developers with various backgrounds to complete 14 Python programming
tasks ranging from basic file manipulation to machine learning or data
visualization, with or without the help of the plugin. While qualitative
surveys of developer experience are largely positive, quantitative results with
regards to increased productivity, code quality, or program correctness are
inconclusive. Analysis identifies several pain points that could improve the
effectiveness of future machine learning based code generation/retrieval
developer assistants, and demonstrates when developers prefer code generation
over code retrieval and vice versa. We release all data and software to pave
the road for future empirical studies and development of better models.
