  Creating programs to correctly manipulate data is a difficult task, as the
underlying programming languages and APIs can be challenging to learn for many
users who are not skilled programmers. Large language models (LLMs) demonstrate
remarkable potential for generating code from natural language, but in the data
manipulation domain, apart from the natural language (NL) description of the
intended task, we also have the dataset on which the task is to be performed,
or the "data context". Existing approaches have utilized data context in a
limited way by simply adding relevant information from the input data into the
prompts sent to the LLM.
  In this work, we utilize the available input data to execute the candidate
programs generated by the LLMs and gather their outputs. We introduce semantic
reranking, a technique to rerank the programs generated by LLMs based on three
signals coming the program outputs: (a) semantic filtering and well-formedness
based score tuning: do programs even generate well-formed outputs, (b) semantic
interleaving: how do the outputs from different candidates compare to each
other, and (c) output-based score tuning: how do the outputs compare to outputs
predicted for the same task. We provide theoretical justification for semantic
interleaving. We also introduce temperature mixing, where we combine samples
generated by LLMs using both high and low temperatures. We extensively evaluate
our approach in three domains, namely databases (SQL), data science (Pandas)
and business intelligence (Excel's Power Query M) on a variety of new and
existing benchmarks. We observe substantial gains across domains, with
improvements of up to 45% in top-1 accuracy and 34% in top-3 accuracy.
