Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Personalized Benchmarking with the Ludwig Benchmarking Toolkit   The rapid proliferation of machine learning models across domains and
deployment settings has given rise to various communities (e.g. industry
practitioners) which seek to benchmark models across tasks and objectives of
personal value. Unfortunately, these users cannot use standard benchmark
results to perform such value-driven comparisons as traditional benchmarks
evaluate models on a single objective (e.g. average accuracy) and fail to
facilitate a standardized training framework that controls for confounding
variables (e.g. computational budget), making fair comparisons difficult. To
address these challenges, we introduce the open-source Ludwig Benchmarking
Toolkit (LBT), a personalized benchmarking toolkit for running end-to-end
benchmark studies (from hyperparameter optimization to evaluation) across an
easily extensible set of tasks, deep learning models, datasets and evaluation
metrics. LBT provides a configurable interface for controlling training and
customizing evaluation, a standardized training framework for eliminating
confounding variables, and support for multi-objective evaluation. We
demonstrate how LBT can be used to create personalized benchmark studies with a
large-scale comparative analysis for text classification across 7 models and 9
datasets. We explore the trade-offs between inference latency and performance,
relationships between dataset attributes and performance, and the effects of
pretraining on convergence and robustness, showing how LBT can be used to
satisfy various benchmarking objectives.

15
The categories of this paper are: nucl-ex
6
The categories of this paper are: astro-ph
16
The categories of this paper are: stat
13
The categories of this paper are: quant-ph
27
The categories of this paper are: comp-gas
19
The categories of this paper are: eess
35
The categories of this paper are: mtrl-th
12
The categories of this paper are: nucl-th
18
The categories of this paper are: econ
17
The categories of this paper are: q-fin
23
The categories of this paper are: solv-int
28
The categories of this paper are: alg-geom
22
The categories of this paper are: chao-dyn
2
The categories of this paper are: math
7
The categories of this paper are: hep-th
32
The categories of this paper are: atom-ph