  Tensor program tuning is a non-convex objective optimization problem, to
which search-based approaches have proven to be effective. At the core of the
search-based approaches lies the design of the cost model. Though deep
learning-based cost models perform significantly better than other methods,
they still fall short and suffer from the following problems. First, their
feature extraction heavily relies on expert-level domain knowledge in hardware
architectures. Even so, the extracted features are often unsatisfactory and
require separate considerations for CPUs and GPUs. Second, a cost model trained
on one hardware platform usually performs poorly on another, a problem we call
cross-hardware unavailability.
  In order to address these problems, we propose TLP and MTLTLP. TLP is a deep
learning-based cost model that facilitates tensor program tuning. Instead of
extracting features from the tensor program itself, TLP extracts features from
the schedule primitives. We treat schedule primitives as tensor languages. TLP
is thus a Tensor Language Processing task. In this way, the task of predicting
the tensor program latency through the cost model is transformed into a natural
language processing (NLP) regression task. MTL-TLP combines Multi-Task Learning
and TLP to cope with the cross-hardware unavailability problem.
  We incorporate these techniques into the Ansor framework and conduct detailed
experiments. Results show that TLP can speed up the average search time by 9.1X
and 3.0X on CPU and GPU workloads, respectively, compared to the
state-of-the-art implementation. MTL-TLP can achieve a speed-up of 4.7X and
2.9X on CPU and GPU workloads, respectively, using only 7% of the target
hardware data.
