  Language models, primarily transformer-based ones, obtained colossal success
in NLP. To be more precise, studies like BERT in NLU and works such as GPT-3
for NLG are very crucial. DNA sequences are very close to natural language in
terms of structure, so if the DNA-related bioinformatics domain is concerned,
discriminative models, like DNABert, exist. Yet, the generative side of the
coin is mainly unexplored to the best of our knowledge. Consequently, we
focused on developing an autoregressive generative language model like GPT-3
for DNA sequences. Because working with whole DNA sequences is challenging
without substantial computational resources, we decided to carry out our study
on a smaller scale, focusing on nucleotide sequences of human genes, unique
parts in DNA with specific functionalities, instead of the whole DNA. This
decision did not change the problem structure a lot due to the fact that both
DNA and genes can be seen as 1D sequences consisting of four different
nucleotides without losing much information and making too much simplification.
First of all, we systematically examined an almost entirely unexplored problem
and observed that RNNs performed the best while simple techniques like N-grams
were also promising. Another beneficial point was learning how to work with
generative models on languages we do not understand, unlike natural language.
How essential using real-life tasks beyond the classical metrics such as
perplexity is observed. Furthermore, checking whether the data-hungry nature of
these models can be changed through selecting a language with minimal
vocabulary size, four owing to four different types of nucleotides, is
examined. The reason for reviewing this was that choosing such a language might
make the problem easier. However, what we observed in this study was it did not
provide that much of a change in the amount of data needed.
