Instruct: Identify the main and secondary category of Arxiv papers based on the titles and abstracts
Query: Tree-based Focused Web Crawling with Reinforcement Learning   A focused crawler aims at discovering as many web pages relevant to a target
topic as possible, while avoiding irrelevant ones; i.e. maximizing the harvest
rate. Reinforcement Learning (RL) has been utilized to optimize the crawling
process, yet it deals with huge state and action spaces, which can constitute a
serious challenge. In this paper, we propose TRES, an end-to-end RL-empowered
framework for focused crawling. Unlike other approaches, we properly model a
crawling environment as a Markov Decision Process, by representing the state as
a subgraph of the Web and actions as its expansion edges. TRES adopts a keyword
expansion strategy based on the cosine similarity of keyword embeddings. To
learn a reward function, we propose a deep neural network, called KwBiLSTM,
leveraging the discovered keywords. To reduce the time complexity of selecting
a best action, we propose Tree-Frontier, a two-fold decision tree, which also
speeds up training by discretizing the state and action spaces. Experimentally,
we show that TRES outperforms state-of-the-art methods in terms of harvest rate
by at least 58%, while it has competitive results in the domain maximization.
Our implementation code can be found on https://github.com/ddaedalus/TRES.

24
The categories of this paper are: dg-ga
26
The categories of this paper are: bayes-an
21
The categories of this paper are: adap-org
7
The categories of this paper are: hep-th
37
The categories of this paper are: supr-con
9
The categories of this paper are: nlin
11
The categories of this paper are: hep-lat
35
The categories of this paper are: mtrl-th
33
The categories of this paper are: plasm-ph
10
The categories of this paper are: q-bio
0
The categories of this paper are: hep-ph
29
The categories of this paper are: funct-an
28
The categories of this paper are: alg-geom
4
The categories of this paper are: cond-mat
31
The categories of this paper are: ao-sci
5
The categories of this paper are: gr-qc