  Performance metrics-driven context caching has a profound impact on
throughput and response time in distributed context management systems for
real-time context queries. This paper proposes a reinforcement learning based
approach to adaptively cache context with the objective of minimizing the cost
incurred by context management systems in responding to context queries. Our
novel algorithms enable context queries and sub-queries to reuse and repurpose
cached context in an efficient manner. This approach is distinctive to
traditional data caching approaches by three main features. First, we make
selective context cache admissions using no prior knowledge of the context, or
the context query load. Secondly, we develop and incorporate innovative
heuristic models to calculate expected performance of caching an item when
making the decisions. Thirdly, our strategy defines a time-aware continuous
cache action space. We present two reinforcement learning agents, a value
function estimating actor-critic agent and a policy search agent using deep
deterministic policy gradient method. The paper also proposes adaptive policies
such as eviction and cache memory scaling to complement our objective. Our
method is evaluated using a synthetically generated load of context sub-queries
and a synthetic data set inspired from real world data and query samples. We
further investigate optimal adaptive caching configurations under different
settings. This paper presents, compares, and discusses our findings that the
proposed selective caching methods reach short- and long-term cost- and
performance-efficiency. The paper demonstrates that the proposed methods
outperform other modes of context management such as redirector mode, and
database mode, and cache all policy by up to 60% in cost efficiency.
