# Reinforcement Learning Based On Active Learning Method

Hesam Sagha1, Saeed Bagheri Shouraki2, Hosein Khasteh1, Ali Akbar Kiaei1 1*ACECR, Nasir Branch, Tehran, Iran* 2Department of Electrical Engineering Sharif University of Technology,Tehran, Iran sagha@ce.sharif.ir, bagheri-s@sharif.edu, h_khasteh@ce.sharif.ir, kiaei@ce.sharif.edu 

## Abstract

In this paper, a new reinforcement learning approach is proposed which is based on a powerful concept named Active Learning Method (ALM) in modeling. ALM expresses any multi-input-single-output system as a fuzzy combination of some single-input-singleoutput systems. The proposed method is an actor-critic system similar to Generalized Approximate Reasoning based Intelligent Control (GARIC) structure to adapt the ALM by delayed reinforcement signals. Our system uses Temporal Difference (TD) learning to model the behavior of useful actions of a control system. The goodness of an action is modeled on Reward-Penalty-Plane. IDS planes will be updated according to this plane. It is shown that the system can learn with a predefined fuzzy system or without it (through random actions).

## 1. **Introduction**

ALM [1,2,3,4] is a recursive fuzzy algorithm, which expresses a multi-input-single-output system as a fuzzy combination of several single-input-single-output systems. It models the input-output relations for each input and combines these models to find out the overall system model. ALM starts with gathering data and projecting them on different data planes. The horizontal axis of each data plane is one of the inputs and the vertical axis is the output. IDS processing engine will look for a behavior curve, hereafter narrow line, on each data plane. If the spread of data over the narrow line is more than a threshold, data domains will be divided and the algorithm runs again. The heart of this learning algorithm is a fuzzy interpolation method which is used to derive a smooth curve among data points. It is done by applying a three-dimensional membership function to each data point, which expresses the belief for the data point and its neighbors. Each data point is considered as a source of light, which has a pyramid-shape illumination pattern. As the vertical distance from this source of light increases, its illuminating pattern will interfere with its neighbors forming new bright areas. The projection of the process on the plane is called IDS. 

As it is shown in Fig 2, we can use a pyramid as a three dimensional fuzzy membership function of a data point and its neighboring points. By applying IDS method to each data plane, two different types of information will be extracted. One is the narrow path and the other is the deviation of the data points around each narrow path. Each narrow path shows the behavior of output relative to an input; and spread of the data points around this path shows the importance degree of that input in overall system behavior. Less deviation of data points around the path represents a higher degree of importance and vice versa. 

Sagha *et al* [5] proposed a method which combines genetic algorithm and IDS to obtain better partitions over input variables. Their method is called GIDS (Genetic IDS). 

Shahdi *et al* [6] proposed *RIDS* method that replaces each two consequent points with their midpoint instead of applying a 2-d fuzzy membership function on each data. *RIDS* converges to the center of gravity of data and increases the number of points in order to keep data expansion in plane. In addition, they proposed another method called (Modified RIDS) *MRIDS* that support negative data points. In *MRIDS,* if two consequent points are positive, the result is similar to that of *RIDS* and the replacing point is their midpoint. Nevertheless, if one of the points is negative, then the replacing point is a point located near the positive point on the line which connects two points; so negative point has an effect of deviating center of gravity from positive points. 

MRIDS considers that the rewards and punishments are accessible after each action, but when they are delayed and this delay is not determined, it will not converge correctly. 

Here we used another method called Reinforcement ALM (*RALM*), to add reinforcement capability to the algorithm. We used the concepts of Action Selection 