Other performance measures that were considered are: the time taken to train the AI system, the time taken to execute the AI system, and the complexity of the model produced by the AI method. 

## C. **Neural Networks Using Standard Approaches**

The neural networks were implemented using the open source NETLAB Toolbox by Ian Nabney. Both the MLP and RBF neural networks were implemented using the standard approaches with this toolbox. The toolbox only constructs a 2 layer feed-forward network for both the MLP and RBF. Therefore, there is only one hidden layer, and only the number of hidden nodes needed to be determined. 

The initialisation of the MLP and RBF networks involves determining the activation functions used and the size of the hidden layer. A linear output activation function is best for regression problems, therefore, it was utilised for both the MLP and RBF. In the case of an MLP, a linear output activation function does not saturate, and as a result can extrapolate a little beyond the training dataset [2]. However, the hidden nodes can saturate which is one reason the inputs and outputs were scaled. In NETLAB, the hidden nodes of the MLP are the hyperbolic tangent. The hidden nodes of the RBF used the Gaussian function as seen in Equation 4. 

The generalisation ability of a network is determined mainly by the correct model complexity and the number of training cycles. There are a number of methods to improve the generalisation ability of a network such as determining the model complexity, early stopping, and regularisation. 

- **Model complexity is represented by the number of** 
hidden nodes in the network, as the hidden nodes are responsible for the number of adjustable parameters available in the network [3]. Therefore, a more complex model has a greater number of hidden nodes. However, if there are too many hidden nodes the system will be unnecessarily complex and prone to modelling the system's data too well (over-fitted). Conversely if there are too few hidden nodes the network will not be able to adequately model the system (under-fitted). One way to determine the optimal number of hidden nodes is to train the neural network with different numbers of hidden nodes, and observe the training and validation errors obtained. Note that a large number of hidden nodes will slow the training process. 

- **The Early Stopping technique uses a training as well** 
as a validation dataset. The main idea behind early stopping is that the training error of the network will gradually decrease as the number of training cycles increases. The degree the network is over-trained is measured using the validation dataset as the validation error will decrease at first and then start to increase as the network is over-trained [3]. Therefore, training should be stopped at the point before the validation error begins to increase. 

- **Regularisation techniques encourage weights that** 
produce smoother network mappings. An over-fitted network models the training data almost exactly, resulting in the mapping produced by the network having areas of large curvature [3]. This results in large weights. The simplest regularisation technique uses a weight-decay where mappings with large weights are penalised [3]. Regularisation prevents over-training. 

The number of inputs and outputs are determined by the problem, and as stated there are 4 inputs and 4 outputs in the system being modelled. Determining the number of hidden nodes used is an iterative and experimental procedure, as it is dependent on the complexity of the relationships in the dataset. A rough estimate for the number of hidden nodes is to take half the sum of the total number of inputs and outputs [2]. Therefore, 4 hidden nodes were used as a starting point and progressively increased while monitoring the training process, to determine an approximate number of hidden nodes. An alternative approach would be to start with a network with a large number of hidden units and prune it iteratively to find a network which will adequately and accurately model the data. 

The approach that was taken was to train a network with a fixed number of hidden nodes, periodically stopping the training process to determine the error on the validation dataset. Therefore, the training and validation errors during the training process could be observed, and an indication of the generalisation ability of the network determined. This was done for a varying number of hidden nodes (4 - 20 for the MLP and 4 -50 for RBF). For each number of hidden nodes, a number of networks were trained, as the optimisation techniques used will result in a different solution each time. 

Therefore, the different solutions for the set number of hidden nodes could be compared and the best network selected. Also, a number of networks with large numbers of hidden nodes were trained to see what the effect was on the resulting network (60, 70 and 200). This procedure was done, in order to determine an optimum number of hidden nodes and training cycles that could adequately and accurately model the system. The goal of this procedure was to find a network that was powerful enough to adequately model the system and generalise well, while being easily trained. 

Using the procedure discussed above, it was found that for the MLP the most appropriate number of hidden nodes and training cycles were 8 and 240 respectively. During the experiments carried out on the MLPs, a few observations were made and are discussed below. 

It was noticed that increasing the number of hidden nodes beyond 8 didn't seem to increase the accuracy by a significant amount to justify utilising a more complex network. The 