RBFN for classification on MNIST

A Radial basis function network

In this blog-post, we will discuss radial basis function networks(RBFN). RBNF's seem to have lost the race for popularity as compared to Neural Network. Yet, they prove a powerful generalization capacity. RBFN's are similar to NN in the sense that they use multiple layers of perceptrons to define a function over the input space. Yet, in an RBFN, the first layer generalizes the input space. That is, the input layer makes use of the distances between fixed center points and input data. The consequent activation functions are radially symmetric. Hence the name, radial basis function networks.

Basic implementation

This code is a mere proof of concept We explore the world of RBFN with a basic implementation. The dataset are two selected classes from the MNIST dataset. We implement both exact interpolation and approximate interpolation:

• In exact interpolation, we have as much hidden neurons in the firsy layer as we have train samples. Moreover, we pick the center points of the hidden neurons to be exactly these train samples. Obviously, we are overfitting in many aspects.
• In approximate interpolation, we pick less hidden neurons. This is a hyperparameter. Still, the center points of the hidden neurons are picked to be train samples.

Evaluation

RBFN's have no deterministic threshold, even when the targets are 0 and 1. Therefore, we evaluate the accuracies over a range of possible thresholds:

Also interesting is to vary the regularization on the final perceptron or the variance of the radial basis function:

Further improvements

This implementation of an RBFN is basic. Anyone who would like to take this further can consider

• Optimize the variance of the RBF. There are many articles and bookchapters on optimizing the variance of the RBF. Many implementations treat the variance as an input parameter and back-propagate into it.
• Back-propagate to the centerpoints. We took the location of the center points as random train samples. Another possibility is to treat the center point as an input parameter as well and back-propagate into it.
• Add an new hidden layer to the RBFN. Learning with back-propagation also allows you to add hidden layers to your liking. Now the radial basis neurons improve the generalization and the multiple perceptrons improve the versatility. Together they will provide a powerful model.

The code is on the associated GitHub repository. RBFN_two_MNIST_main.m is the file to start from.

As always, I am curious to any comments and questions. Reach me at romijndersrob@gmail.com