LSTM's for change point detection

In this post, we revisit LSTM's. LSTM's are a form of RNN's with both a hidden state and a cell state that is recurrent through time. This solves mainly the vanishing gradient problem that original RNN's suffered from. These two posts use LSTM's for basic tasks, like change point detection and integer addition. With these models, we gain intuition for later use in larger environments and complexer datasets.

Classification, annotation and bidirectional

The project consists of three scripts

LSTM for classification. The lstm_cpd_main.py implements a binary classification task. The 0-class is without change point. The 1-class contains a change point. The first implementation classifies between these.
More interesting is to learn the moment of change point. lstm_cpd_main_ann.py does exactly that.
Our data is a dummy dataset and relatively easy. More complicated cases require information from past and future. Therefore, lstm_cpd_main_bidirec.py relies on a bidirectional LSTM.

Annotation

In time-series, classification and annotation are problems of different sorts. In classification, there's two labels for a time-series, being the 0-class of 1-class. (or any number of classes). In annotation, we are concerned with the exact location of some event. For our case, we are concerned with the onset of the change.

A probability distribution over the time-series captures the onset of the change point. The LSTM has two heads. The first head outputs a probability for the classification. A real number that is squashed by the sigmoid. The second head outputs a probability of the annotation happening at that location. For a time-series with T time-steps, imagine a probability mass function of T bins, where each bin represents the probability of the annotation. example The green line represents this distribution. The LSTM outputs any real number. A softmax layer transforms this to the distribution and can be trained by cross-entropy.

Bidirectional LSTM

Annotation requires also future information in the case of more complicated task. In NLP, for example, the model requires future words to determine the meaning of the current word. In ECG analysis, the model requires future electrograms to determine the nature of the current wave. Like for forward model is an LSTM with a hidden state, we can also imagine another LSTM running backward through time with hidden states. At one particular moment, the forward hidden state and backward hidden state together capture the meaning of the current time-step. In practise, the output layer uses the concatenated vectors of the forward and backward LSTM.

As always, I am curious to any comments and questions. Reach me at romijndersrob@gmail.com