MIST101 Workshop 4: Recurrent Neural Networks(RNN) was held this Thursday, October 26th. Having talked about Supervised Learnings and Convolutional Neural Network(CNN), this workshop covered RNN’s basic modeling principles, variants and various applications.
RNN Models and Training Methods
We first defined RNN as a classical neural network where connection units form a directed cycle. The proposed model takes in a sequence of data inputs and produce a sequence of predicted outputs as feedback information to the model itself. Computations are done in a series of hidden cells chaining together and sharing the same parameters.
To train a RNN, we showed the unfolded diagram of RNN. The steps involve defining a proper loss function that represents the task on hand and using the standard method backpropagation through time(BPTT) to perform gradient descent to minimize loss across time steps. An example of generating a sentence was introduced.
We then briefly introduced the variants of RNN, including Bidirectional RNN, stacked RNN and the Encoder-Decoder Model. A commonly used architecture called Long Short-Term Memory was covered as well. LSTM is augmented by gating mechanisms to prevent back propagated errors from vanishing or exploding, hence effectively capturing long term dependency from the sequential data.
RNN excels in machine translation and language modeling, which are commonly used by search engines. As mentioned in the previous workshops, RNN can as well be combined with CNNs to achieve image captioning.
Hands-on TensorFlow Tutorial
The practical session for workshop #4 was about an example of modeling a sine curve. Students were given historical data of the process(Training Dataset) and asked to predict the price for the next K days. You can access the tutorial here: https://github.com/ColinQiyangLi/MIST101/blob/master/Tensorflow_RNN%20(Part%201).ipynb
If you would like to learn more about workshop #4, the slides can be found under this link: https://docs.google.com/presentation/d/1zXyLmsSHlwy8KQpYvrw5WquxFKAciw0BU5QYktKO4kQ/edit?usp=sharing