Recurrent Neural Network Tensorflow | Introduction

This tutorial demonstrates how to generate text using a character-based RNN. You will work with a dataset of Shakespeare’s writing from Andrej Karpathy’s The Unreasonable Effectiveness of Recurrent Neural Networks. Given a sequence of characters from this data (“Shakespear”), train a model to predict the next character in the sequence (“e”). Longer sequences of text can be generated by calling the model repeatedly.

This tutorial includes runnable code implemented using tf.keras and eager execution. The following is the sample output when the model in this tutorial trained for 30 epochs, and started with the prompt “Q”:

QUEENE: I had thought thou hadst a Roman; for the oracle, Thus by All bids the man against the word, Which are so weak of care, by old care done; Your children were in your holy love, And the precipitation through the bleeding throne. BISHOP OF ELY: Marry, and will, my lord, to weep in such a one were prettiest; Yet now I was adopted heir Of the world’s lamentable day, To watch the next way with his father with his face? ESCALUS: The cause why then we are all resolved more sons. VOLUMNIA: O, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, no, it is no sin it should be dead, And love and pale as any will to that word. QUEEN ELIZABETH: But how long have I heard the soul for this world, And show his hands of life be proved to stand. PETRUCHIO: I say he look’d on, if I must be content To stay him from the fatal of our country’s bliss. His lordship pluck’d from this sentence then for prey, And then let us twain, being the moon, were she such a case as fills m

While some of the sentences are grammatical, most do not make sense. The model has not learned the meaning of words, but consider:

• The model is character-based. When training started, the model did not know how to spell an English word, or that words were even a unit of text.

• The structure of the output resembles a play—blocks of text generally begin with a speaker name, in all capital letters similar to the dataset.

• As demonstrated below, the model is trained on small batches of text (100 characters each), and is still able to generate a longer sequence of text with coherent structure.

Why Recurrent Neural Networks?

Recurrent Neural Networks have unique capacities as opposed to other kinds of Neural Networks, which open a wide range of possibilities for their users still also bringing some challenges with them. Then’s a rundown of the main benefits

• It’s the only neural network with memory and binary data processing.
• It can plan out several inputs and productions. Unlike other algorithms that deliver one product for one input, the benefit of RNN is that it can plot out many to many, one to many, and many to one input and productions.

Export the generator

This single-step model can easily be saved and restored, allowing you to use it anywhere a

tf.saved_model

is accepted.


tf.saved_model.save(one_step_model, 'one_step') one_step_reloaded = tf.saved_model.load('one_step')


WARNING:tensorflow:Skipping full serialization of Keras layer <__main__.OneStep object at 0x7f1a9c2e6880>, because it is not built. WARNING:tensorflow:Model’s `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config. WARNING:tensorflow:Model’s `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config. INFO:tensorflow:Assets written to: one_step/assets INFO:tensorflow:Assets written to: one_step/assets


states = None next_char = tf.constant(['ROMEO:']) result = [next_char] for n in range(100): next_char, states = one_step_reloaded.generate_one_step(next_char, states=states) result.append(next_char) print(tf.strings.join(result)[0].numpy().decode("utf-8"))


ROMEO: While shall we toward them, gaunt and married. Urped me say I doubt not, for this world is gentle,

A Caveat: Masking and Padding

Padding and masking are very useful techniques that will optimize our training. But what exactly is its function, and why do we need it?

Imagine you have a corpus of text. The sentences within this corpus are of different lengths. This means that every sentence has to be standardized to a common length.

Practice makes perfectPerfection is a dream worth chasing

Becomes …

Practice makes perfect [pad] [pad] [pad] [pad] Perfection is a dream worth chasing [pad]

Now these

[pad]

tokens are of no use in the model except for standardizing the length. So we mask these tokens while training. The process of masking and padding is demonstrated in Figure 5.

Modeling Sequential Data with MLPs

But before we dive into modeling sequential data with Recurrent Neural Networks, let us first understand the shortcomings of the models we would have generally used.

What if we tried to model sequential data with Multilayer Perceptrons (MLPs)?

The answer lies in the question. We create a sequence of our data and pass it through the MLP. The MLP starts learning the data, but something is missing. The process is demonstrated in Figure 6.

Can you tell what is missing in Figure 6?

The MLP learns individual data points. Each time a data point is passed through the MLP, the color of the entire network changes. This signifies that the MLP models that particular data point. So far, so good.

But there is something else here: is succeeded by and then by and so on. This makes the data sequential. While the MLP can learn the individual data points perfectly, it will not be able to learn the order of the data.

And therein lies the problem.

The Recurrence Formula

To solve this problem, we need a function that takes care of the current and previous states.

is the current input.
is the current output and
is the past output. The function that models this relation is represented by
.

Let’s unfold this equation and understand its importance.

• At
• At

It is quite evident from the above examples that the equation can model the previous states along with the current state. This is an extremely important equation we will use throughout our tutorial.

In the case of Recurrent Neural Networks the function is a simple that introduces nonlinearity. If you need a quick brush-up on the tanh function, feel free to play around with the interactive graph below:

The recurrence formula for Recurrent Neural Network can be represented as:

is called the hidden state of the network. Notice how the current hidden state is a function of the current input
and the previous hidden state
. This is demonstrated in Figure 7.

Recurrent Neural Network (an overview)

Now that we know the formula behind Recurrent Neural Networks, let us see how to build one. The diagram of a Recurrent Neural Network is shown in Figure 8.

An RNN cell consists of three parts:

1. The input or
2. The output or
3. And the hidden state

The hidden state is fed back as input to the next state and the input for that state. Fortunately, this is implemented using the

SimpleRNN

function inside the

tf.keras.layers

API.

The relation between the input, output, and the hidden state is demonstrated in the following. Figure 9 shows an unfolded RNN.

And now, we begin to build our RNN model. Let’s go through the

model.py

file.

# import the necessary packages from tensorflow.keras import layers from tensorflow import keras

We begin with the necessary imports on Lines 2-3.

def get_rnn_model(vocabSize): # input for variable-length sequences of integers inputs = keras.Input(shape=(None,), dtype=”int32″) # embed the tokens in a 128-dimensional vector with masking # applied and apply dropout x = layers.Embedding(vocabSize, 128, mask_zero=True)(inputs) x = layers.Dropout(0.2)(x) # add 3 simple RNNs x = layers.SimpleRNN(64, return_sequences=True)(x) x = layers.SimpleRNN(64, return_sequences=True)(x) x = layers.SimpleRNN(64)(x) # add a classifier head x = layers.Dense(units=64, activation=”relu”)(x) x = layers.Dense(units=32, activation=”relu”)(x) x = layers.Dropout(0.2)(x) outputs = layers.Dense(1, activation=”sigmoid”)(x) # build the RNN model model = keras.Model(inputs, outputs, name=”RNN”) # return the RNN model return model

On Line 5, we define the

get_rnn_model

we use to create the model. On Line 7, we create the input layer for a variable length sequence of input. The padding and masking mechanism is taken care of by our

Embedding

layer, which we initialize in Line 11. The

Embedding

layer also embeds the variable-length sequence into a

128

-dimensional vector. For those unclear, consider this 128-dimensional matrix as a way the computers view and assign meaning to the text.

On Line 12, we apply dropout to the inputs. We add 3 simple RNN cells on Lines 15-17 and a classifier head with a sigmoid activation on Lines 20-23.

Finally, the model is built using keras.Model API on Line 26 and returned on Line 29.

Training and Visualizations

With the model created and ready, we can finally begin our training procedure. But before that, we must first look at some helper functions that will assist with visualizations and saving.

We begin with a function inside

plot.py

that will help us plot the loss and accuracy.

# import the necessary packages import matplotlib.pyplot as plt def plot_loss_accuracy(history, filepath): # plot the training and validation loss plt.style.use(“ggplot”) (fig, axs) = plt.subplots(2, 1) axs[0].plot(history[“loss”], label=”train_loss”) axs[0].plot(history[“val_loss”], label=”val_loss”) axs[0].set_xlabel(“Epoch #”) axs[0].set_ylabel(“Loss”) axs[0].legend() axs[1].plot(history[“accuracy”], label=”train_accuracy”) axs[1].plot(history[“val_accuracy”], label=”val_accuracy”) axs[1].set_xlabel(“Epoch #”) axs[1].set_ylabel(“Accuracy”) axs[1].legend() fig.savefig(filepath)

On Line 2, we begin by importing matplotlib. On Line 4, we define the

plot_loss_accuracy

function that takes the model

history

and

filepath

as input.

Next, on Lines 6-17, we plot the loss and accuracy for training and validation and save the figure in the specified

filepath

(Line 18).

Next, we define another helper function called

save_load.py

to save the adapted vectorization layer for later use.

from tensorflow.keras.layers import TextVectorization import tensorflow as tf import pickle def save_vectorizer(vectorizer, name): # pickle the weights of the vectorization layer pickle.dump({“weights”: vectorizer.get_weights()}, open(f”{name}.pkl”, “wb”))

We begin with the necessary imports on Lines 1-3. Next, we define a function called

save_vectorizer

on Lines 5-8 that pickles and saves the weight of the vectorization layer.

And finally, with all the necessary functions defined, we can start with

train.py

, where we actually train our RNN model.

# USAGE # python train.py # set the seed for reproducibility import tensorflow as tf tf.keras.utils.set_random_seed(42) # import the necessary packages from pyimagesearch.standardization import custom_standardization from pyimagesearch.plot import plot_loss_accuracy from pyimagesearch.save_load import save_vectorizer from pyimagesearch.dataset import get_imdb_dataset from pyimagesearch.model import get_rnn_model from pyimagesearch.model import get_lstm_model from pyimagesearch import config from tensorflow.keras import layers from tensorflow import keras import os

We begin with all the necessary imports on Lines 5-18.

# get the IMDB dataset print(“[INFO] getting the IMDB dataset…”) (trainDs, valDs) = get_imdb_dataset(folderName=config.DATASET_PATH, batchSize=config.BATCH_SIZE, bufferSize=config.BUFFER_SIZE, autotune=tf.data.AUTOTUNE, test=False) # initialize the text vectorization layer vectorizeLayer = layers.TextVectorization( max_tokens=config.VOCAB_SIZE, output_mode=”int”, output_sequence_length=config.MAX_SEQUENCE_LENGTH, standardize=custom_standardization, ) # grab the text from the training dataset and adapt the text # vectorization layer on it trainText = trainDs.map(lambda text, label: text) vectorizeLayer.adapt(trainText) # vectorize the training and the validation dataset trainDs = trainDs.map(lambda text, label: (vectorizeLayer(text), label)) valDs = valDs.map(lambda text, label: (vectorizeLayer(text), label)) # get the RNN model and compile it print(“[INFO] building the RNN model…”) modelRNN = get_rnn_model(vocabSize=config.VOCAB_SIZE) modelRNN.compile(metrics=[“accuracy”], optimizer=keras.optimizers.Adam(learning_rate=config.LR), loss=keras.losses.BinaryCrossentropy(from_logits=False), ) # train the RNN model print(“[INFO] training the RNN model…”) historyRNN = modelRNN.fit(trainDs, epochs=config.EPOCHS, validation_data=valDs, )

Next, on Lines 22-24, we get the IMDb dataset for movie reviews.

On Lines 27-32, we initialize the text vectorization layer with:

• 
max_tokens

  : The maximum number of tokens inside the vocabulary.
• 
output_mode

  : The data type for the output.
• 
output_sequence_length

  : The maximum sequence length that we will need for padding and masking.
• 
standardize

  : The custom standardization function that we defined previously.

On Lines 36 and 37, we adapt the vectorization layer on the training dataset.

When a text vectorization layer is initialized, it does not hold any information about the vocabulary of the training corpus. To build a vocabulary, we need to pass the entire training dataset through the vectorization layer. This is known as adapting to the text.

Finally, on Lines 40 and 41, we vectorize the text of the training and validation dataset using the adapted vectorization layer.

On Lines 44-49, we initialize the predefined RNN model and compile it with Adam optimizer and Binary Cross-Entropy loss.

On Lines 52-55, we fit the model on the training data and save its history onto the

historyRNN

variable.

# check whether the output folder exists, if not build the output folder if not os.path.exists(config.OUTPUT_PATH): os.makedirs(config.OUTPUT_PATH) # save the loss and accuracy plots of RNN and LSTM models plot_loss_accuracy(history=historyRNN.history, filepath=config.RNN_PLOT) plot_loss_accuracy(history=historyLSTM.history, filepath=config.LSTM_PLOT)

On Lines 72 and 73, we check whether the output path exists or create it otherwise. The

plot_loss_accuracy

function is called on Line 76 to visualize the loss and accuracy of the RNN model.

# save the trained RNN and LSTM models to disk print(f”[INFO] saving the RNN model to {config.RNN_MODEL_PATH}…”) keras.models.save_model(model=modelRNN, filepath=config.RNN_MODEL_PATH, include_optimizer=False) print(f”[INFO] saving the LSTM model to {config.LSTM_MODEL_PATH}…”) keras.models.save_model(model=modelLSTM, filepath=config.LSTM_MODEL_PATH, include_optimizer=False)

On Lines 80-82, we then save the trained RNN model on our disk.

# save the text vectorization layer to disk save_vectorizer(vectorizer=vectorizeLayer, name=config.TEXT_VEC_PATH)

Finally, on Line 88, we save the text vectorization layer we used to vectorize the training and validation data.

We can verify the model output’s training, validation loss, and accuracy. We have a 67.88% validation accuracy in just 10 epochs!

$ python train.py [INFO] getting the IMDB dataset… [INFO] building the RNN model… [INFO] training the RNN model… Epoch 1/10 22/22 [==============================] – 11s 329ms/step – loss: 0.7018 – accuracy: 0.4971 – val_loss: 0.6981 – val_accuracy: 0.4820 Epoch 2/10 22/22 [==============================] – 7s 320ms/step – loss: 0.6935 – accuracy: 0.5152 – val_loss: 0.6967 – val_accuracy: 0.4916 Epoch 3/10 22/22 [==============================] – 7s 293ms/step – loss: 0.6883 – accuracy: 0.5405 – val_loss: 0.6959 – val_accuracy: 0.5000 Epoch 4/10 22/22 [==============================] – 7s 307ms/step – loss: 0.6850 – accuracy: 0.5509 – val_loss: 0.6952 – val_accuracy: 0.5064 Epoch 5/10 22/22 [==============================] – 7s 303ms/step – loss: 0.6802 – accuracy: 0.5673 – val_loss: 0.6950 – val_accuracy: 0.5100 Epoch 6/10 22/22 [==============================] – 7s 302ms/step – loss: 0.6729 – accuracy: 0.5915 – val_loss: 0.6953 – val_accuracy: 0.5136 Epoch 7/10 22/22 [==============================] – 7s 294ms/step – loss: 0.6650 – accuracy: 0.6094 – val_loss: 0.6943 – val_accuracy: 0.5232 Epoch 8/10 22/22 [==============================] – 7s 303ms/step – loss: 0.6493 – accuracy: 0.6402 – val_loss: 0.6812 – val_accuracy: 0.5668 Epoch 9/10 22/22 [==============================] – 7s 294ms/step – loss: 0.6141 – accuracy: 0.6774 – val_loss: 0.6379 – val_accuracy: 0.6380 Epoch 10/10 22/22 [==============================] – 7s 296ms/step – loss: 0.5501 – accuracy: 0.7335 – val_loss: 0.5945 – val_accuracy: 0.6788

Loading and Inference

Now that the model has been trained and saved to disk, we need to perform inference to understand the model performance. Before starting with inference, let us first open

save_load.py

again and look at the

load_vectorizer

function.

def load_vectorizer(name, maxTokens, outputLength, standardize=None): # load the pickles data fromDisk = pickle.load(open(f”{name}.pkl”, “rb”)) # build a new vectorization layer newVectorizer = TextVectorization(max_tokens=maxTokens, output_mode=”int”, output_sequence_length=outputLength, standardize=standardize) # call the adap method with some dummy data for the vectorization # layer to initialize properly newVectorizer.adapt(tf.data.Dataset.from_tensor_slices([“xyz”])) newVectorizer.set_weights(fromDisk[“weights”]) # return the vectorization layer return newVectorizer

We define the

load_vectorizer

function on Line 10, which takes the following input:

• 
name

  : the file path of the saved
  
TextVectorization

  layer weights
• 
maxTokens

  : the maximum number of tokens in the vocabulary
• 
outputLength

  : the length of the output sequence
• 
standardize

  : we do not need any standardization function here

In the training pipeline, we build a

TextVectorization

layer to tokenize and vectorize our training data. In the inference pipeline, we need the same TextVectorization layer as the training pipeline.

To do this, we save the weights of the adapted

TextVectorization

layer and load the weights on top of a newly initialized layer. On Line 12, we load the weights of the saved

TextVectorization

layer. On Line 25, we return the loaded vectorizer.

# USAGE # python inference.py # import the necessary packages from pyimagesearch.standardization import custom_standardization from pyimagesearch.save_load import load_vectorizer from pyimagesearch.dataset import get_imdb_dataset from pyimagesearch import config from tensorflow import keras import tensorflow as tf # load the pre-trained RNN and LSTM model print(“[INFO] loading the pre-trained RNN model…”) modelRnn = keras.models.load_model(filepath=config.RNN_MODEL_PATH) modelRnn.compile(optimizer=”adam”, metrics=[“accuracy”], loss=keras.losses.BinaryCrossentropy(from_logits=False), ) print(“[INFO] loading the pre-trained LSTM model…”) modelLstm = keras.models.load_model(filepath=config.LSTM_MODEL_PATH) modelLstm.compile(optimizer=”adam”, metrics=[“accuracy”], loss=keras.losses.BinaryCrossentropy(from_logits=False), )

On Lines 5-10, we import the necessary packages. On Line 14, the saved RNN model is loaded back to disk. The loaded model is then compiled with the suitable metrics, loss, and optimizer on Lines 15-17.

# get the IMDB dataset print(“[INFO] getting the IMDB test dataset…”) testDs = get_imdb_dataset(folderName=config.DATASET_PATH, batchSize=config.BATCH_SIZE, bufferSize=config.BUFFER_SIZE, autotune=tf.data.AUTOTUNE, test=True) # load the pre-trained text vectorization layer vectorizeLayer = load_vectorizer(name=config.TEXT_VEC_PATH, maxTokens=config.VOCAB_SIZE, outputLength=config.MAX_SEQUENCE_LENGTH, standardize=custom_standardization) # vectorize the test dataset testDs = testDs.map(lambda text, label: (vectorizeLayer(text), label)) # evaluate the trained RNN and LSTM model on the test dataset for model in [modelRnn, modelLstm]: print(f”[INFO] test evaluation for {model.name}:”) (testLoss, testAccuracy) = model.evaluate(testDs) print(f”\t[INFO] test loss: {testLoss:0.2f}”) print(f”\t[INFO] test accuracy: {testAccuracy * 100:0.2f}%”)

On Lines 26-28, we get the testing dataset and pre-process it. Line 37 will map the dataset to get the vectorized tokens and labels.

On Lines 40-44, we evaluate the testing accuracy and the testing loss of the RNN model. Our model achieves 68.42% accuracy at inference!

$ python inference.py [INFO] loading the pre-trained RNN model… [INFO] loading the pre-trained LSTM model… [INFO] getting the IMDB test dataset… [INFO] test evaluation for RNN: 25/25 [==============================] – 4s 96ms/step – loss: 0.6035 – accuracy: 0.6842 [INFO] test loss: 0.60 [INFO] test accuracy: 68.42%

What’s next? We recommend PyImageSearch University.

84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024

★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

I strongly believe that if you had the right teacher you could master computer vision and deep learning.

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s not the case.

All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you’re serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.

Inside PyImageSearch University you’ll find:

• ✓ 84 courses on essential computer vision, deep learning, and OpenCV topics
• ✓ 84 Certificates of Completion
• ✓ 114+ hours of on-demand video
• ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
• ✓ Pre-configured Jupyter Notebooks in Google Colab
• ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
• ✓ Access to centralized code repos for all 536+ tutorials on PyImageSearch
• ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
• ✓ Access on mobile, laptop, desktop, etc.

Cross-batch statefulness

When processing very long sequences (possibly infinite), you may want to use the pattern of cross-batch statefulness.

Normally, the internal state of a RNN layer is reset every time it sees a new batch (i.e. every sample seen by the layer is assumed to be independent of the past). The layer will only maintain a state while processing a given sample.

If you have very long sequences though, it is useful to break them into shorter sequences, and to feed these shorter sequences sequentially into a RNN layer without resetting the layer’s state. That way, the layer can retain information about the entirety of the sequence, even though it’s only seeing one sub-sequence at a time.

You can do this by setting

stateful=True

in the constructor.

If you have a sequence

s = [t0, t1, ... t1546, t1547]

, you would split it into e.g.


s1 = [t0, t1, ... t100] s2 = [t101, ... t201] ... s16 = [t1501, ... t1547]


Then you would process it via:


lstm_layer = layers.LSTM(64, stateful=True) for s in sub_sequences: output = lstm_layer(s)


When you want to clear the state, you can use

layer.reset_states()

.

Here is a complete example:


paragraph1 = np.random.random((20, 10, 50)).astype(np.float32) paragraph2 = np.random.random((20, 10, 50)).astype(np.float32) paragraph3 = np.random.random((20, 10, 50)).astype(np.float32) lstm_layer = layers.LSTM(64, stateful=True) output = lstm_layer(paragraph1) output = lstm_layer(paragraph2) output = lstm_layer(paragraph3) # reset_states() will reset the cached state to the original initial_state. # If no initial_state was provided, zero-states will be used by default. lstm_layer.reset_states()


RNN State Reuse

The recorded states of the RNN layer are not included in the

layer.weights()

. If you
would like to reuse the state from a RNN layer, you can retrieve the states value by

layer.states

and use it as the
initial state for a new layer via the Keras functional API like

new_layer(inputs,
initial_state=layer.states)

, or model subclassing.

Please also note that sequential model might not be used in this case since it only supports layers with single input and output, the extra input of initial state makes it impossible to use here.


paragraph1 = np.random.random((20, 10, 50)).astype(np.float32) paragraph2 = np.random.random((20, 10, 50)).astype(np.float32) paragraph3 = np.random.random((20, 10, 50)).astype(np.float32) lstm_layer = layers.LSTM(64, stateful=True) output = lstm_layer(paragraph1) output = lstm_layer(paragraph2) existing_state = lstm_layer.states new_lstm_layer = layers.LSTM(64) new_output = new_lstm_layer(paragraph3, initial_state=existing_state)


Built-in RNN layers: a simple example

There are three built-in RNN layers in Keras:

1. 
keras.layers.SimpleRNN

   , a fully-connected RNN where the output from previous timestep is to be fed to next timestep.
2. 
keras.layers.GRU

   , first proposed in Cho et al., 2014.
3. 
keras.layers.LSTM

   , first proposed in Hochreiter & Schmidhuber, 1997.

In early 2015, Keras had the first reusable open-source Python implementations of LSTM and GRU.

Here is a simple example of a

Sequential

model that processes sequences of integers,
embeds each integer into a 64-dimensional vector, then processes the sequence of
vectors using a

LSTM

layer.


model = keras.Sequential() # Add an Embedding layer expecting input vocab of size 1000, and # output embedding dimension of size 64. model.add(layers.Embedding(input_dim=1000, output_dim=64)) # Add a LSTM layer with 128 internal units. model.add(layers.LSTM(128)) # Add a Dense layer with 10 units. model.add(layers.Dense(10)) model.summary()


Model: “sequential” _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (None, None, 64) 64000 lstm (LSTM) (None, 128) 98816 dense (Dense) (None, 10) 1290 ================================================================= Total params: 164106 (641.04 KB) Trainable params: 164106 (641.04 KB) Non-trainable params: 0 (0.00 Byte) _________________________________________________________________

Built-in RNNs support a number of useful features:

• Recurrent dropout, via the
  
dropout

  and
  
recurrent_dropout

  arguments
• Ability to process an input sequence in reverse, via the
  
go_backwards

  argument
• Loop unrolling (which can lead to a large speedup when processing short sequences on
  CPU), via the
  
unroll

  argument
• …and more.

For more information, see the RNN API documentation.

Performance optimization and CuDNN kernels

In TensorFlow 2.0, the built-in LSTM and GRU layers have been updated to leverage CuDNN
kernels by default when a GPU is available. With this change, the prior

keras.layers.CuDNNLSTM/CuDNNGRU

layers have been deprecated, and you can build your
model without worrying about the hardware it will run on.

Since the CuDNN kernel is built with certain assumptions, this means the layer will not be able to use the CuDNN kernel if you change the defaults of the built-in LSTM or GRU layers. E.g.:

• Changing the
  
activation

  function from
  
tanh

  to something else.
• Changing the
  
recurrent_activation

  function from
  
sigmoid

  to something else.
• Using
  
recurrent_dropout

  > 0.
• Setting
  
unroll

  to True, which forces LSTM/GRU to decompose the inner
  
tf.while_loop

  into an unrolled
  
for

  loop.
• Setting
  
use_bias

  to False.
• Using masking when the input data is not strictly right padded (if the mask corresponds to strictly right padded data, CuDNN can still be used. This is the most common case).

For the detailed list of constraints, please see the documentation for the LSTM and GRU layers.

Using CuDNN kernels when available

Let’s build a simple LSTM model to demonstrate the performance difference.

We’ll use as input sequences the sequence of rows of MNIST digits (treating each row of pixels as a timestep), and we’ll predict the digit’s label.


batch_size = 64 # Each MNIST image batch is a tensor of shape (batch_size, 28, 28). # Each input sequence will be of size (28, 28) (height is treated like time). input_dim = 28 units = 64 output_size = 10 # labels are from 0 to 9 # Build the RNN model def build_model(allow_cudnn_kernel=True): # CuDNN is only available at the layer level, and not at the cell level. # This means `LSTM(units)` will use the CuDNN kernel, # while RNN(LSTMCell(units)) will run on non-CuDNN kernel. if allow_cudnn_kernel: # The LSTM layer with default options uses CuDNN. lstm_layer = keras.layers.LSTM(units, input_shape=(None, input_dim)) else: # Wrapping a LSTMCell in a RNN layer will not use CuDNN. lstm_layer = keras.layers.RNN( keras.layers.LSTMCell(units), input_shape=(None, input_dim) ) model = keras.models.Sequential( [ lstm_layer, keras.layers.BatchNormalization(), keras.layers.Dense(output_size), ] ) return model


Let’s load the MNIST dataset:


mnist = keras.datasets.mnist (x_train, y_train), (x_test, y_test) = mnist.load_data() x_train, x_test = x_train / 255.0, x_test / 255.0 sample, sample_label = x_train[0], y_train[0]


Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz 11490434/11490434 [==============================] – 0s 0us/step

Let’s create a model instance and train it.

We choose

sparse_categorical_crossentropy

as the loss function for the model. The
output of the model has shape of

[batch_size, 10]

. The target for the model is an
integer vector, each of the integer is in the range of 0 to 9.


model = build_model(allow_cudnn_kernel=True) model.compile( loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), optimizer="sgd", metrics=["accuracy"], ) model.fit( x_train, y_train, validation_data=(x_test, y_test), batch_size=batch_size, epochs=1 )


WARNING: All log messages before absl::InitializeLog() is called are written to STDERR I0000 00:00:1700136618.250305 9824 device_compiler.h:186] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process. 938/938 [==============================] – 7s 5ms/step – loss: 0.9965 – accuracy: 0.6845 – val_loss: 0.5699 – val_accuracy: 0.8181

Now, let’s compare to a model that does not use the CuDNN kernel:


noncudnn_model = build_model(allow_cudnn_kernel=False) noncudnn_model.set_weights(model.get_weights()) noncudnn_model.compile( loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True), optimizer="sgd", metrics=["accuracy"], ) noncudnn_model.fit( x_train, y_train, validation_data=(x_test, y_test), batch_size=batch_size, epochs=1 )


938/938 [==============================] – 20s 20ms/step – loss: 0.4268 – accuracy: 0.8698 – val_loss: 0.3017 – val_accuracy: 0.9145

When running on a machine with a NVIDIA GPU and CuDNN installed, the model built with CuDNN is much faster to train compared to the model that uses the regular TensorFlow kernel.

The same CuDNN-enabled model can also be used to run inference in a CPU-only
environment. The

tf.device

annotation below is just forcing the device placement.
The model will run on CPU by default if no GPU is available.

You simply don’t have to worry about the hardware you’re running on anymore. Isn’t that pretty cool?


import matplotlib.pyplot as plt with tf.device("CPU:0"): cpu_model = build_model(allow_cudnn_kernel=True) cpu_model.set_weights(model.get_weights()) result = tf.argmax(cpu_model.predict_on_batch(tf.expand_dims(sample, 0)), axis=1) print( "Predicted result is: %s, target result is: %s" % (result.numpy(), sample_label) ) plt.imshow(sample, cmap=plt.get_cmap("gray"))


Predicted result is: [3], target result is: 5

Setup

Import TensorFlow and other libraries


import tensorflow as tf import numpy as np import os import time


2023-11-16 12:28:52.207051: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2023-11-16 12:28:52.207090: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2023-11-16 12:28:52.208630: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

Download the Shakespeare dataset

Change the following line to run this code on your own data.


path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt')


Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt 1115394/1115394 [==============================] – 0s 0us/step

Read the data

First, look in the text:


# Read, then decode for py2 compat. text = open(path_to_file, 'rb').read().decode(encoding='utf-8') # length of text is the number of characters in it print(f'Length of text: {len(text)} characters')


Length of text: 1115394 characters


# Take a look at the first 250 characters in text print(text[:250])


First Citizen: Before we proceed any further, hear me speak. All: Speak, speak. First Citizen: You are all resolved rather to die than to famish? All: Resolved. resolved. First Citizen: First, you know Caius Marcius is chief enemy to the people.


# The unique characters in the file vocab = sorted(set(text)) print(f'{len(vocab)} unique characters')


65 unique characters

Types of Recurrent Neural Networks

Feedforward networks have single input and output, while recurrent neural networks are flexible as the length of inputs and outputs can be changed. This flexibility allows RNNs to generate music, sentiment classification, and machine translation.

There are four types of RNN based on different lengths of inputs and outputs.

• One-to-one is a simple neural network. It is commonly used for machine learning problems that have a single input and output.
• One-to-many has a single input and multiple outputs. This is used for generating image captions.
• Many-to-one takes a sequence of multiple inputs and predicts a single output. It is popular in sentiment classification, where the input is text and the output is a category.
• Many-to-many takes multiple inputs and outputs. The most common application is machine translation.

Built-in RNN layers: a simple example

There are three built-in RNN layers in Keras:

1. 
keras.layers.SimpleRNN

   , a fully-connected RNN where the output from previous timestep is to be fed to next timestep.
2. 
keras.layers.GRU

   , first proposed in Cho et al., 2014.
3. 
keras.layers.LSTM

   , first proposed in Hochreiter & Schmidhuber, 1997.

In early 2015, Keras had the first reusable open-source Python implementations of LSTM and GRU.

Here is a simple example of a

Sequential

model that processes sequences of integers,
embeds each integer into a 64-dimensional vector, then processes the sequence of
vectors using a

LSTM

layer.


model = keras.Sequential() # Add an Embedding layer expecting input vocab of size 1000, and # output embedding dimension of size 64. model.add(layers.Embedding(input_dim=1000, output_dim=64)) # Add a LSTM layer with 128 internal units. model.add(layers.LSTM(128)) # Add a Dense layer with 10 units. model.add(layers.Dense(10)) model.summary()


Model: “sequential” _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (None, None, 64) 64000 lstm (LSTM) (None, 128) 98816 dense (Dense) (None, 10) 1290 ================================================================= Total params: 164106 (641.04 KB) Trainable params: 164106 (641.04 KB) Non-trainable params: 0 (0.00 Byte) _________________________________________________________________

Built-in RNNs support a number of useful features:

• Recurrent dropout, via the
  
dropout

  and
  
recurrent_dropout

  arguments
• Ability to process an input sequence in reverse, via the
  
go_backwards

  argument
• Loop unrolling (which can lead to a large speedup when processing short sequences on
  CPU), via the
  
unroll

  argument
• …and more.

For more information, see the RNN API documentation.

Outputs and states

By default, the output of a RNN layer contains a single vector per sample. This vector
is the RNN cell output corresponding to the last timestep, containing information
about the entire input sequence. The shape of this output is

(batch_size, units)

where

units

corresponds to the

units

argument passed to the layer’s constructor.

A RNN layer can also return the entire sequence of outputs for each sample (one vector
per timestep per sample), if you set

return_sequences=True

. The shape of this output
is

(batch_size, timesteps, units)

.


model = keras.Sequential() model.add(layers.Embedding(input_dim=1000, output_dim=64)) # The output of GRU will be a 3D tensor of shape (batch_size, timesteps, 256) model.add(layers.GRU(256, return_sequences=True)) # The output of SimpleRNN will be a 2D tensor of shape (batch_size, 128) model.add(layers.SimpleRNN(128)) model.add(layers.Dense(10)) model.summary()


Model: “sequential_1” _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= embedding_1 (Embedding) (None, None, 64) 64000 gru (GRU) (None, None, 256) 247296 simple_rnn (SimpleRNN) (None, 128) 49280 dense_1 (Dense) (None, 10) 1290 ================================================================= Total params: 361866 (1.38 MB) Trainable params: 361866 (1.38 MB) Non-trainable params: 0 (0.00 Byte) _________________________________________________________________

In addition, a RNN layer can return its final internal state(s). The returned states can be used to resume the RNN execution later, or to initialize another RNN. This setting is commonly used in the encoder-decoder sequence-to-sequence model, where the encoder final state is used as the initial state of the decoder.

To configure a RNN layer to return its internal state, set the

return_state

parameter
to

True

when creating the layer. Note that

LSTM

has 2 state tensors, but

GRU

only has one.

To configure the initial state of the layer, just call the layer with additional
keyword argument

initial_state

.
Note that the shape of the state needs to match the unit size of the layer, like in the
example below.


encoder_vocab = 1000 decoder_vocab = 2000 encoder_input = layers.Input(shape=(None,)) encoder_embedded = layers.Embedding(input_dim=encoder_vocab, output_dim=64)( encoder_input ) # Return states in addition to output output, state_h, state_c = layers.LSTM(64, return_state=True, name="encoder")( encoder_embedded ) encoder_state = [state_h, state_c] decoder_input = layers.Input(shape=(None,)) decoder_embedded = layers.Embedding(input_dim=decoder_vocab, output_dim=64)( decoder_input ) # Pass the 2 states to a new LSTM layer, as initial state decoder_output = layers.LSTM(64, name="decoder")( decoder_embedded, initial_state=encoder_state ) output = layers.Dense(10)(decoder_output) model = keras.Model([encoder_input, decoder_input], output) model.summary()


Model: “model” __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) [(None, None)] 0 [] input_2 (InputLayer) [(None, None)] 0 [] embedding_2 (Embedding) (None, None, 64) 64000 [‘input_1[0][0]’] embedding_3 (Embedding) (None, None, 64) 128000 [‘input_2[0][0]’] encoder (LSTM) [(None, 64), 33024 [’embedding_2[0][0]’] (None, 64), (None, 64)] decoder (LSTM) (None, 64) 33024 [’embedding_3[0][0]’, ‘encoder[0][1]’, ‘encoder[0][2]’] dense_2 (Dense) (None, 10) 650 [‘decoder[0][0]’] ================================================================================================== Total params: 258698 (1010.54 KB) Trainable params: 258698 (1010.54 KB) Non-trainable params: 0 (0.00 Byte) __________________________________________________________________________________________________

RNN layers and RNN cells

In addition to the built-in RNN layers, the RNN API also provides cell-level APIs. Unlike RNN layers, which processes whole batches of input sequences, the RNN cell only processes a single timestep.

The cell is the inside of the

for

loop of a RNN layer. Wrapping a cell inside a

keras.layers.RNN

layer gives you a layer capable of processing batches of
sequences, e.g.

RNN(LSTMCell(10))

.

Mathematically,

RNN(LSTMCell(10))

produces the same result as

LSTM(10)

. In fact,
the implementation of this layer in TF v1.x was just creating the corresponding RNN
cell and wrapping it in a RNN layer. However using the built-in

GRU

and

LSTM

layers enable the use of CuDNN and you may see better performance.

There are three built-in RNN cells, each of them corresponding to the matching RNN layer.

• 
keras.layers.SimpleRNNCell

  corresponds to the
  
SimpleRNN

  layer.
• 
keras.layers.GRUCell

  corresponds to the
  
GRU

  layer.
• 
keras.layers.LSTMCell

  corresponds to the
  
LSTM

  layer.

The cell abstraction, together with the generic

keras.layers.RNN

class, make it
very easy to implement custom RNN architectures for your research.

Limitations of RNN

In theory, RNN is supposed to carry the information up to times. However, it is quite challenging to propagate all this information when the time step is too long. When a network has too many deep layers, it becomes untrainable. This problem is called: vanishing gradient problem. If you remember, the neural network updates the weight using the gradient descent algorithm. The gradients grow smaller when the network progress down to lower layers.

In conclusion, the gradients stay constant meaning there is no space for improvement. The model learns from a change in the gradient; this change affects the network’s output. However, if the difference in the gradient is too small (i.e., the weights change a little), the network can’t learn anything and so the output. Therefore, a network facing a vanishing gradient problem cannot converge toward a good solution.

Cross-batch statefulness

When processing very long sequences (possibly infinite), you may want to use the pattern of cross-batch statefulness.

Normally, the internal state of a RNN layer is reset every time it sees a new batch (i.e. every sample seen by the layer is assumed to be independent of the past). The layer will only maintain a state while processing a given sample.

If you have very long sequences though, it is useful to break them into shorter sequences, and to feed these shorter sequences sequentially into a RNN layer without resetting the layer’s state. That way, the layer can retain information about the entirety of the sequence, even though it’s only seeing one sub-sequence at a time.

You can do this by setting

stateful=True

in the constructor.

If you have a sequence

s = [t0, t1, ... t1546, t1547]

, you would split it into e.g.


s1 = [t0, t1, ... t100] s2 = [t101, ... t201] ... s16 = [t1501, ... t1547]


Then you would process it via:


lstm_layer = layers.LSTM(64, stateful=True) for s in sub_sequences: output = lstm_layer(s)


When you want to clear the state, you can use

layer.reset_states()

.

Here is a complete example:


paragraph1 = np.random.random((20, 10, 50)).astype(np.float32) paragraph2 = np.random.random((20, 10, 50)).astype(np.float32) paragraph3 = np.random.random((20, 10, 50)).astype(np.float32) lstm_layer = layers.LSTM(64, stateful=True) output = lstm_layer(paragraph1) output = lstm_layer(paragraph2) output = lstm_layer(paragraph3) # reset_states() will reset the cached state to the original initial_state. # If no initial_state was provided, zero-states will be used by default. lstm_layer.reset_states()


RNN State Reuse

The recorded states of the RNN layer are not included in the

layer.weights()

. If you
would like to reuse the state from a RNN layer, you can retrieve the states value by

layer.states

and use it as the
initial state for a new layer via the Keras functional API like

new_layer(inputs,
initial_state=layer.states)

, or model subclassing.

Please also note that sequential model might not be used in this case since it only supports layers with single input and output, the extra input of initial state makes it impossible to use here.


paragraph1 = np.random.random((20, 10, 50)).astype(np.float32) paragraph2 = np.random.random((20, 10, 50)).astype(np.float32) paragraph3 = np.random.random((20, 10, 50)).astype(np.float32) lstm_layer = layers.LSTM(64, stateful=True) output = lstm_layer(paragraph1) output = lstm_layer(paragraph2) existing_state = lstm_layer.states new_lstm_layer = layers.LSTM(64) new_output = new_lstm_layer(paragraph3, initial_state=existing_state)


MasterCard Stock Price Prediction Using LSTM & GRU

In this project, we are going to use Kaggle’s MasterCard stock dataset from May-25-2006 to Oct-11-2021 and train the LSTM and GRU models to forecast the stock price. This is a simple project-based tutorial where we will analyze data, preprocess the data to train it on advanced RNN models, and finally evaluate the results.

The project requires Pandas and Numpy for data manipulation, Matplotlib.pyplot for data visualization, scikit-learn for scaling and evaluation, and TensorFlow for modeling. We will also set seeds for reproducibility.


# Importing the libraries import numpy as np import pandas as pd import matplotlib.pyplot as plt from sklearn.preprocessing import MinMaxScaler from sklearn.metrics import mean_squared_error from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, LSTM, Dropout, GRU, Bidirectional from tensorflow.keras.optimizers import SGD from tensorflow.random import set_seed set_seed(455) np.random.seed(455)


Data Analysis

In this part, we will import the MasterCard dataset by adding the Date column to the index and converting it to DateTime format. We will also drop irrelevant columns from the dataset as we are only interested in stock prices, volume, and date.

The dataset has Date as index and Open, High, Low, Close, and Volume as columns. It looks like we have successfully imported a cleaned dataset.


dataset = pd.read_csv( "data/Mastercard_stock_history.csv", index_col="Date", parse_dates=["Date"] ).drop(["Dividends", "Stock Splits"], axis=1) print(dataset.head()) Open High Low Close Volume Date 2006-05-25 3.748967 4.283869 3.739664 4.279217 395343000 2006-05-26 4.307126 4.348058 4.103398 4.179680 103044000 2006-05-30 4.183400 4.184330 3.986184 4.093164 49898000 2006-05-31 4.125723 4.219679 4.125723 4.180608 30002000 2006-06-01 4.179678 4.474572 4.176887 4.419686 62344000


The .describe() function helps us analyze the data in depth. Let’s focus on the High column as we are going to use it to train the model. We can also choose Close or Open columns for a model feature, but High makes more sense as it provides us information of how high the values of the share went on the given day.

The minimum stock price is $4.10, and the highest is $400.5. The mean is at $105.9 and the standard deviation $107.3, which means that stocks have high variance.


print(dataset.describe()) Open High Low Close Volume count 3872.000000 3872.000000 3872.000000 3872.000000 3.872000e+03 mean 104.896814 105.956054 103.769349 104.882714 1.232250e+07 std 106.245511 107.303589 105.050064 106.168693 1.759665e+07 min 3.748967 4.102467 3.739664 4.083861 6.411000e+05 25% 22.347203 22.637997 22.034458 22.300391 3.529475e+06 50% 70.810079 71.375896 70.224002 70.856083 5.891750e+06 75% 147.688448 148.645373 146.822013 147.688438 1.319775e+07 max 392.653890 400.521479 389.747812 394.685730 3.953430e+08


By using .isna().sum() we can determine the missing values in the dataset. It seems that the dataset has no missing values.


dataset.isna().sum() Open 0 High 0 Low 0 Close 0 Volume 0 dtype: int64


The train_test_plot function takes three arguments: dataset, tstart, and tend and plots a simple line graph. The tstart and tend are time limits in years. We can change these arguments to analyze specific periods. The line plot is divided into two parts: train and test. This will allow us to decide the distribution of the test dataset.

MasterCard stock prices have been on the rise since 2016. It had a dip in the first quarter of 2020 but it gained a stable position in the latter half of the year. Our test dataset consists of one year, from 2021 to 2022, and the rest of the dataset is used for training.


tstart = 2016 tend = 2020 def train_test_plot(dataset, tstart, tend): dataset.loc[f"{tstart}":f"{tend}", "High"].plot(figsize=(16, 4), legend=True) dataset.loc[f"{tend+1}":, "High"].plot(figsize=(16, 4), legend=True) plt.legend([f"Train (Before {tend+1})", f"Test ({tend+1} and beyond)"]) plt.title("MasterCard stock price") plt.show() train_test_plot(dataset,tstart,tend)


Data Preprocessing

The train_test_split function divides the dataset into two subsets: training_set and test_set.


def train_test_split(dataset, tstart, tend): train = dataset.loc[f"{tstart}":f"{tend}", "High"].values test = dataset.loc[f"{tend+1}":, "High"].values return train, test training_set, test_set = train_test_split(dataset, tstart, tend)


We will use the MinMaxScaler function to standardize our training set, which will help us avoid the outliers or anomalies. You can also try using StandardScaler or any other scalar function to normalize your data and improve model performance.


sc = MinMaxScaler(feature_range=(0, 1)) training_set = training_set.reshape(-1, 1) training_set_scaled = sc.fit_transform(training_set)


The split_sequence function uses a training dataset and converts it into inputs (X_train) and outputs (y_train).

For example, if the sequence is [1,2,3,4,5,6,7,8,9,10,11,12] and the n_step is three, then it will convert the sequence into three input timestamps and one output as shown below:

1,2,3

2,3,4

3,4,5

4,5,6

In this project, we are using 60 n_steps. We can also reduce or increase the number of steps to optimize model performance.


def split_sequence(sequence, n_steps): X, y = list(), list() for i in range(len(sequence)): end_ix = i + n_steps if end_ix > len(sequence) - 1: break seq_x, seq_y = sequence[i:end_ix], sequence[end_ix] X.append(seq_x) y.append(seq_y) return np.array(X), np.array(y) n_steps = 60 features = 1 # split into samples X_train, y_train = split_sequence(training_set_scaled, n_steps)


We are working with univariate series, so the number of features is one, and we need to reshape the X_train to fit on the LSTM model. The X_train has [samples, timesteps], and we will reshape it to [samples, timesteps, features].


# Reshaping X_train for model X_train = X_train.reshape(X_train.shape[0],X_train.shape[1],features)


LSTM Model

The model consists of a single hidden layer of LSTM and an output layer. You can experiment with the number of units, as more units will give you better results. For this experiment, we will set LSTM units to 125, tanh as activation, and set input size.

Author’s Note: Tensorflow library is user-friendly, so we don’t have to create LSTM or GRU models from scratch. We will simply use the LSTM or GRU modules to construct the model.

Finally, we will compile the model with an RMSprop optimizer and mean square error as a loss function.


# The LSTM architecture model_lstm = Sequential() model_lstm.add(LSTM(units=125, activation="tanh", input_shape=(n_steps, features))) model_lstm.add(Dense(units=1)) # Compiling the model model_lstm.compile(optimizer="RMSprop", loss="mse") model_lstm.summary() Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= lstm (LSTM) (None, 125) 63500 _________________________________________________________________ dense (Dense) (None, 1) 126 ================================================================= Total params: 63,626 Trainable params: 63,626 Non-trainable params: 0 _________________________________________________________________


The model will train on 50 epochs with 32 batch sizes. You can change the hyperparameters to reduce training time or improve the results. The model training was successfully completed with the best possible loss.


model_lstm.fit(X_train, y_train, epochs=50, batch_size=32) Epoch 50/50 38/38 [==============================] - 1s 30ms/step - loss: 3.1642e-04


Results

We are going to repeat preprocessing and normalize the test set. First of all we will transform then split the dataset into samples, reshape it, predict, and inverse transform the predictions into standard form.


dataset_total = dataset.loc[:,"High"] inputs = dataset_total[len(dataset_total) - len(test_set) - n_steps :].values inputs = inputs.reshape(-1, 1) #scaling inputs = sc.transform(inputs) # Split into samples X_test, y_test = split_sequence(inputs, n_steps) # reshape X_test = X_test.reshape(X_test.shape[0], X_test.shape[1], features) #prediction predicted_stock_price = model_lstm.predict(X_test) #inverse transform the values predicted_stock_price = sc.inverse_transform(predicted_stock_price)


The plot_predictions function will plot a real versus predicted line chart. This will help us visualize the difference between actual and predicted values.

The return_rmse function takes in test and predicted arguments and prints out the root mean square error (rmse) metric.


def plot_predictions(test, predicted): plt.plot(test, color="gray", label="Real") plt.plot(predicted, color="red", label="Predicted") plt.title("MasterCard Stock Price Prediction") plt.xlabel("Time") plt.ylabel("MasterCard Stock Price") plt.legend() plt.show() def return_rmse(test, predicted): rmse = np.sqrt(mean_squared_error(test, predicted)) print("The root mean squared error is {:.2f}.".format(rmse))


According to the line plot below, the single-layered LSTM model has performed well.


plot_predictions(test_set,predicted_stock_price)


The results look promising as the model got 6.70 rmse on the test dataset.


return_rmse(test_set,predicted_stock_price) >>> The root mean squared error is 6.70.


GRU Model

We are going to keep everything the same and just replace the LSTM layer with the GRU layer to properly compare the results. The model structure contains a single GRU layer with 125 units and an output layer.


model_gru = Sequential() model_gru.add(GRU(units=125, activation="tanh", input_shape=(n_steps, features))) model_gru.add(Dense(units=1)) # Compiling the RNN model_gru.compile(optimizer="RMSprop", loss="mse") model_gru.summary() Model: "sequential_5" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= gru_4 (GRU) (None, 125) 48000 _________________________________________________________________ dense_5 (Dense) (None, 1) 126 ================================================================= Total params: 48,126 Trainable params: 48,126 Non-trainable params: 0 _________________________________________________________________


The model has successfully trained with 50 epochs and a batch size of 32.


model_gru.fit(X_train, y_train, epochs=50, batch_size=32) Epoch 50/50 38/38 [==============================] - 1s 29ms/step - loss: 2.6691e-04


Results

As we can see, the real and predicted values are relatively close. The predicted line chart almost fits the actual values.


GRU_predicted_stock_price = model_gru.predict(X_test) GRU_predicted_stock_price = sc.inverse_transform(GRU_predicted_stock_price) plot_predictions(test_set, GRU_predicted_stock_price)


GRU model got 5.50 rmse on the test dataset, which is an improvement from the LSTM model.


return_rmse(test_set,GRU_predicted_stock_price) >>> The root mean squared error is 5.50.


Training Recurrent Neural Networks (RNN)

• To train an RNN, the trick is to unroll it through time and then actually use regular backpropagation. This strategy is known as backpropagation through time (BPTT).
• There’s a first forward pass via the unrolled network. Then the output sequence is evaluated with the use of a cost function C.
• The gradients of that cost feature are then propagated backwards via the unrolled network.
• Now the model parameters have updated the use of the gradients computed all through BPTT.

Limitations of RNN

Simple RNN models usually run into two major issues. These issues are related to gradient, which is the slope of the loss function along with the error function.

1. Vanishing Gradient problem occurs when the gradient becomes so small that updating parameters becomes insignificant; eventually the algorithm stops learning.
2. Exploding Gradient problem occurs when the gradient becomes too large, which makes the model unstable. In this case, larger error gradients accumulate, and the model weights become too large. This issue can cause longer training times and poor model performance.

The simple solution to these issues is to reduce the number of hidden layers within the neural network, which will reduce some complexity in RNNs. These issues can also be solved by using advanced RNN architectures such as LSTM and GRU.

Why do we need a Recurrent Neural Network (RNN)?

Recurrent Neural Network (RNN) allows you to model memory units to persist data and model short term dependencies. It is also used in time-series forecasting for the identification of data correlations and patterns. It also helps to produce predictive results for sequential data by delivering similar behavior as a human brain.

The structure of an Artificial Neural Network is relatively simple and is mainly about matrix multiplication. During the first step, inputs are multiplied by initially random weights, and bias, transformed with an activation function and the output values are used to make a prediction. This step gives an idea of how far the network is from the reality.

The metric applied is the loss. The higher the loss function, the dumber the model is. To improve the knowledge of the network, some optimization is required by adjusting the weights of the net. The stochastic gradient descent is the method employed to change the values of the weights in the rights direction. Once the adjustment is made, the network can use another batch of data to test its new knowledge.

The error, fortunately, is lower than before, yet not small enough. The optimization step is done iteratively until the error is minimized, i.e., no more information can be extracted.

The problem with this type of model is, it does not have any memory. It means the input and output are independent. In other words, the model does not care about what came before. It raises some question when you need to predict time series or sentences because the network needs to have information about the historical data or past words.

To overcome this issue, a new type of architecture has been developed: Recurrent Neural network (RNN hereafter)

Setup input pipeline

The IMDB large movie review dataset is a binary classification dataset—all the reviews have either a positive or negative sentiment.

Download the dataset using TFDS. See the loading text tutorial for details on how to load this sort of data manually.


dataset, info = tfds.load('imdb_reviews', with_info=True, as_supervised=True) train_dataset, test_dataset = dataset['train'], dataset['test'] train_dataset.element_spec


(TensorSpec(shape=(), dtype=tf.string, name=None), TensorSpec(shape=(), dtype=tf.int64, name=None))

Initially this returns a dataset of (text, label pairs):


for example, label in train_dataset.take(1): print('text: ', example.numpy()) print('label: ', label.numpy())


text: b”This was an absolutely terrible movie. Don’t be lured in by Christopher Walken or Michael Ironside. Both are great actors, but this must simply be their worst role in history. Even their great acting could not redeem this movie’s ridiculous storyline. This movie is an early nineties US propaganda piece. The most pathetic scenes were those when the Columbian rebels were making their cases for revolutions. Maria Conchita Alonso appeared phony, and her pseudo-love affair with Walken was nothing but a pathetic emotional plug in a movie that was devoid of any real meaning. I am disappointed that there are movies like this, ruining actor’s like Christopher Walken’s good name. I could barely sit through it.” label: 0

Next shuffle the data for training and create batches of these

(text, label)

pairs:


BUFFER_SIZE = 10000 BATCH_SIZE = 64



train_dataset = train_dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE) test_dataset = test_dataset.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)



for example, label in train_dataset.take(1): print('texts: ', example.numpy()[:3]) print() print('labels: ', label.numpy()[:3])


texts: [b’Watching beautiful women sneaking around, playing cops and robbers is one of the most delightful guilty pleasures the medium film lets me enjoy. So The House on Carroll Street was not entirely a waste of time, although the story is contrived and the screenplay uninspired and somewhat irritating.There are many allusions to different Hitchcock pictures, not least the choice of Kelly McGillis in the starring role. She is dressed up as Grace Kelly, and she is not far off the mark. Not at all. But her character is not convincing. The way she is introduced to the audience, she should be someone with political convictions and a purpose in life. After all the movie deals with a clearly defined time period, true events and a specific issue. But the story degenerates within the first minutes into a sorry run-off-the-mill crime story with unbelievable coincidences, high predictability and a set of two dimensional characters. This is all the more regrettable, as the performances of the actors are good, as are the photography and the set design.The finale in Central Station, New York is breath taking. It starts in the subterranean section and then moves up to the roof. The movie can be praised for its good use of architecture.’ b’A group of people are invited to there high school reunion, but after they arrive they discover it to be a scam by an old classmate they played an almost fatal prank on. Now, he seeks to get revenge on all those that hurt him by sealing all the exits and cutting off all telephone lines.Dark slasher film with an unexceptional premise. Bringing it up a notch are a few good performances, some rather creative death scenes, plenty of excitement & scares, some humor and an original ending.Unrated for Extreme Violence, Graphic Nudity, Sexual Situations, Profanity and Drug Use.’ b’The short that starts this film is the true footage of a guy named Gary, apparently it was taken randomly in the parking lot of a television station where Gary works in the town of Beaver. Gary is a little “different”; he is an impersonator and drives an old Chevy named Farrah (after Fawcett). Lo and behold the filmmaker gets a letter from Gary some time later inviting him to return to Beaver to get some footage of the local talent contest he has put together, including Gary\’s staggering performace as Olivia Newton Dawn. Oh, my. The two shorts that follow are Gary\’s story, the same one you just witnessed only the first is portrayed by Sean Penn and the second by Crispin Glover titled “The Orkly Kid.” If you are in the mood for making fun of someone this is definitely the film to watch. I was doubled over with laughter through most of it, especially Crispins performance which could definitely stand on it\’s own. When it was over, I had to rewind the film to once again watch the real Gary and all his shining idiocy. Although Olivia was the focus, I would have liked to have seen one of the “fictitious” shorts take a jab at Gary\’s Barry Manilow impersonation, whic h was equally ridiculous.’] labels: [0 0 1]

Build an RNN to predict Time Series in TensorFlow

Now in this RNN training, it is time to build your first RNN to predict the series above. You need to specify some hyperparameters (the parameters of the model, i.e., number of neurons, etc.) for the model:

• Number of input: 1
• Time step (windows in time series): 10
• Number of neurons: 120
• Number of output: 1

Your network will learn from a sequence of 10 days and contain 120 recurrent neurons. You feed the model with one input, i.e., one day. Feel free to change the values to see if the model improved.

Before to construct the model, you need to split the dataset into a train set and test set. The full dataset has 222 data points; you will use the first 201 point to train the model and the last 21 points to test your model.

After you define a train and test set, you need to create an object containing the batches. In this batches, you have X values and Y values. Remember that the X values are one period lagged. Therefore, you use the first 200 observations and the time step is equal to 10. The X_batches object should contain 20 batches of size 10*1. The y_batches has the same shape as the X_batches object but with one period ahead.

Step 1) Create the train and test

First of all, you convert the series into a numpy array; then you define the windows (i.e., the number of time the network will learn from), the number of input, output and the size of the train set as shown in the TensorFlow RNN example below.

series = np.array(ts) n_windows = 20 n_input = 1 n_output = 1 size_train = 201

After that, you simply split the array into two datasets.

## Split data train = series[:size_train] test = series[size_train:] print(train.shape, test.shape) (201,) (21,)

Step 2) Create the function to return X_batches and y_batches

To make it easier, you can create a function that returns two different arrays, one for X_batches and one for y_batches.

Let’s write a RNN TensorFlow function to construct the batches.

Note that, the X batches are lagged by one period (we take value t-1). The output of the function should have three dimensions. The first dimensions equal the number of batches, the second the size of the windows and last one the number of input.

The tricky part is to select the data points correctly. For the X data points, you choose the observations from t = 1 to t =200, while for the Y data point, you return the observations from t = 2 to 201. Once you have the correct data points, it is straightforward to reshape the series.

To construct the object with the batches, you need to split the dataset into ten batches of equal length (i.e., 20). You can use the reshape method and pass -1 so that the series is similar to the batch size. The value 20 is the number of observations per batch and 1 is the number of input.

You need to do the same step but for the label.

Note that, you need to shift the data to the number of times you want to forecast. For instance, if you want to predict one timeahead, then you shift the series by 1. If you want to forecast two days, then shift the data by 2.

x_data = train[:size_train-1]: Select all the training instance minus one day X_batches = x_data.reshape(-1, windows, input): create the right shape for the batch e.g (10, 20, 1) def create_batches(df, windows, input, output): ## Create X x_data = train[:size_train-1] # Select the data X_batches = x_data.reshape(-1, windows, input) # Reshape the data ## Create y y_data = train[n_output:size_train] y_batches = y_data.reshape(-1, windows, output) return X_batches, y_batches

Now that the function is defined, you can call it to create the batches as shown in the below RNN example.

X_batches, y_batches = create_batches(df = train, windows = n_windows, input = n_input, output = n_output)

You can print the shape to make sure the dimensions are correct.

print(X_batches.shape, y_batches.shape) (10, 20, 1) (10, 20, 1)

You need to create the test set with only one batch of data and 20 observations.

Note that, you forecast days after days, it means the second predicted value will be based on the true value of the first day (t+1) of the test dataset. In fact, the true value will be known.

If you want to forecast t+2 (i.e., two days ahead), you need to use the predicted value t+1; if you’re going to predict t+3 (three days ahead), you need to use the predicted value t+1 and t+2. It makes sense that, it is difficult to predict accurately t+n days ahead.

X_test, y_test = create_batches(df = test, windows = 20,input = 1, output = 1) print(X_test.shape, y_test.shape) (10, 20, 1) (10, 20, 1)

Alright, your batch size is ready, you can build the RNN architecture. Remember, you have 120 recurrent neurons.

Step 3) Build the model

To create the model, you need to define three parts:

1. The variable with the tensors
2. The RNN
3. The loss and optimization

Step 3.1) Variables

You need to specify the X and y variables with the appropriate shape. This step is trivial. The tensor has the same dimension as the objects X_batches and y_batches.

For instance, the tensor X is a placeholder (Check the tutorial on Introduction to Tensorflow to refresh your mind about variable declaration) has three dimensions:

• Note: size of the batch
• n_windows: Lenght of the windows. i.e., the number of time the model looks backward
• n_input: Number of input

The result is:

tf.placeholder(tf.float32, [None, n_windows, n_input])

## 1. Construct the tensors X = tf.placeholder(tf.float32, [None, n_windows, n_input]) y = tf.placeholder(tf.float32, [None, n_windows, n_output])

Step 3.2) Create the RNN

In the second part of this RNN TensorFlow example, you need to define the architecture of the network. As before, you use the object BasicRNNCell and dynamic_rnn from TensorFlow estimator.

## 2. create the model basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=r_neuron, activation=tf.nn.relu) rnn_output, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32)

The next part is a bit trickier but allows faster computation. You need to transform the run output to a dense layer and then convert it again to have the same dimension as the input.

stacked_rnn_output = tf.reshape(rnn_output, [-1, r_neuron]) stacked_outputs = tf.layers.dense(stacked_rnn_output, n_output) outputs = tf.reshape(stacked_outputs, [-1, n_windows, n_output])

Step 3.3) Create the loss and optimization

The model optimization depends of the task you are performing. In the previous tutorial on CNN, your objective was to classify images, in this RNN tutorial, the objective is slightly different. You are asked to make a prediction on a continuous variable compare to a class.

This difference is important because it will change the optimization problem. The optimization problem for a continuous variable is to minimize the mean square error. To construct these metrics in TF, you can use:

• tf.reduce_sum(tf.square(outputs – y))

The remaining of the RNN code is the same as before; you use an Adam optimizer to reduce the loss (i.e., MSE):

• tf.train.AdamOptimizer(learning_rate=learning_rate)
• optimizer.minimize(loss)

That’s it, you can pack everything together, and your model is ready to train.

tf.reset_default_graph() r_neuron = 120 ## 1. Construct the tensors X = tf.placeholder(tf.float32, [None, n_windows, n_input]) y = tf.placeholder(tf.float32, [None, n_windows, n_output]) ## 2. create the model basic_cell = tf.contrib.rnn.BasicRNNCell(num_units=r_neuron, activation=tf.nn.relu) rnn_output, states = tf.nn.dynamic_rnn(basic_cell, X, dtype=tf.float32) stacked_rnn_output = tf.reshape(rnn_output, [-1, r_neuron]) stacked_outputs = tf.layers.dense(stacked_rnn_output, n_output) outputs = tf.reshape(stacked_outputs, [-1, n_windows, n_output]) ## 3. Loss + optimization learning_rate = 0.001 loss = tf.reduce_sum(tf.square(outputs – y)) optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate) training_op = optimizer.minimize(loss) init = tf.global_variables_initializer()

You will train the model using 1500 epochs and print the loss every 150 iterations. Once the model is trained, you evaluate the model on the test set and create an object containing the predictions as shown in the below Recurrent Neural Network example.

iteration = 1500 with tf.Session() as sess: init.run() for iters in range(iteration): sess.run(training_op, feed_dict={X: X_batches, y: y_batches}) if iters % 150 == 0: mse = loss.eval(feed_dict={X: X_batches, y: y_batches}) print(iters, “\tMSE:”, mse) y_pred = sess.run(outputs, feed_dict={X: X_test}) 0 MSE: 502893.34 150 MSE: 13839.129 300 MSE: 3964.835 450 MSE: 2619.885 600 MSE: 2418.772 750 MSE: 2110.5923 900 MSE: 1887.9644 1050 MSE: 1747.1377 1200 MSE: 1556.3398 1350 MSE: 1384.6113

At last in this RNN Deep Learning tutorial, you can plot the actual value of the series with the predicted value. If your model is corrected, the predicted values should be put on top of the actual values.

As you can see, the model has room of improvement. It is up to you to change the hyperparameters like the windows, the batch size of the number of recurrent neurons.

plt.title(“Forecast vs Actual”, fontsize=14) plt.plot(pd.Series(np.ravel(y_test)), “bo”, markersize=8, label=”Actual”, color=’green’) plt.plot(pd.Series(np.ravel(y_pred)), “r.”, markersize=8, label=”Forecast”, color=’red’) plt.legend(loc=”lower left”) plt.xlabel(“Time”) plt.show()

CNNRNN

The convolutional neural network (CNN) is a feed-forward neural network capable of processing spatial data. It is commonly used for computer vision applications such as image classification. The simple neural networks are good at simple binary classifications, but they can’t handle images with pixel dependencies. The CNN model architecture consists of convolutional layers, ReLU layers, pooling layers, and fully connected output layers. You can learn CNN by working on a project such as Convolutional Neural Networks in Python.

Key Differences Between CNN and RNN

• CNN is applicable for sparse data like images. RNN is applicable for time series and sequential data.
• While training the model, CNN uses a simple backpropagation and RNN uses backpropagation through time to calculate the loss.
• RNN can have no restriction in length of inputs and outputs, but CNN has finite inputs and finite outputs.
• CNN has a feedforward network and RNN works on loops to handle sequential data.
• CNN can also be used for video and image processing. RNN is primarily used for speech and text analysis.

Two Issues of Standard RNNs

Vanishing Gradient Problem

Recurrent Neural Networks enable you to model time-dependent and sequential data problems, like stock exchange prediction, artificial intelligence, and text generation. you’ll find, however, RNN is tough to train due to the gradient problem.

RNNs suffer from the matter of vanishing gradients. The gradients carry information utilized in the RNN, and when the gradient becomes too small, the parameter updates become insignificant. This makes the training of long data sequences difficult.

Exploding Gradient Problem

While training a neural network, if the slope tends to grow exponentially rather than decaying, this is often called an Exploding Gradient. This problem arises when large error gradients accumulate, leading to very large updates to the neural network model weights during the training process.

Long training time, poor performance, and bad accuracy are the key issues in gradient problems.

Gradient Problem Solutions

Now, let’s discuss the foremost popular and efficient thanks to cope with gradient problems, i.e., Long immediate memory Network (LSTMs).First, let’s understand Long-Term Dependencies.Suppose you wish to predict the last word within the text: “The clouds are within the ______.”The most obvious answer to the present is that the “sky.” We don’t need from now on context to predict the last word within the above sentence.Consider this sentence: “I are staying in Spain for the last 10 years…I can speak fluent ______.”The word you are expecting will rely on the previous couple of words in context. Here, you would like the context of Spain to predict the last word within the text, and also the most fitted answer to the present sentence is “Spanish.” The gap between the relevant information and also the point where it’s needed may became very large. LSTMs facilitate your solve this problem.

Recurrent Neural Network Implementation with TensorFlow

In this section, we will learn how to implement recurrent neural network with TensorFlow.

Step 1 − TensorFlow includes various libraries for specific implementation of the recurrent neural network module.

#Import necessary modules from __future__ import print_function import tensorflow as tf from tensorflow.contrib import rnn from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets(“/tmp/data/”, one_hot = True)

As mentioned above, the libraries help in defining the input data, which forms the primary part of recurrent neural network implementation.

Step 2 − Our primary motive is to classify the images using a recurrent neural network, where we consider every image row as a sequence of pixels. MNIST image shape is specifically defined as 28*28 px. Now we will handle 28 sequences of 28 steps for each sample that is mentioned. We will define the input parameters to get the sequential pattern done.

n_input = 28 # MNIST data input with img shape 28*28 n_steps = 28 n_hidden = 128 n_classes = 10 # tf Graph input x = tf.placeholder(“float”, [None, n_steps, n_input]) y = tf.placeholder(“float”, [None, n_classes] weights = { ‘out’: tf.Variable(tf.random_normal([n_hidden, n_classes])) } biases = { ‘out’: tf.Variable(tf.random_normal([n_classes])) }

Step 3 − Compute the results using a defined function in RNN to get the best results. Here, each data shape is compared with current input shape and the results are computed to maintain the accuracy rate.

def RNN(x, weights, biases): x = tf.unstack(x, n_steps, 1) # Define a lstm cell with tensorflow lstm_cell = rnn.BasicLSTMCell(n_hidden, forget_bias=1.0) # Get lstm cell output outputs, states = rnn.static_rnn(lstm_cell, x, dtype = tf.float32) # Linear activation, using rnn inner loop last output return tf.matmul(outputs[-1], weights[‘out’]) + biases[‘out’] pred = RNN(x, weights, biases) # Define loss and optimizer cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits = pred, labels = y)) optimizer = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(cost) # Evaluate model correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1)) accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32)) # Initializing the variables init = tf.global_variables_initializer()

Step 4 − In this step, we will launch the graph to get the computational results. This also helps in calculating the accuracy for test results.

with tf.Session() as sess: sess.run(init) step = 1 # Keep training until reach max iterations while step * batch_size < training_iters: batch_x, batch_y = mnist.train.next_batch(batch_size) batch_x = batch_x.reshape((batch_size, n_steps, n_input)) sess.run(optimizer, feed_dict={x: batch_x, y: batch_y}) if step % display_step == 0: # Calculate batch accuracy acc = sess.run(accuracy, feed_dict={x: batch_x, y: batch_y}) # Calculate batch loss loss = sess.run(cost, feed_dict={x: batch_x, y: batch_y}) print(“Iter ” + str(step*batch_size) + “, Minibatch Loss= ” + \ “{:.6f}”.format(loss) + “, Training Accuracy= ” + \ “{:.5f}”.format(acc)) step += 1 print(“Optimization Finished!”) test_len = 128 test_data = mnist.test.images[:test_len].reshape((-1, n_steps, n_input)) test_label = mnist.test.labels[:test_len] print(“Testing Accuracy:”, \ sess.run(accuracy, feed_dict={x: test_data, y: test_label}))

The screenshots below show the output generated −