Convolutional LSTM for spatial forecasting

This put up is the primary in a unfastened collection exploring forecasting of spatially-determined knowledge over the years. Via spatially-determined I imply that regardless of the amounts we’re looking to are expecting – be they univariate or multivariate time collection, of spatial dimensionality or now not – the enter knowledge are given on a spatial grid.

For instance, the enter may well be atmospheric measurements, comparable to sea floor temperature or power, given at some set of latitudes and longitudes. The objective to be predicted may then span that very same (or every other) grid. Then again, it generally is a univariate time collection, like a meteorological index.

However wait a 2nd, you’ll be pondering. For time-series prediction, now we have that time-honored set of recurrent architectures (e.g., LSTM, GRU), proper? Proper. We do; however, after we feed spatial knowledge to an RNN, treating other places as other enter options, we lose an crucial structural dating. Importantly, we wish to function in each house and time. We wish each: recurrence members of the family and convolutional filters. Input convolutional RNNs.

What to anticipate from this put up

Nowadays, we gained’t soar into real-world programs simply but. As a substitute, we’ll take our time to construct a convolutional LSTM (henceforth: convLSTM) in torch. For one, we need to – there is not any professional PyTorch implementation.

What’s extra, this put up can function an advent to construction your individual modules. That is one thing you’ll be aware of from Keras or now not – relying on whether or not you’ve used customized fashions or relatively, most popular the declarative outline -> collect -> have compatibility taste. (Sure, I’m implying there’s some switch occurring if one involves torch from Keras customized coaching. Syntactic and semantic main points is also other, however each proportion the object-oriented taste that permits for excellent flexibility and keep an eye on.)

Closing however now not least, we’ll additionally use this as a hands-on revel in with RNN architectures (the LSTM, in particular). Whilst the overall idea of recurrence is also simple to snatch, it isn’t essentially self-evident how the ones architectures must, or may, be coded. Individually, I to find that impartial of the framework used, RNN-related documentation leaves me puzzled. What precisely is being returned from calling an LSTM, or a GRU? (In Keras this relies on the way you’ve outlined the layer in query.) I think that after we’ve determined what we need to go back, the real code gained’t be that difficult. Because of this, we’ll take a detour clarifying what it’s that torch and Keras are giving us. Enforcing our convLSTM will likely be much more easy thereafter.

A torch convLSTM

The code mentioned right here is also discovered on GitHub. (Relying on while you’re studying this, the code in that repository will have developed even though.)

My start line used to be one of the most PyTorch implementations discovered on the web, particularly, this one. In the event you seek for “PyTorch convGRU” or “PyTorch convLSTM”, you’ll to find shocking discrepancies in how those are discovered – discrepancies now not simply in syntax and/or engineering ambition, however at the semantic point, proper on the heart of what the architectures is also anticipated to do. As they are saying, let the consumer beware. (In regards to the implementation I stopped up porting, I’m assured that whilst a large number of optimizations will likely be probable, the fundamental mechanism suits my expectancies.)

What do I be expecting? Let’s means this job in a top-down means.

Enter and output

The convLSTM’s enter will likely be a time collection of spatial knowledge, each and every remark being of dimension (time steps, channels, top, width).

Evaluate this with the standard RNN enter layout, be it in torch or Keras. In each frameworks, RNNs be expecting tensors of dimension (timesteps, input_dim). input_dim is (1) for univariate time collection and larger than (1) for multivariate ones. Conceptually, we would possibly fit this to convLSTM’s channels measurement: There generally is a unmarried channel, for temperature, say – or there may well be a number of, comparable to for power, temperature, and humidity. The 2 further dimensions present in convLSTM, top and width, are spatial indexes into the information.

In sum, we wish so to go knowledge that:

• include a number of options,

• evolve in time, and

• are listed in two spatial dimensions.

How concerning the output? We wish so to go back forecasts for as many time steps as now we have within the enter series. That is one thing that torch RNNs do through default, whilst Keras equivalents don’t. (You need to go return_sequences = TRUE to procure that impact.) If we’re thinking about predictions for only a unmarried time limit, we will be able to all the time select the final time step within the output tensor.

Alternatively, with RNNs, it isn’t all about outputs. RNN architectures additionally lift via hidden states.

What are hidden states? I moderately phrased that sentence to be as basic as probable – intentionally circling across the confusion that, individually, regularly arises at this level. We’ll try to transparent up a few of that confusion in a 2nd, however let’s first end our high-level necessities specification.

We wish our convLSTM to be usable in several contexts and programs. More than a few architectures exist that employ hidden states, maximum prominently possibly, encoder-decoder architectures. Thus, we wish our convLSTM to go back the ones as smartly. Once more, that is one thing a torch LSTM does through default, whilst in Keras it’s accomplished the usage of return_state = TRUE.

Now even though, it in point of fact is time for that interlude. We’ll type out the techniques issues are known as through each torch and Keras, and investigate cross-check what you get again from their respective GRUs and LSTMs.

Interlude: Outputs, states, hidden values … what’s what?

For this to stay an interlude, I summarize findings on a excessive point. The code snippets within the appendix display the best way to arrive at those effects. Closely commented, they probe go back values from each Keras and torch GRUs and LSTMs. Operating those will make the approaching summaries appear so much much less summary.

First, let’s have a look at the techniques you create an LSTM in each frameworks. (I will be able to in most cases use LSTM because the “prototypical RNN instance”, and simply point out GRUs when there are variations important within the context in query.)

In Keras, to create an LSTM you could write one thing like this:

lstm <- layer_lstm(gadgets = 1)

The torch similar can be:

lstm <- nn_lstm(
  input_size = 2, # choice of enter options
  hidden_size = 1 # choice of hidden (and output!) options
)

Don’t center of attention on torch‘s input_size parameter for this dialogue. (It’s the choice of options within the enter tensor.) The parallel happens between Keras’ gadgets and torch’s hidden_size. In the event you’ve been the usage of Keras, you’re most certainly pondering of gadgets as the item that determines output dimension (equivalently, the choice of options within the output). So when torch shall we us arrive on the similar consequence the usage of hidden_size, what does that imply? It implies that by some means we’re specifying the similar factor, the usage of other terminology. And it does make sense, since at each and every time step present enter and former hidden state are added:

[
mathbf{h}_t = mathbf{W}_{x}mathbf{x}_t + mathbf{W}_{h}mathbf{h}_{t-1}
]

Now, about the ones hidden states.

When a Keras LSTM is outlined with return_state = TRUE, its go back price is a construction of 3 entities known as output, reminiscence state, and lift state. In torch, the similar entities are known as output, hidden state, and mobile state. (In torch, we all the time get they all.)

So are we coping with 3 several types of entities? We aren’t.

The mobile, or lift state is that particular factor that units aside LSTMs from GRUs deemed chargeable for the “lengthy” in “lengthy momentary reminiscence”. Technically, it may well be reported to the person in any respect deadlines; as we’ll see in a while even though, it isn’t.

What about outputs and hidden, or reminiscence states? Confusingly, those in point of fact are the similar factor. Recall that for each and every enter merchandise within the enter series, we’re combining it with the former state, leading to a brand new state, to be made used of in the next move:

[
mathbf{h}_t = mathbf{W}_{x}mathbf{x}_t + mathbf{W}_{h}mathbf{h}_{t-1}
]

Now, say that we’re thinking about taking a look at simply the ultimate time step – this is, the default output of a Keras LSTM. From that viewpoint, we will be able to believe the ones intermediate computations as “hidden”. Noticed like that, output and hidden states really feel other.

Alternatively, we will be able to additionally request to peer the outputs for each and every time step. If we accomplish that, there is not any distinction – the outputs (plural) equivalent the hidden states. This may also be verified the usage of the code within the appendix.

Thus, of the 3 issues returned through an LSTM, two are in point of fact the similar. How concerning the GRU, then? As there is not any “mobile state”, we in point of fact have only one form of factor left over – name it outputs or hidden states.

Let’s summarize this in a desk.

Now, about that public vs. personal difference. In each frameworks, we will be able to download outputs (hidden states) for each and every time step. The mobile state, alternatively, we will be able to get entry to just for the very final time step. That is purely an implementation choice. As we’ll see when construction our personal recurrent module, there are not any hindrances inherent in keeping an eye on mobile states and passing them again to the person.

In the event you dislike the pragmatism of this difference, you’ll all the time cross with the maths. When a brand new mobile state has been computed (in line with prior mobile state, enter, omit, and mobile gates – the specifics of which we aren’t going to get into right here), it’s reworked to the hidden (a.ok.a. output) state applying but every other, particularly, the output gate:

[
h_t = o_t odot tanh(c_t)
]

Indisputably, then, hidden state (output, resp.) builds on mobile state, including further modeling energy.

Now it’s time to get again to our authentic function and construct that convLSTM. First even though, let’s summarize the go back values out there from torch and Keras.

convLSTM, the plan

In each torch and Keras RNN architectures, unmarried time steps are processed through corresponding Mobile categories: There may be an LSTM Mobile matching the LSTM, a GRU Mobile matching the GRU, and so forth. We do the similar for ConvLSTM. In convlstm_cell(), we first outline what must occur to a unmarried remark; then in convlstm(), we increase the recurrence common sense.

After we’re executed, we create a dummy dataset, as reduced-to-the-essentials as may also be. With extra advanced datasets, even synthetic ones, likelihood is that that if we don’t see any coaching development, there are loads of probable explanations. We wish a sanity verify that, if failed, leaves no excuses. Reasonable programs are left to long term posts.

A unmarried step: convlstm_cell

Our convlstm_cell’s constructor takes arguments input_dim , hidden_dim, and bias, identical to a torch LSTM Mobile.

However we’re processing two-dimensional enter knowledge. As a substitute of the standard affine mixture of recent enter and former state, we use a convolution of kernel dimension kernel_size. Inside of convlstm_cell, it’s self$conv that looks after this.

Observe how the channels measurement, which within the authentic enter knowledge would correspond to other variables, is creatively used to consolidate 4 convolutions into one: Every channel output will likely be handed to only one of the most 4 mobile gates. As soon as in ownership of the convolution output, ahead() applies the gate common sense, ensuing within the two kinds of states it must ship again to the caller.

library(torch)
library(zeallot)

convlstm_cell <- nn_module(
  
  initialize = serve as(input_dim, hidden_dim, kernel_size, bias) {
    
    self$hidden_dim <- hidden_dim
    
    padding <- kernel_size %/% 2
    
    self$conv <- nn_conv2d(
      in_channels = input_dim + self$hidden_dim,
      # for each and every of enter, omit, output, and mobile gates
      out_channels = 4 * self$hidden_dim,
      kernel_size = kernel_size,
      padding = padding,
      bias = bias
    )
  },
  
  ahead = serve as(x, prev_states) {

    c(h_prev, c_prev) %<-% prev_states
    
    mixed <- torch_cat(record(x, h_prev), dim = 2)  # concatenate alongside channel axis
    combined_conv <- self$conv(mixed)
    c(cc_i, cc_f, cc_o, cc_g) %<-% torch_split(combined_conv, self$hidden_dim, dim = 2)
    
    # enter, omit, output, and mobile gates (akin to torch's LSTM)
    i <- torch_sigmoid(cc_i)
    f <- torch_sigmoid(cc_f)
    o <- torch_sigmoid(cc_o)
    g <- torch_tanh(cc_g)
    
    # mobile state
    c_next <- f * c_prev + i * g
    # hidden state
    h_next <- o * torch_tanh(c_next)
    
    record(h_next, c_next)
  },
  
  init_hidden = serve as(batch_size, top, width) {
    
    record(
      torch_zeros(batch_size, self$hidden_dim, top, width, software = self$conv$weight$software),
      torch_zeros(batch_size, self$hidden_dim, top, width, software = self$conv$weight$software))
  }
)

Now convlstm_cell must be known as for each and every time step. That is executed through convlstm.

Iteration over the years steps: convlstm

A convlstm would possibly include a number of layers, identical to a torch LSTM. For each and every layer, we’re in a position to specify hidden and kernel sizes in my opinion.

All over initialization, each and every layer will get its personal convlstm_cell. On name, convlstm executes two loops. The outer one iterates over layers. On the finish of each and every iteration, we retailer the ultimate pair (hidden state, mobile state) for later reporting. The internal loop runs over enter sequences, calling convlstm_cell at each and every time step.

We additionally stay observe of intermediate outputs, so we’ll be capable to go back your complete record of hidden_states noticed all over the method. In contrast to a torch LSTM, we do that for each and every layer.

convlstm <- nn_module(
  
  # hidden_dims and kernel_sizes are vectors, with one part for each and every layer in n_layers
  initialize = serve as(input_dim, hidden_dims, kernel_sizes, n_layers, bias = TRUE) {
 
    self$n_layers <- n_layers
    
    self$cell_list <- nn_module_list()
    
    for (i in 1:n_layers) {
      cur_input_dim <- if (i == 1) input_dim else hidden_dims[i - 1]
      self$cell_list$append(convlstm_cell(cur_input_dim, hidden_dims[i], kernel_sizes[i], bias))
    }
  },
  
  # we all the time suppose batch-first
  ahead = serve as(x) {
    
    c(batch_size, seq_len, num_channels, top, width) %<-% x$dimension()
   
    # initialize hidden states
    init_hidden <- vector(mode = "record", duration = self$n_layers)
    for (i in 1:self$n_layers) {
      init_hidden[[i]] <- self$cell_list[[i]]$init_hidden(batch_size, top, width)
    }
    
    # record containing the outputs, of duration seq_len, for each and every layer
    # this is equal to h, at each and every step within the series
    layer_output_list <- vector(mode = "record", duration = self$n_layers)
    
    # record containing the final states (h, c) for each and every layer
    layer_state_list <- vector(mode = "record", duration = self$n_layers)

    cur_layer_input <- x
    hidden_states <- init_hidden
    
    # loop over layers
    for (i in 1:self$n_layers) {
      
      # each and every layer's hidden state begins from 0 (non-stateful)
      c(h, c) %<-% hidden_states[[i]]
      # outputs, of duration seq_len, for this residue
      # equivalently, record of h states for each and every time step
      output_sequence <- vector(mode = "record", duration = seq_len)
      
      # loop over the years steps
      for (t in 1:seq_len) {
        c(h, c) %<-% self$cell_list[[i]](cur_layer_input[ , t, , , ], record(h, c))
        # stay observe of output (h) for each and every time step
        # h has dim (batch_size, hidden_size, top, width)
        output_sequence[[t]] <- h
      }

      # stack hs all the time steps over seq_len measurement
      # stacked_outputs has dim (batch_size, seq_len, hidden_size, top, width)
      # similar as enter to ahead (x)
      stacked_outputs <- torch_stack(output_sequence, dim = 2)
      
      # go the record of outputs (hs) to subsequent layer
      cur_layer_input <- stacked_outputs
      
      # stay observe of record of outputs or this residue
      layer_output_list[[i]] <- stacked_outputs
      # stay observe of final state for this residue
      layer_state_list[[i]] <- record(h, c)
    }
 
    record(layer_output_list, layer_state_list)
  }
    
)

Calling the convlstm

Let’s see the enter layout anticipated through convlstm, and the best way to get entry to its other outputs.

Right here is an appropriate enter tensor.

# batch_size, seq_len, channels, top, width
x <- torch_rand(c(2, 4, 3, 16, 16))

First we employ a unmarried layer.

fashion <- convlstm(input_dim = 3, hidden_dims = 5, kernel_sizes = 3, n_layers = 1)

c(layer_outputs, layer_last_states) %<-% fashion(x)

We get again a listing of duration two, which we instantly cut up up into the 2 kinds of output returned: intermediate outputs from all layers, and last states (of each sorts) for the final layer.

With only a unmarried layer, layer_outputs[[1]]holds all the layer’s intermediate outputs, stacked on measurement two.

dim(layer_outputs[[1]])
# [1]  2  4  5 16 16

layer_last_states[[1]]is a listing of tensors, the primary of which holds the only layer’s last hidden state, and the second one, its last mobile state.

dim(layer_last_states[[1]][[1]])
# [1]  2  5 16 16
dim(layer_last_states[[1]][[2]])
# [1]  2  5 16 16

For comparability, that is how go back values search for a multi-layer structure.

fashion <- convlstm(input_dim = 3, hidden_dims = c(5, 5, 1), kernel_sizes = rep(3, 3), n_layers = 3)
c(layer_outputs, layer_last_states) %<-% fashion(x)

# for each and every layer, tensor of dimension (batch_size, seq_len, hidden_size, top, width)
dim(layer_outputs[[1]])
# 2  4  5 16 16
dim(layer_outputs[[3]])
# 2  4  1 16 16

# record of two tensors for each and every layer
str(layer_last_states)
# Listing of three
#  $ :Listing of two
#   ..$ :Go with the flow [1:2, 1:5, 1:16, 1:16]
#   ..$ :Go with the flow [1:2, 1:5, 1:16, 1:16]
#  $ :Listing of two
#   ..$ :Go with the flow [1:2, 1:5, 1:16, 1:16]
#   ..$ :Go with the flow [1:2, 1:5, 1:16, 1:16]
#  $ :Listing of two
#   ..$ :Go with the flow [1:2, 1:1, 1:16, 1:16]
#   ..$ :Go with the flow [1:2, 1:1, 1:16, 1:16]

# h, of dimension (batch_size, hidden_size, top, width)
dim(layer_last_states[[3]][[1]])
# 2  1 16 16

# c, of dimension (batch_size, hidden_size, top, width)
dim(layer_last_states[[3]][[2]])
# 2  1 16 16

Now we need to sanity-check this module with the simplest-possible dummy knowledge.

Sanity-checking the convlstm

We generate black-and-white “motion pictures” of diagonal beams successively translated in house.

Every series is composed of six time steps, and each and every beam of six pixels. Only a unmarried series is created manually. To create that one series, we commence from a unmarried beam:

library(torchvision)

beams <- vector(mode = "record", duration = 6)
beam <- torch_eye(6) %>% nnf_pad(c(6, 12, 12, 6)) # left, proper, peak, backside
beams[[1]] <- beam

The use of torch_roll() , we create a development the place this beam strikes up diagonally, and stack the person tensors alongside the timesteps measurement.

for (i in 2:6) {
  beams[[i]] <- torch_roll(beam, c(-(i-1),i-1), c(1, 2))
}

init_sequence <- torch_stack(beams, dim = 1)

That’s a unmarried series. Because of torchvision::transform_random_affine(), we virtually without problems produce a dataset of 100 sequences. Shifting beams get started at random issues within the spatial body, however all of them proportion that upward-diagonal movement.

sequences <- vector(mode = "record", duration = 100)
sequences[[1]] <- init_sequence

for (i in 2:100) {
  sequences[[i]] <- transform_random_affine(init_sequence, levels = 0, translate = c(0.5, 0.5))
}

enter <- torch_stack(sequences, dim = 1)

# upload channels measurement
enter <- enter$unsqueeze(3)
dim(enter)
# [1] 100   6  1  24  24

That’s it for the uncooked knowledge. Now we nonetheless want a dataset and a dataloader. Of the six time steps, we use the primary 5 as enter and take a look at to are expecting the final one.

dummy_ds <- dataset(
  
  initialize = serve as(knowledge) {
    self$knowledge <- knowledge
  },
  
  .getitem = serve as(i) {
    record(x = self$knowledge[i, 1:5, ..], y = self$knowledge[i, 6, ..])
  },
  
  .duration = serve as() {
    nrow(self$knowledge)
  }
)

ds <- dummy_ds(enter)
dl <- dataloader(ds, batch_size = 100)

Here’s a tiny-ish convLSTM, educated for movement prediction:

fashion <- convlstm(input_dim = 1, hidden_dims = c(64, 1), kernel_sizes = c(3, 3), n_layers = 2)

optimizer <- optim_adam(fashion$parameters)

num_epochs <- 100

for (epoch in 1:num_epochs) {
  
  fashion$educate()
  batch_losses <- c()
  
  for (b in enumerate(dl)) {
    
    optimizer$zero_grad()
    
    # last-time-step output from final layer
    preds <- fashion(b$x)[[2]][[2]][[1]]
  
    loss <- nnf_mse_loss(preds, b$y)
    batch_losses <- c(batch_losses, loss$merchandise())
    
    loss$backward()
    optimizer$step()
  }
  
  if (epoch %% 10 == 0)
    cat(sprintf("nEpoch %d, coaching loss:%3fn", epoch, imply(batch_losses)))
}

Epoch 10, coaching loss:0.008522

Epoch 20, coaching loss:0.008079

Epoch 30, coaching loss:0.006187

Epoch 40, coaching loss:0.003828

Epoch 50, coaching loss:0.002322

Epoch 60, coaching loss:0.001594

Epoch 70, coaching loss:0.001376

Epoch 80, coaching loss:0.001258

Epoch 90, coaching loss:0.001218

Epoch 100, coaching loss:0.001171

Loss decreases, however that during itself isn’t a ensure the fashion has realized anything else. Has it? Let’s investigate cross-check its forecast for the first actual series and spot.

For printing, I’m zooming in at the related area within the 24×24-pixel body. Here’s the bottom reality for time step six:

0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0
0  0  1  0  0  0  0  0  0  0
0  0  0  1  0  0  0  0  0  0
0  0  0  0  1  0  0  0  0  0
0  0  0  0  0  1  0  0  0  0
0  0  0  0  0  0  1  0  0  0
0  0  0  0  0  0  0  1  0  0
0  0  0  0  0  0  0  0  0  0
0  0  0  0  0  0  0  0  0  0

And this is the forecast. This doesn’t glance unhealthy in any respect, given there used to be neither experimentation nor tuning concerned.

       [,1]  [,2]  [,3]  [,4]  [,5]  [,6]  [,7]  [,8]  [,9] [,10]
 [1,]  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00     0
 [2,] -0.02  0.36  0.01  0.06  0.00  0.00  0.00  0.00  0.00     0
 [3,]  0.00 -0.01  0.71  0.01  0.06  0.00  0.00  0.00  0.00     0
 [4,] -0.01  0.04  0.00  0.75  0.01  0.06  0.00  0.00  0.00     0
 [5,]  0.00 -0.01 -0.01 -0.01  0.75  0.01  0.06  0.00  0.00     0
 [6,]  0.00  0.01  0.00 -0.07 -0.01  0.75  0.01  0.06  0.00     0
 [7,]  0.00  0.01 -0.01 -0.01 -0.07 -0.01  0.75  0.01  0.06     0
 [8,]  0.00  0.00  0.01  0.00  0.00 -0.01  0.00  0.71  0.00     0
 [9,]  0.00  0.00  0.00  0.01  0.01  0.00  0.03 -0.01  0.37     0
[10,]  0.00  0.00  0.00  0.00  0.00  0.00 -0.01 -0.01 -0.01     0

This must suffice for a sanity verify. In the event you made it until the tip, thank you in your persistence! In the most productive case, you’ll be capable to follow this structure (or a an identical one) on your personal knowledge – however even supposing now not, I’m hoping you’ve loved studying about torch fashion coding and/or RNN weirdness

I, for one, am no doubt taking a look ahead to exploring convLSTMs on real-world issues within the close to long term. Thank you for studying!

Appendix

This appendix comprises the code used to create tables 1 and a couple of above.

Keras

LSTM

library(keras)

# batch of three, with 4 time steps each and every and a unmarried characteristic
enter <- k_random_normal(form = c(3L, 4L, 1L))
enter

# default args
# go back form = (batch_size, gadgets)
lstm <- layer_lstm(
  gadgets = 1,
  kernel_initializer = initializer_constant(price = 1),
  recurrent_initializer = initializer_constant(price = 1)
)
lstm(enter)

# return_sequences = TRUE
# go back form = (batch_size, time steps, gadgets)
#
# word how for each and every merchandise within the batch, the worth for time step 4 equals that received above
lstm <- layer_lstm(
  gadgets = 1,
  return_sequences = TRUE,
  kernel_initializer = initializer_constant(price = 1),
  recurrent_initializer = initializer_constant(price = 1)
  # bias is through default initialized to 0
)
lstm(enter)

# return_state = TRUE
# go back form = record of:
#                - outputs, of form: (batch_size, gadgets)
#                - "reminiscence states" for the final time step, of form: (batch_size, gadgets)
#                - "lift states" for the final time step, of form: (batch_size, gadgets)
#
# word how the primary and 2nd record pieces are equivalent!
lstm <- layer_lstm(
  gadgets = 1,
  return_state = TRUE,
  kernel_initializer = initializer_constant(price = 1),
  recurrent_initializer = initializer_constant(price = 1)
)
lstm(enter)

# return_state = TRUE, return_sequences = TRUE
# go back form = record of:
#                - outputs, of form: (batch_size, time steps, gadgets)
#                - "reminiscence" states for the final time step, of form: (batch_size, gadgets)
#                - "lift states" for the final time step, of form: (batch_size, gadgets)
#
# word how once more, the "reminiscence" state present in record merchandise 2 suits the final-time step outputs reported in merchandise 1
lstm <- layer_lstm(
  gadgets = 1,
  return_sequences = TRUE,
  return_state = TRUE,
  kernel_initializer = initializer_constant(price = 1),
  recurrent_initializer = initializer_constant(price = 1)
)
lstm(enter)

GRU

# default args
# go back form = (batch_size, gadgets)
gru <- layer_gru(
  gadgets = 1,
  kernel_initializer = initializer_constant(price = 1),
  recurrent_initializer = initializer_constant(price = 1)
)
gru(enter)

# return_sequences = TRUE
# go back form = (batch_size, time steps, gadgets)
#
# word how for each and every merchandise within the batch, the worth for time step 4 equals that received above
gru <- layer_gru(
  gadgets = 1,
  return_sequences = TRUE,
  kernel_initializer = initializer_constant(price = 1),
  recurrent_initializer = initializer_constant(price = 1)
)
gru(enter)

# return_state = TRUE
# go back form = record of:
#    - outputs, of form: (batch_size, gadgets)
#    - "reminiscence" states for the final time step, of form: (batch_size, gadgets)
#
# word how the record pieces are equivalent!
gru <- layer_gru(
  gadgets = 1,
  return_state = TRUE,
  kernel_initializer = initializer_constant(price = 1),
  recurrent_initializer = initializer_constant(price = 1)
)
gru(enter)

# return_state = TRUE, return_sequences = TRUE
# go back form = record of:
#    - outputs, of form: (batch_size, time steps, gadgets)
#    - "reminiscence states" for the final time step, of form: (batch_size, gadgets)
#
# word how once more, the "reminiscence state" present in record merchandise 2 suits the final-time-step outputs reported in merchandise 1
gru <- layer_gru(
  gadgets = 1,
  return_sequences = TRUE,
  return_state = TRUE,
  kernel_initializer = initializer_constant(price = 1),
  recurrent_initializer = initializer_constant(price = 1)
)
gru(enter)

torch

LSTM (non-stacked structure)

library(torch)

# batch of three, with 4 time steps each and every and a unmarried characteristic
# we can specify batch_first = TRUE when growing the LSTM
enter <- torch_randn(c(3, 4, 1))
enter

# default args
# go back form = (batch_size, gadgets)
#
# word: there may be an extra argument num_layers that lets use to specify a stacked LSTM - successfully composing two LSTM modules
# default for num_layers is 1 even though 
lstm <- nn_lstm(
  input_size = 1, # choice of enter options
  hidden_size = 1, # choice of hidden (and output!) options
  batch_first = TRUE # for simple comparison with Keras
)

nn_init_constant_(lstm$weight_ih_l1, 1)
nn_init_constant_(lstm$weight_hh_l1, 1)
nn_init_constant_(lstm$bias_ih_l1, 0)
nn_init_constant_(lstm$bias_hh_l1, 0)

# returns a listing of duration 2, particularly
#   - outputs, of form (batch_size, time steps, hidden_size) - given we specified batch_first
#       Observe 1: If this can be a stacked LSTM, those are the outputs from the final layer simplest.
#               For our present goal, that is inappropriate, as we are proscribing ourselves to single-layer LSTMs.
#       Observe 2: hidden_size this is similar to gadgets in Keras - each specify choice of options
#  - record of:
#    - hidden state for the final time step, of form (num_layers, batch_size, hidden_size)
#    - mobile state for the final time step, of form (num_layers, batch_size, hidden_size)
#      Observe 3: For a single-layer LSTM, the hidden states are already supplied within the first record merchandise.

lstm(enter)

GRU (non-stacked structure)

# default args
# go back form = (batch_size, gadgets)
#
# word: there may be an extra argument num_layers that lets use to specify a stacked GRU - successfully composing two GRU modules
# default for num_layers is 1 even though 
gru <- nn_gru(
  input_size = 1, # choice of enter options
  hidden_size = 1, # choice of hidden (and output!) options
  batch_first = TRUE # for simple comparison with Keras
)

nn_init_constant_(gru$weight_ih_l1, 1)
nn_init_constant_(gru$weight_hh_l1, 1)
nn_init_constant_(gru$bias_ih_l1, 0)
nn_init_constant_(gru$bias_hh_l1, 0)

# returns a listing of duration 2, particularly
#   - outputs, of form (batch_size, time steps, hidden_size) - given we specified batch_first
#       Observe 1: If this can be a stacked GRU, those are the outputs from the final layer simplest.
#               For our present goal, that is inappropriate, as we are proscribing ourselves to single-layer GRUs.
#       Observe 2: hidden_size this is similar to gadgets in Keras - each specify choice of options
#  - record of:
#    - hidden state for the final time step, of form (num_layers, batch_size, hidden_size)
#    - mobile state for the final time step, of form (num_layers, batch_size, hidden_size)
#       Observe 3: For a single-layer GRU, those values are already supplied within the first record merchandise.
gru(enter)