bayesian layers

[neginjv]

@dustinvtran
Hi, I have a question. Can I build bayesian layers with sequential models with lstm using edward2?
I mean building sequential model with LSTMCellReparameterization.
If yes please help me. I dont know how to build a sequential model with LSTMCellReparameterization layers.

[neginjv]

@dustinvtran Please help me. I want to build a sequential model with LSTMCellReparameterization.
I really need help.

[dustinvtran]

@dusenberrymw

There's a minimal code snippet in the paper. It should be plug-and-play with any LSTM example you currently have.

[dusenberrymw]

Here's a runnable snippet using the reparameterized LSTM cell. As Dustin said, ed.layers.LSTMCellReparameterization is a drop-in replacement for tf.keras.layers.LSTMCell, provided that model.losses is used as part of the loss.

import tensorflow.compat.v2 as tf
import edward2 as ed

num_examples = 2
num_timesteps = 3
input_dim = 4
rnn_dim = 10

inputs = tf.random.normal([num_examples, num_timesteps, input_dim])
labels = tf.random.normal([num_examples])
cell = ed.layers.LSTMCellReparameterization(rnn_dim)
model = tf.keras.Sequential([
  tf.keras.layers.RNN(cell),
  tf.keras.layers.Dense(1)])
optimizer = tf.keras.optimizers.Adam(0.001)

for i in range(10):
  with tf.GradientTape() as tape:
    outputs = model(inputs)
    nll = tf.reduce_mean((labels-outputs)**2)
    kl = sum(model.losses)
    loss = nll + kl
  grads = tape.gradient(loss, model.variables)
  grads_and_vars = zip(grads, model.variables)
  optimizer.apply_gradients(grads_and_vars)
  print(f"Loss at step {i}: {loss}")

Loss at step 0: 1412.79296875
Loss at step 1: 1412.1939697265625
Loss at step 2: 1411.6484375
Loss at step 3: 1411.123779296875
Loss at step 4: 1410.535888671875
Loss at step 5: 1410.012451171875
Loss at step 6: 1409.4478759765625
Loss at step 7: 1408.9200439453125
Loss at step 8: 1408.29638671875
Loss at step 9: 1407.8123779296875

[neginjv]

@dusenberrymw Hi. Thanks a lot for your help. One another question, why the value of loss function is so high in this approach?

[neginjv]

@dusenberrymw Hi. I have some questions. What is the benefit of this LSTMCellReparameterization compared to ordinary lstm? Because I think the value of loss function is so high in this approach. And what is the difference between LSTMCellReparameterization and LSTMCellFlipout?
How can I calculate accuracy in every epoch?

[neginjv]

@dusenberrymw Hi, could you please help me with my questions? I really need them for my master thesis and there isn't so much information about that questions anywhere else. Thanks a lot.

[YutianPangASU]

Here's a runnable snippet using the reparameterized LSTM cell. As Dustin said, ed.layers.LSTMCellReparameterization is a drop-in replacement for tf.keras.layers.LSTMCell, provided that model.losses is used as part of the loss.

import tensorflow.compat.v2 as tf
import edward2 as ed

num_examples = 2
num_timesteps = 3
input_dim = 4
rnn_dim = 10

inputs = tf.random.normal([num_examples, num_timesteps, input_dim])
labels = tf.random.normal([num_examples])
cell = ed.layers.LSTMCellReparameterization(rnn_dim)
model = tf.keras.Sequential([
  tf.keras.layers.RNN(cell),
  tf.keras.layers.Dense(1)])
optimizer = tf.keras.optimizers.Adam(0.001)

for i in range(10):
  with tf.GradientTape() as tape:
    outputs = model(inputs)
    nll = tf.reduce_mean((labels-outputs)**2)
    kl = sum(model.losses)
    loss = nll + kl
  grads = tape.gradient(loss, model.variables)
  grads_and_vars = zip(grads, model.variables)
  optimizer.apply_gradients(grads_and_vars)
  print(f"Loss at step {i}: {loss}")

Loss at step 0: 1412.79296875
Loss at step 1: 1412.1939697265625
Loss at step 2: 1411.6484375
Loss at step 3: 1411.123779296875
Loss at step 4: 1410.535888671875
Loss at step 5: 1410.012451171875
Loss at step 6: 1409.4478759765625
Loss at step 7: 1408.9200439453125
Loss at step 8: 1408.29638671875
Loss at step 9: 1407.8123779296875

@dustinvtran
Is this running with TF2.0? I'm getting some type error in my code,

lstm_cell = ed.layers.LSTMCellReparameterization(self.lstm_hidden_dim, activation='tanh')
lstm_out = tf.keras.layers.RNN(cell=lstm_cell, return_sequences=True)(lstm_in)

{TypeError}Failed to convert object of type <class 'tuple'> to Tensor. Contents: (Dimension(3), 256). Consider casting elements to a supported type.

The error comes from the input tensor lstm_in. I'm using TF1.14, is this causing trouble?

[YutianPangASU]

Here's a runnable snippet using the reparameterized LSTM cell. As Dustin said, ed.layers.LSTMCellReparameterization is a drop-in replacement for tf.keras.layers.LSTMCell, provided that model.losses is used as part of the loss.

import tensorflow.compat.v2 as tf
import edward2 as ed

num_examples = 2
num_timesteps = 3
input_dim = 4
rnn_dim = 10

inputs = tf.random.normal([num_examples, num_timesteps, input_dim])
labels = tf.random.normal([num_examples])
cell = ed.layers.LSTMCellReparameterization(rnn_dim)
model = tf.keras.Sequential([
  tf.keras.layers.RNN(cell),
  tf.keras.layers.Dense(1)])
optimizer = tf.keras.optimizers.Adam(0.001)

for i in range(10):
  with tf.GradientTape() as tape:
    outputs = model(inputs)
    nll = tf.reduce_mean((labels-outputs)**2)
    kl = sum(model.losses)
    loss = nll + kl
  grads = tape.gradient(loss, model.variables)
  grads_and_vars = zip(grads, model.variables)
  optimizer.apply_gradients(grads_and_vars)
  print(f"Loss at step {i}: {loss}")

Loss at step 0: 1412.79296875
Loss at step 1: 1412.1939697265625
Loss at step 2: 1411.6484375
Loss at step 3: 1411.123779296875
Loss at step 4: 1410.535888671875
Loss at step 5: 1410.012451171875
Loss at step 6: 1409.4478759765625
Loss at step 7: 1408.9200439453125
Loss at step 8: 1408.29638671875
Loss at step 9: 1407.8123779296875

@dustinvtran
Is this running with TF2.0? I'm getting some type error in my code,

lstm_cell = ed.layers.LSTMCellReparameterization(self.lstm_hidden_dim, activation='tanh')
lstm_out = tf.keras.layers.RNN(cell=lstm_cell, return_sequences=True)(lstm_in)

{TypeError}Failed to convert object of type <class 'tuple'> to Tensor. Contents: (Dimension(3), 256). Consider casting elements to a supported type.

The error comes from the input tensor lstm_in. I'm using TF1.14, is this causing trouble?

@dustinvtran
I upgraded everything to TF2.0 and it works. However, I still want to ask if there is a way you could let me use TF1.14 with the lstmReparameterization layer? Like modify the source code somewhere?