EUNN-tensorflow

Unitary neural network is able to solve gradient vanishing and gradient explosion problem and help learning long term correlation. Unitary RNN is promising to replace LSTM in multiple tasks. EUNN is an efficient unitary architecture based on SU(2) group. This repository contains an implementation of Efficient Unitary Neural Network(EUNN) and its Recurrent Neural Network implementation(EURNN).

If you find this work useful, please cite arXiv:1612.05231. The current implementation is developed by Ivan Ivanov.

The paper and slides of the current implementation are provided here.

Installation

requires TensorFlow 1.2.0

Demo

./demo.sh

Usage

Use EUNN in RNN

To use EURNN in your model, simply copy EUNN.py and modrelu.py files.

Then you can use EUNN in the same way you use built-in LSTM:

from EUNN import EUNNCell
cell = EUNNCell(n_hidden, capacity=2, FFT=False, comp=False)

Args:

• n_hidden: Integer.
• capacity: Optional. Integer. Only works for tunable style.
• FFT: Optional. Bool. If True, EUNN is set to FFT style. Default is False.
• comp: Optional. Bool. If True, EUNN is set to complex domain. Default is False.

Note:

• For complex domain, the data type should be tf.complex64
• For real domain, the data type should be tf.float32

Use EUNN in other applications

To use EUNN in your model, simply copy EUNN.py file.

Then you can use EUNN in the following way:

from EUNN import EUNN
output = EUNN(input, capacity=2, FFT=False, comp=False)

Args:

• input: 2D-Tensor.
• capacity: Optional. Integer. Only works for tunable style.
• FFT: Optional. Bool. If True, EUNN is set to FFT style. Default is False.
• comp: Optional. Bool. If True, EUNN is set to complex domain. Default is False.

Note:

• For complex domain, the data type should be tf.complex64
• For real domain, the data type should be tf.float32

Example tasks for EURNN

Two tasks for RNN in the paper are shown here. Use -h for more information

Copying Memory Task

requires: Model name (EUNN or LSTM);

optional parameters for the task:

delay time-T, number of iterations-I, batch size-B, hidden size-H;

optional parameters for EUNN:

capacity-L, complex or real-C, FFT style or tunable style-F.

Example:

python copying_task.py EUNN -T 100 -I 2000 -B 128 -H 128 -C True -F True

Pixel-Permuted MNIST Task

requires: Model name (EUNN or LSTM);

optional parameters for the task:

number of iterations-I, batch size-B, hidden size-H;

optional parameters for EUNN:

capacity-L, complex or real-C, FFT style or tunable style-F.

Example:

python mnist_task.py EUNN -I 2000 -B 128 -H 128 -L 4 -C True