# Multiplicative LSTM (Workshop Track in ICLR 2017)をTensorFlowで実装した

### ICLR2017のレビューコメント

ここより引用する：

Comment: The paper presents a new way of doing multiplicative / tensored recurrent weights in RNNs. The multiplicative weights are input dependent. Results are presented on language modeling (PTB and Hutter). We found the paper to be clearly written, and the idea well motivated. However, as pointed out by the reviewers, the results were not state of the art. We feel that is that this is because the authors did not make a strong attempt at regularizing the training. Better results on a larger set of tasks would have probably made this paper easier to accept.

Pros:
- interesting idea, and reasonable results
Cons:
- only shown on language modeling tasks
- results were not very strong, when compared to other methods (which typically used strong regularization and training like batch normalization etc).
- reviewers did not find the experiments convincing enough, and felt that a fair comparison would be to compare with dynamic weights on the competing RNNs.

• 他手法と比較したときに、有効性を示すには実験結果がまだちょっと弱い
• 評価タスクが言語モデルだけで行われており、手法の比較もまだ足りない

### 先行研究 : Multiplicative RNN (mRNN) *1

１時刻前の隠れ状態に掛けられる重み行列$W_{hh}$を（テンソル的に）分解し、当該時刻の入力に依存するように拡張したもの。character-levelの言語モデリングに有効性が示されている。また、生成モデルとして用いて文章を生成した実験結果から、mRNNが高次の言語的構造、文法的構造を捉えていることが示された。

### 提案手法

$W_{mx}$と$W_{mh}$の分、パラメータ数は増える。先のレビューの通り、性能が劇的に向上するというわけではないので、タスクを選ぶネットワーク構造なのかもしれない。

### 参考文献

1. I. Sutskever, J. Martens, and G. E. Hinton. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1017–1024, 2011.

*1:参考文献 1