For a more compete description of attention, see Yoav Goldberg's excellent book here: TensorFlow's Tutorial about Seq2Seq: a good tutorial of an seq2seq model with attention being implemented. Contains major references. How does the multi-head attention mechanism work in deep learning? The title of the paper is: "A Primer on Neural Network Models for Natural Language Processing". It is available for free on ArXiv and was last dated 2015. It is a technical report or tutorial more than a paper and provides a comprehensive introduction to Deep Learning methods for Natural Language Processing (NLP), intended for researchers Best Practices for Document Classification with Deep Learning Photo by storebukkebruse, Yoav Goldberg, in his primer on deep learning for natural language processing, comments that neural networks in general offer better performance than classical linear classifiers, especially when used Practical Neural Networks for NLP Chris Dyer, Yoav Goldberg, Graham Neubig November 1, 2016 EMNLP. Neural Nets and Language • This tutorial is about one framework that lets you use the power of neural nets without abandoning familiar NLP algorithms. The tutorial assumes basic familiarity with neural network models, and will focus on how to implement them with DyNet. We will explore several common NLP models and their implementation using the DyNet package. Tutorial Slides - Tutorial Additional Material. Short Bio. Yoav Goldberg has been working in natural language processing for over a Yoav Goldberg. A Primer on Neural Network Models for Natural Language Processing ; Ian Goodfellow, Yoshua Bengio, and Aaron Courville. All class assignments will be in Python (using numpy and tensorflow). There is a tutorial here for those who aren't as familiar with Python. If you have a However, as shown by Gail Weiss & Yoav Goldberg & Eran Yahav, the LSTM is "strictly stronger" than the GRU as it can easily perform unbounded counting, while the GRU cannot.