Representation Learning at the Simons Institute

The Simons Institute for the Theory of Computing from the UC Berkeley has been presenting some talks on representation learning (RL) in this beginning of 2017. Representation Learning is an important field of research within current Machine Learning/Deep Learning complex strands of research. Indeed it is viewed by some top researchers as a promising field of activity where Artificial Intelligence (AI) may find further lines of improvement. It is mainly about the foundations of machine learning, though. But often the most important advancements in science & technology come from revising our established beliefs rather than pursuing further line of research in an already fully understood domain of study.

One such researcher is Ruslan Salakhutdinov, Carnegie Mellon University. In the talk I share below Dr. Salakhutdinov talks about his own research on RL, after introducing important other efforts like the one we can check in the paper Neural Semantic Encoders, with its GitHub repository, or the other nice effort Teaching Machines to Read and Comprehend, which can also be openly source checked here.

The talk is titled Representation Learning for Reading Comprehension, and it is about how current efforts in Representation Learning, namely trying to teach machines to better comprehend text, are unfolding.

I would like the reader to take a close attentive mindset to the way the speaker interacts with the audience, the quality of the questions and answers. This requires some prior knowledge of the material being discussed. The slides of the talk might be of help, where we can see the broader context of what Dr. Salakhutdinov is claiming.


The table above is one of the most interesting features of the talk, when Dr. Salakhudtinov considers the somewhat crazy pace of the development taking place in the field. The number of differentiated models for representation learning coming up on a weekly basis is short of mind-boggling, almost mindless additional capacity. The speaker’s own set of gaining models features just reasonably well in the picture . the bottom set of lines in the table.

Another important part of the talk concerned the topic of character representation and the word vs. character embedding by the models. Within the context of Natural Language Processing  (NLP) or machine translation these are cutting-edge issues. Recurrent neural networks (RNNs) such as LSTM (Long-term short memory) frameworks have been state-of-the-art in the field; we can confirm Dr. Salakhutdinov’s ideas and the way the incorporation of prior knowledge with the detailed integration of the techniques in the presented gaining models is achieved; again here the interaction with the audience is rewarding – notice here in the discussion how the speaker explains how the model of language representation does its job by treating a sentence as multiple sequences of connections/chains with the RNN’s hidden state update propagating the appropriate information through the network.


Finally the last good takeaway from this talk was the part on extractive question answering. Here the model manages to – based on available data – generate questions for later to get the right answer, a tough task according to the speaker. Here we learn that the machine actually demonstrate some semantic understanding of a whole paragraph or piece of text (might be disputable…). Starting from a standard generative adversarial net (GAN), the speaker then explain his (and his team) proposal of a Generative Domain-Adaptive Nets (GDANS) succinctly. From here the team started to think about variational auto-encoders to do better (transform)  sampling from the datasets/code.

featured and body text images: Representation Learning for Reading Comprehension


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s