Intuitive Shorts

Search This Blog

Transformer Based Models

April 12, 2021

Get link
Facebook
X
Pinterest
Email
Other Apps

Fig1: Types of Transformer based models

Get link
Facebook
X
Pinterest
Email
Other Apps

Comments

Stateless Vs Stateful LSTMs

In machine learning, it is generally assumed that the training samples are Independent and Identically Distributed (IID). As far as the sequence data is concerned, this isn't always true. If the sequence values have temporal dependence among them, such as Time Series data, the IID assumption fails. The sequence modelling algorithms hence come in two flavours, Stateless and Stateful, depending upon the architecture used while training. Following is a discussion using LSTM as an example, but the notion is applicable to other variants as well namely RNN, GRU etc. Stateless This architecture is used when the IID assumption holds. While creating batches for training, this means that there is no inter-relationship across the batches and each batch is independent from one another. The typical training process in a stateless LSTM architecture is shown below: The way these the two architectures differ is the manner in which the states (cell and hidden states...

[Project] Speech emotion based Face generation using Conditional GAN

Links : [ Code ], [ Video ] The aim of this project is to develop and combine two standalone deep learning techniques - Speech Emotion Recognition (SER) and Conditional GAN (cGAN) to develop an end-to-end system for generating human faces using cGAN based on emotions identified from human speech using SER. The project pipeline can be summarized as follows: Project Pipeline : Emotion identified using SER is used to condition face generation by cGAN The project demo is shown below: Dataset:

Softmax Temperature

Temperature is a hyperparameter of LSTMs (and neural networks generally) used to control the randomness of predictions by scaling the logits before applying softmax. Temperature scaling has been widely used to improve performance for NLP tasks that utilize Softmax decision layer. For explaining its utility, we will consider the case of Natural Language Generation, wherein we need to generate text by sampling out novel sequences from the language model (using the decoder part of the seq-to-seq architecture). At each time step in the decoding phase, we need to predict a token, which is done by sampling from a softmax distribution (over the vocabulary) using one of the sampling techniques. In short, once the logits are obtained, the quality and the diversity of the predictions is controlled by the softmax distribution and the sampling technique applied thereupon. This article is about tweaking the softmax distribution to control how diverse and novel the predictions are. Th...

Powered by Blogger

Transformer Based Models

Comments

Post a Comment