Skip to main content

Bias and Variance trade off

What is Bias and Variance?


The figure on the left side classifies the circles and the cross "Just right". The second figure includes the cross and fails to classify optimally this is called as High Bias which is also an indication of Underfitting the data. The figure on the right side does something weird and classifies circles and the cross which is way on the opposite side, this classifier fails terribly and ends up overfitting the data. This is called as High Variance or Overfitting.

The concepts of Bias and Variance is very important to understand because looking at High/ Low Bias and Variance we will be able to predict if our training data error is at fault or development data error. Analysis of these errors we will get an insight into HOW well are we fitting our data to the model. To understand Bias and Variance we need to understand the training set error and dev set error.

Let's talk more about High Bias 
For simplification, let's also assume that the human error is: 0 % and optimal Bayes error is: 0 %.
Now let's see a few cases where we analyze Train Set Error and Dev Set Error with respect to the Human Error.
Case 1: Train Set Error: 2 % & Dev Set Error is: 20 % 
In Case 1, when your Train set error(2 %)is minimal but your Dev set error(20%) is too large, it means that your training set data is not generalizing well. The data is overfitting. We call this case High Variance. 
Case 2: Train Set Error: 16 % & Dev Set Error: 17 %
In Case 2, when your Train set error(16 %) is worse than the Human error(0 %). Then we can conclude that the data is underfitting. We call this High Bias.
Case 3: Train Set Error: 16 %  & Dev Set Error: 31 %
In Case 3, when the Train set error(16 %) and Dev set error(31 %) both are worse than the Human error and Optimal Bayes error. We call this case as High Bias and High Variance.
Case 4: Train Set Error: 0.5 % & Dev Set Error: 1 %
In Case 4, when the Train Set Error and Dev Set Error are absolutely minimum, we call this Low Bias and Low Variance.

So, the takeaway from this description is that train set error gives you a sense of how well you are fitting your data to the model, so it will tell you if you have a bias problem. And then looking at the difference between train set error and dev set error you can get a sense of how bad is your Variance problem.

How to solve Bias and Variance problem?
High Bias:  
  • Get a Bigger Network.
  • Train it for a longer period of time.
  • Put more Hidden Layers.
  • Put more Hidden Units.
  • Try different Advanced Optimization Algorithms.
High Variance:
  • Most Important Try Regularization(L2 Regularization, Dropout Regularization, Early Stopping, Data Augmentation)
  • Get more Data, this option could be expensive.
  • Apply a better Neural Network Architecture.

Comments

Post a Comment

Popular posts from this blog

Convolutional Nets Part 1

Why ConvNets? ConvNets are used for Image Analysis. ConvNets reduces a given image into a form which is easier to work on without losing its important features, Which helps in prediction. ConvNets are vastly prefered due to their 1) Parameter Sharing property & 2) Sparsity of Connections. 1)Parameter Sharing: A feature detector (edges, shapes) filter could be used for multiple parts of the same image. 2) Sparsity of Connections: In each layer, each output value depends only on a small number of inputs. What are ConvNets? ConvNets, also known as Convolutional Neural Networks are Neural Nets with Convolutional Layers between them. ConvNets also known as Feed Forward networks. The convolutional layers are made of filters. These filters detect various patterns in an image. The simple filters detect simple features such as Horizontal edges, vertical edges, circles and so on. The filters in the deeper layers of CNN can detect more complex patterns. These filters are initial...
Post a Comment
Read more