Hello, welcome to my blog. In this post, I want to talk about
the Bias-Variance trade-off which is a very important topic in Machine
Learning. Before I do that let me lay the foundation. In my post on Linear
Regression, I said that the goal of a linear regression model is to find
parameters for our linear regression line that minimize the error between our
predictions and the actual observations.
This does not only apply to linear
regression; in fact that’s the goal of the majority of machine learning
techniques. We want to minimize the error between what we predict and what we
actually observe. This leads us to the 3 sources of error:- Irreducible error
- Bias error
- Variance error
Irreducible error refers to error that exists due to noise in
the data. This kind of error cannot be eliminated no matter how we try to
optimize the parameters of our machine learning algorithm. This is because the
function (i.e. the model) we use to estimate the relationship between the input
variable(s) and the output is only an approximation of the true (target) function
which is unknown. Therefore, we are always bound to have some error in our
predictions. The next two types of error will be the major focus of this post.
First, I will define two important terms:
BIAS
Bias are the simplifying assumptions made by an algorithm in
order to make the target function easier to learn. Machine Learning (ML)
algorithms with a high bias are inflexible, simple and may have few parameters.
ML algorithms with low bias are flexible, can be complex and potentially may
have lots of parameters. Examples of high bias ML algorithms are linear
regression and logistic regression.
VARIANCE
Variance refers to the amount the estimate of the target
function will change if a different data was used for training. Even though the
estimation of the target function will change for a different dataset, we do
not want the change to be too much for various datasets meaning that the model
is able to capture the underlying pattern between the input and output
variables. ML algorithms with high variance tend to be very flexible and are
able to capture subtle relationships in the training data. Examples of high
variance ML algorithms are decision trees, k-nearest neighbours and support
vector machines.
HOW DO THESE LEAD TO ERROR?
Bias Error: If a model has high bias it will
make prediction errors because it fits the relationship between the input and
output variables using a simple estimate of the target function which is rarely
the case in real-life. This simple estimate is bound to make errors if it is
used for prediction. This problem is called underfitting.
I like to think of this as a student who does not prepare well for an examination.
Clearly, this student is bound to fail the examination.
Variance Error: A model has high variance might
perform well on the training data because it is flexible enough to capture the
relationship between the input and output variables. Good right? Not really,
this is because if the model fits the training data too well it may perform
poorly on future data which is obviously a problem. This problem is called overfitting. Using the same student
analogy, a model with high variance is like a student who reads the study
material too well to the point of
cramming it. While he may pass a question based on the study material (training
data), he will fail questions that do not
originate from the study material which is likely for an examination (future
data). Like the previous student he is bound to also fail the examination.
BIAS-VARIANCE TRADE-OFF
We can see a pattern here, algorithms with high bias have low
variance while those with low bias have high variance. Concretely, as variance
increases bias decreases and vice versa. This is why it’s called a trade-off
because you rarely can have the best of both worlds which would be low bias and
low variance.
CONTROLLING BIAS AND VARIANCE
How do we control bias and variance? Below are some ways of
controlling bias and variance for some ML algorithms.
- Linear regression & Logistic regression: These algorithms have high bias and low variance but the trade-off can be altered using a technique called regularization. This is done by adding a tuning parameter λ to their respective cost functions. Increasing λ increases the bias (and reduces the variance); while decreasing λ reduces the bias (and increases the variance) of the model.
- K-Nearest Neighbours: This algorithm has low bias and high variance. This can be adjusted by tuning k which dictates how many neighbours that will contribute to our prediction. Increasing k increases the bias and reduces the variance of the model.
- Support Vector Machines: Like k-nearest neighbours, this algorithm also has low bias and high variance. This can be adjusted by tuning the C parameter that influences the number of violations of the margin allowed in the training data which increases the bias but decreases the variance.
Conclusion
Now you know what the bias-variance trade-off is. For every
problem you encounter in machine learning you will need to find a balance
between these two. If you are interested in learning more about this I suggest reading
Section 2.2.2 of the book An Introduction
to Statistical Learning.
Hope you enjoyed reading this post. If you have any questions
or suggestions please leave a comment and I will happy to attend to you. That
is all for now. Have a wonderful weekend. Cheers!!!