Image by Editor
# Introduction
One of the most difficult pieces of machine learning is not creating the model itself, but evaluating its performance.
A model might look excellent on a single train/test split, but fall apart when used in practice. The reason is that a single split tests the model only once, and that test set may not capture the full variability of the data it will face in the future. As a result, the model can appear better than it really is, leading to overfitting or misleadingly high scores. That’s where cross-validation comes in.
In this article, we’re going to break down cross-validation in plain English, provide reasons why it is more reliable than the hold-out method, and demonstrate how to use it with basic code and images.
# What is Cross-Validation?
Cross-validation is a machine learning validation procedure to evaluate the performance of a model using multiple subsets of data, as opposed to relying on only one subset. The basic idea behind this concept is to give every data point a chance to appear in the training set and testing set as part of determining the final performance. The model is therefore evaluated multiple times using different splits, and the performance measure you have chosen is then averaged.
Image by Author
The main advantage of cross-validation over a single train-test split is that cross-validation estimates performance more reliably, because it allows the performance of the model to be averaged across folds, smoothing out randomness in which points were set aside as a test set.
To put it simply, one test set could happen to include examples that lead to the model’s unusually high accuracy, or occur in such a way that, with a different mix of examples, it would lead to unusually low performance. In addition, cross-validation makes better use of our data, which is critical if you are working with small datasets. Cross-validation does not require you to waste your valuable information by setting a large part aside permanently. Instead, cross-validation means the same observation can play the train or test role at various times. In plain terms, your model takes multiple mini-exams, as opposed to one big test.
Image by Author
# The Most Common Types of Cross-Validation
There are different types of cross-validation, and here we take a look at the four most common.
// 1. k-Fold Cross-Validation
The most familiar method of cross-validation is k-fold cross-validation. In this method, the dataset is split into k equal parts, also known as folds. The model is trained on k-1 folds and tested on the fold that was left out. The process continues until every fold has been a test set one time. The scores from all the folds are averaged together to form a stable measure of the model’s accuracy.
For example, in the 5-fold cross-validation case, the dataset will be divided into five parts, and each part becomes the test set once before everything is averaged to calculate the final performance score.
Image by Author
// 2. Stratified k-Fold
When dealing with classification problems, where real-world datasets are often imbalanced, stratified k-fold cross-validation is preferred. In standard k-fold, we may happen to end up with a test fold with a highly skewed class distribution, for instance, if one of the test folds has very few or no class B instances. Stratified k-fold guarantees that all folds share approximately the same proportions of classes. If your dataset has 90% Class A and 10% Class B, each fold will have, in this case, about a 90%:10% ratio, giving you a more consistent and fair evaluation.
Image by Author
// 3. Leave-One-Out Cross-Validation (LOOCV)
Leave-One-Out Cross-Validation (LOOCV) is an extreme case of k-fold where the number of folds equals the number of data points. This means that for each run, the model is trained on all but one observation, and that single observation is used as the test set.
The process repeats until every point has been tested once, and the results are averaged. LOOCV can provide nearly unbiased estimates of performance, but it is extremely computationally expensive on larger datasets because the model must be trained as many times as there are data points.
Image by Author
// 4. Time-Series Cross-Validation
When working with temporal data such as financial prices, sensor readings, or user activity logs, time-series cross-validation is required. Randomly shuffling the data would break the natural order of time and risk data leakage, using information from the future to predict the past.
Instead, folds are built chronologically using either an expanding window (gradually increasing the size of the training set) or a rolling window (keeping a fixed-size training set that moves forward with time). This approach respects temporal dependencies and produces realistic performance estimates for forecasting tasks.
Image by Author
# Bias-Variance Tradeoff and Cross-Validation
Cross-validation goes a long way in addressing the bias-variance tradeoff in model evaluation. With a single train-test split, the variance of your performance estimate is high because your result depends heavily on which rows end up in the test set.
However, when you utilize cross-validation you average the performance over multiple test sets, which reduces variance and gives a much more stable estimate of your model’s performance. Certainly, cross-validation will not completely eliminate bias, as no amount of cross-validation will resolve a dataset with bad labels or systematic errors. But in nearly all practical cases, it will be a much better approximation of your model’s performance on unseen data than a single test.
# Example in Python with Scikit-learn
This brief example trains a logistic regression model on the Iris dataset using 5-fold cross-validation (via scikit-learn). The output shows the scores for each fold and the average accuracy, which is much more indicative of performance than any one-off test could provide.
from sklearn.model_selection import cross_val_score, KFold
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import load_iris
X, y = load_iris(return_X_y=True)
model = LogisticRegression(max_iter=1000)
kfold = KFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=kfold)
print("Cross-validation scores:", scores)
print("Average accuracy:", scores.mean())
# Wrapping Up
Cross-validation is one of the most robust techniques for evaluating machine learning models, as it turns one data test into many data tests, giving you a much more reliable picture of the performance of your model. As opposed to the hold-out method, or a single train-test split, it reduces the likelihood of overfitting to one arbitrary dataset partition and makes better use of each piece of data.
As we wrap this up, some of the best practices to keep in mind are:
- Shuffle your data before splitting (except in time-series)
- Use Stratified k-Fold for classification tasks
- Watch out for computation cost with large k or LOOCV
- Prevent data leakage by fitting scalers, encoders, and feature selection only on the training fold
While developing your next model, remember that simply relying on one test set can be fraught with misleading interpretations. Using k-fold cross-validation or similar methods will help you understand better how your model may perform in the real world, and that is what counts after all.
Josep Ferrer is an analytics engineer from Barcelona. He graduated in physics engineering and is currently working in the data science field applied to human mobility. He is a part-time content creator focused on data science and technology. Josep writes on all things AI, covering the application of the ongoing explosion in the field.
