Image by Editor
# Introduction
Dask is a set of packages that leverage parallel computing capabilities — extremely useful when handling large datasets or building efficient, data-intensive applications such as advanced analytics and machine learning systems. Among its most prominent advantages is Dask’s seamless integration with existing Python frameworks, including support for processing large datasets alongside scikit-learn modules through parallelized workflows. This article uncovers how to harness Dask for scalable data processing, even under limited hardware constraints.
# Step-by-Step Walkthrough
Even though it is not particularly massive, the California Housing dataset is reasonably large, making it a great choice for a gentle, illustrative coding example that demonstrates how to jointly leverage Dask and scikit-learn for data processing at scale.
Dask provides a dataframe module that mimics many aspects of the Pandas DataFrame objects to handle large datasets that might not completely fit into memory. We will use this Dask DataFrame structure to load our data from a CSV in a GitHub repository, as follows:
import dask.dataframe as dd
url = "https://raw.githubusercontent.com/gakudo-ai/open-datasets/refs/heads/main/housing.csv"
df = dd.read_csv(url)
df.head()
An important note here. If you want to see the “shape” of the dataset — the number of rows and columns — the method is slightly trickier than just using df.shape. Instead, you should do something like:
num_rows = df.shape[0].compute()
num_cols = df.shape[1]
print(f"Number of rows: {num_rows}")
print(f"Number of columns: {num_cols}")
Output:
Number of rows: 20640
Number of columns: 10
Note that we used Dask’s compute() to lazily compute the number of rows, but not the number of columns. The dataset’s metadata allows us to obtain the number of columns (features) immediately, whereas determining the number of rows in a dataset that might (hypothetically) be larger than memory — and thus partitioned — requires a distributed computation: something that compute() transparently handles for us.
Data preprocessing is most often a previous step to building a machine learning model or estimator. Before moving on to that part, and since the main focus of this hands-on article is to show how Dask can be used for processing data, let’s clean and prepare it.
One common step in data preparation is dealing with missing values. With Dask, the process is as seamless as if we were just using Pandas. For example, the code below removes rows for instances that contain missing values in any of their attributes:
df = df.dropna()
num_rows = df.shape[0].compute()
num_cols = df.shape[1]
print(f"Number of rows: {num_rows}")
print(f"Number of columns: {num_cols}")
Now the dataset has been reduced by over 200 instances, having 20433 rows in total.
Next, we can scale some numerical features in the dataset by incorporating scikit-learn’s StandardScaler or any other suitable scaling method:
from sklearn.preprocessing import StandardScaler
numeric_df = df.select_dtypes(include=["number"])
X_pd = numeric_df.drop("median_house_value", axis=1).compute()
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_pd)
Importantly, notice that for a sequence of dataset-intensive operations we perform in Dask, like dropping rows containing missing values followed by dropping the target column "median_house_value", we must add compute() at the end of the sequence of chained operations. This is because dataset transformations in Dask are performed lazily. Once compute() is called, the result of the chained transformation on the dataset is materialized as a Pandas DataFrame (Dask depends on Pandas, hence you won’t need to explicitly import the Pandas library in your code unless you are directly calling a Pandas-exclusive function).
What if we want to train a machine learning model? Then we should extract the target variable "median_house_value" and apply the same principle to convert it to a Pandas object:
y = df["median_house_value"]
y_pd = y.compute()
From now on, the process to split the dataset into training and test sets, train a regression model like RandomForestRegressor, and evaluate its error on the test data fully resembles a traditional approach using Pandas and scikit-learn in an orchestrated manner. Since tree-based models are insensitive to feature scaling, you can use either the unscaled features (X_pd) or the scaled ones (X_scaled). Below we proceed with the scaled features computed above:
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import mean_squared_error
import numpy as np
# Use the scaled feature matrix produced earlier
X_train, X_test, y_train, y_test = train_test_split(X_scaled, y_pd, test_size=0.2, random_state=42)
model = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
print(f"RMSE: {rmse:.2f}")
Output:
# Wrapping Up
Dask and scikit-learn can be used together to leverage scalable, parallelized data processing workflows, for example, to efficiently preprocess large datasets for building machine learning models. This article demonstrated how to load, clean, prepare, and transform data using Dask, subsequently applying standard scikit-learn tools for machine learning modeling — all while optimizing memory usage and speeding up the pipeline when dealing with massive datasets.
Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.
