Image by Editor
# Introduction
Feature engineering is an essential process in data science and machine learning workflows, as well as in any AI system as a whole. It entails the construction of meaningful explanatory variables from raw — and often rather messy — data. The processes behind feature engineering can be extremely simple or overly complex, depending on the volume, structure, and heterogeneity of the dataset(s) as well as the machine learning modeling objectives. While the most popular Python libraries for data manipulation and modeling, like Pandas and scikit-learn, enable basic and moderately scalable feature engineering to some extent, there are specialized libraries that go the extra mile in dealing with massive datasets and automating complex transformations, yet they are largely unknown to many.
This article lists 7 under-the-radar Python libraries that push the boundaries of feature engineering processes at scale.
# 1. Accelerating with NVTabular
First up, we have NVIDIA-Merlin’s NVTabular: a library designed to apply preprocessing and feature engineering to datasets that are — yes, you guessed it! — tabular. Its distinctive characteristic is its GPU-accelerated approach formulated to easily manipulate very large-scale datasets needed to train vast deep learning models. The library has been particularly designed to help scale pipelines for modern recommender system engines based on deep neural networks (DNNs).
# 2. Automating with FeatureTools
FeatureTools, designed by Alteryx, focuses on leveraging automation in feature engineering processes. This library applies deep feature synthesis (DFS), an algorithm that creates new, “deep” features upon analyzing relationships mathematically. The library can be used on both relational and time series data, making it possible in both of them to yield complex feature generation with minimal coding burden.
This code excerpt shows an example of what applying DFS with the featuretools library looks like, on a dataset of customers:
customers_df = pd.DataFrame({'customer_id': [101, 102]})
es = es.add_dataframe(
dataframe_name="customers",
dataframe=customers_df,
index="customer_id"
)
es = es.add_relationship(
parent_dataframe_name="customers",
parent_column_name="customer_id",
child_dataframe_name="transactions",
child_column_name="customer_id"
)
# 3. Parallelizing with Dask
Dask is growing its popularity as a library to make parallel Python computations faster and simpler. The master recipe behind Dask is to scale traditional Pandas and scikit-learn feature transformations through cluster-based computations, thereby facilitating faster and affordable feature engineering pipelines on large datasets that would otherwise exhaust memory.
This article shows a practical Dask walkthrough to perform data preprocessing.
# 4. Optimizing with Polars
Rivalling with Dask in terms of growing popularity, and with Pandas to aspire to a place on the Python data science podium, we have Polars: a Rust-based dataframe library that uses lazy expression API and lazy computations to drive efficient, scalable feature engineering and transformations on very large datasets. Deemed by many as Pandas’ high-performance counterpart, Polars is very easy to learn and familiarize with if you are fairly familiar with Pandas.
Interested to know more about Polars? This article showcases several practical Polars one-liners for common data science tasks, including feature engineering.
# 5. Storing with Feast
Feast is an open-source library conceived as a feature store, helping deliver structured data sources to production-level or production-ready AI applications at scale, especially those based on large language models (LLMs), both for model training and inference tasks. One of its attractive properties consists of ensuring consistency between both stages: training and inference in production. Its use as a feature store has become closely tied to feature engineering processes as well, namely by using it in conjunction with other open-source frameworks, for instance, denormalized.
# 6. Extracting with tsfresh
Shifting the focus toward large time series datasets, we have the tsfresh library, with a package that specializes in scalable feature extraction. Ranging from statistical to spectral properties, this library is capable of computing up to hundreds of meaningful features upon large time series, as well as applying relevance filtering, which entails, as its name suggests, filtering features by relevance in the machine learning modeling process.
This example code excerpt takes a DataFrame containing a time series dataset that has been previously rolled into windows, and applies tsfresh feature extraction on it:
features_rolled = extract_features(
rolled_df,
column_id='id',
column_sort="time",
default_fc_parameters=settings,
n_jobs=0
)
# 7. Streamlining with River
Let’s finish dipping our toes into the river stream (pun intended), with the River library, designed to streamline online machine learning workflows. As part of its suite of functionalities, it has the capability to enable online or streaming feature transformation and feature learning techniques. This can help efficiently deal with issues like unbounded data and concept drift in production. River is built to robustly handle issues rarely occurring in batch machine learning systems, such as the appearance and disappearance of data features over time.
# Wrapping Up
This article has listed 7 notable Python libraries that can help make feature engineering processes more scalable. Some of them are directly focused on providing distinctive feature engineering approaches, while others can be used to further support feature engineering tasks in certain scenarios, in conjunction with other frameworks.
Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.
