dirty_cat: machine learning on dirty categories¶

dirty_cat facilitates machine-learning on non-curated categories: robust to morphological variants, such as typos.

Automatically ingest a heterogeneous dataframe

SuperVectorizer: a simple transformer to easily turn a non-normalized pandas dataframe into a numpy array for machine learning.

An example

OneHotEncoder but for non-normalized categories

GapEncoder, scalable and interpretable, where each encoding dimension corresponds to a topic that summarizes substrings captured.
SimilarityEncoder, a simple modification of one-hot encoding to capture the strings.
MinHashEncoder, very scalable

For a detailed description of the problem of encoding dirty categorical data, see Similarity encoding for learning with dirty categorical variables 1 and Encoding high-cardinality string categorical variables 2.

Recent changes

Installing: $ pip install –user dirty_cat

Usage¶

API documentation¶

Encoders / Vectorizers¶

`GapEncoder`	This encoder can be understood as a continuous encoding on a set of latent categories estimated from the data.
`MinHashEncoder`	Encode string categorical features as a numeric array, minhash method applied to ngram decomposition of strings based on ngram decomposition of the string.
`SimilarityEncoder`	Encode string categorical features as a numeric array.
`TargetEncoder`	Encode categorical features as a numeric array given a target vector.
`SuperVectorizer`	Easily transforms a heterogeneous data table (such as a dataframe) to a numerical array for machine learning.

Data download¶

`datasets.fetch_employee_salaries`	Fetches the employee_salaries dataset.
`datasets.fetch_medical_charge`	Fetches the medical charge dataset.
`datasets.fetch_midwest_survey`	Fetches the midwest survey dataset.
`datasets.fetch_open_payments`	Fetches the open payments dataset.
`datasets.fetch_road_safety`	Fetches the road safety dataset.
`datasets.fetch_traffic_violations`	Fetches the traffic violations dataset.
`datasets.get_data_dir`	Returns the directory in which dirty_cat looks for data.

About¶

dirty_cat is for now a repository for ideas coming out of a research project: there is still little known about the problems of dirty categories. Tradeoffs will emerge in the long run. We really need people giving feedback on success and failures with the different techniques and pointing us to open datasets on which we can do more empirical work.

1: Patricio Cerda, Gaël Varoquaux, Balázs Kégl. Similarity encoding for learning with dirty categorical variables. 2018. Machine Learning journal, Springer.
2: Patricio Cerda, Gaël Varoquaux. Encoding high-cardinality string categorical variables. 2020. IEEE Transactions on Knowledge & Data Engineering.