dirty_cat: machine learning on dirty categories¶

dirty_cat helps with machine-learning on non-curated categories. It provides encoders that are robust to morphological variants, such as typos, in the category strings.

The SimilarityEncoder is a drop-in replacement for scikit-learn’s OneHotEncoder. If speed and scalability are an issue, the MinHashEncoder provides a fast encoding method.

If interpretability is important,the GapEncoder is a good alternative, as it can be interpreted as one-hot encoding, where each encoding dimension corresponds to a topic that summarizes the substrings captured.

For a detailed description of the problem of encoding dirty categorical data, see Similarity encoding for learning with dirty categorical variables 1 and Encoding high-cardinality string categorical variables 2.

Installing: $ pip install –user dirty_cat

Recent changes

Requires Python 3

Using dirty_cat¶

Investigating dirty categories¶

Predicting the salary of employees¶

Semantic variation in the “Midwest”¶

Feature interpretation with the GapEncoder¶

Fitting scalable, non-linear models on data with dirty categories¶

Scalability considerations for similarity encoding¶

API documentation¶

Encoders¶

`GapEncoder`	This encoder can be understood as a continuous encoding on a set of latent categories estimated from the data.
`MinHashEncoder`	Encode string categorical features as a numeric array, minhash method applied to ngram decomposition of strings based on ngram decomposition of the string.
`SimilarityEncoder`	Encode string categorical features as a numeric array.
`TargetEncoder`	Encode categorical features as a numeric array given a target vector.

Data download¶

`datasets.fetch_employee_salaries`	fetches the employee_salaries dataset
`datasets.fetch_medical_charge`	fetches the medical charge dataset
`datasets.fetch_midwest_survey`	fetches the midwest survey dataset
`datasets.fetch_open_payments`	fetches the open payements dataset
`datasets.fetch_road_safety`	fetches the road safety dataset
`datasets.fetch_traffic_violations`	fetches the traffic violations dataset
`datasets.get_data_dir`	Returns the directories in which dirty_cat looks for data.

About¶

dirty_cat is for now a repository for developing ideas with high-quality implementations, a form of a research project: there is still little known about the problems of dirty categories. We hope that tradeoffs will emerge in the long run, and that these tradeoffs will enable us to do better software. We really need people giving feedback on success and failures with the different techniques and pointing us to open datasets on which we can do more empirical work. We also welcome contributions in the scope of dirty categories.

dirty_cat

Table of Contents

Related Topics