Find most important words for k-means clustering using sklearn_pandas

Question

I am new to sklearn. I want my code to group data with k-means clustering based on a text column and some additional categorical variables. CountVectorizer transforms the text to a bag-of-words and OneHotEncoder transforms the categorical variables to sets of dummies.

import pandas as pd
from sklearn.preprocessing import OneHotEncoder
from sklearn.feature_extraction.text import CountVectorizer
from sklearn_pandas import DataFrameMapper
from sklearn.cluster import MiniBatchKMeans

def import_vectorizer():   
    vectorizer = CountVectorizer(lowercase   = True,  
                             ngram_range = (1,1),
                             min_df      = .00005,  
                             max_df      = .01)
    return vectorizer

The DataFrameMapper from sklearn_pandas combines the bag-of-words with the dummy variables.

def get_X(df):
    mapper = DataFrameMapper(
        [
            ('text_col', import_vectorizer()),
            (['cat_col1', 'cat_col2', 'cat_col3', 'cat_col4'], OneHotEncoder())
        ]
    )
    return mapper.fit_transform(df)

To predict groups I run

df = pd.read_json(mydata.json)
X = get_X(df)
kmeans = MiniBatchKMeans(n_clusters=50)
kmeans.fit(X)

Now I want to see which features that are most important in predicting the groups. There are posts along the lines of

print("Top terms per cluster:")
order_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]
terms = vectorizer.get_feature_names()
for i in range(50):
    print("Cluster %d:" % i),
    for ind in order_centroids[i, :10]:
        print(' %s' % terms[ind])

However, this does not work in this case since

terms = vectorizer.get_feature_names()

would only contain the feature names from the bag of words and not those produced by OneHotEncoder. Any help would be greatly appreciated.

jon · Accepted Answer · 2020-05-28T08:33:48.660

For the record, I was able to solve the problem after reading this post.

Modified get_X function:

def get_X(df):
    mapper = DataFrameMapper(
        [
        ('text_col', import_vectorizer()),
        (['cat_col1', 'cat_col2', 'cat_col3', 'cat_col4'], OneHotEncoder())
       ]
    )

    X = mapper.fit_transform(df)
    X_cols = (
            mapper.features[0][1].get_feature_names()
            + mapper.features[1][1].get_feature_names().tolist()
            )
    return X, X_cols

And then run the following code in the end.

print("Top terms per cluster:")
order_centroids = kmeans.cluster_centers_.argsort()[:, ::-1]
for i in range(50):
    print("Cluster %d:" % i),
    for ind in order_centroids[i, :10]:
        print(' %s' % X_cols[ind])

Find most important words for k-means clustering using sklearn_pandas

1 Answers1