5

After using Pandas get_dummies on 3 categorical columns to get a one hot-encoded Dataframe, I've trained (with some success) a Perceptron model.

Now I would like to predict the result from a new observation, that it is not hot-encoded.

Is there any way to record the get_dummies column mapping to re-use it?

Hugo
  • 1,558
  • 12
  • 35
  • 68

1 Answers1

5

There is no automatic procedure to do it at the moment, to my knowledge. In the future release of sklearn CategoricalEncoder will be very handy for this job. You can already get your hands on it, if you clone sklearn github master branch and build in yourself. At the moment 2 options come to my mind:

  • use LabelEncoder+OneHotEncoder combination, see this answer, for example;
  • simply retrieve (and store, if needed) the list of columns after the training OHE output. Then run pd.get_dummies on the test set/example. Loop through the output test OHE columns, drop those that do not appear in the training OHE and add those that are missing in test OHE filled with zeros.
Mischa Lisovyi
  • 3,207
  • 18
  • 29
  • 1
    I was thinking of doing the second one. That will work, but I thought this could not possibly be the expected way since any process that creates features for training (such as One Hot Encoding) must also "work" for prediction for a model to be worth anything, – demongolem May 08 '20 at 17:12
  • You are right, but sadly i'm not aware of the "proper" implementation. If you find it in an open-source package, do not hesitate to share – Mischa Lisovyi May 10 '20 at 06:49
  • @demongolem Did you manage to find anything better? I think the first is actually the "correct" solution, as described [here](https://stackoverflow.com/a/62241515/12162243) and [here](https://stackoverflow.com/a/63432953/12162243). On the first link there's the 2nd solution also described using [pandas reindex](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reindex.html). And here's [one more example for using reindex](https://predictivehacks.com/?all-tips=how-to-deal-with-get_dummies-in-train-and-test-dataset) – PajLe Apr 25 '22 at 11:43