302

Without using groupby how would I filter out data without NaN?

Let say I have a matrix where customers will fill in 'N/A','n/a' or any of its variations and others leave it blank:

import pandas as pd
import numpy as np


df = pd.DataFrame({'movie': ['thg', 'thg', 'mol', 'mol', 'lob', 'lob'],
                  'rating': [3., 4., 5., np.nan, np.nan, np.nan],
                  'name': ['John', np.nan, 'N/A', 'Graham', np.nan, np.nan]})

nbs = df['name'].str.extract('^(N/A|NA|na|n/a)')
nms=df[(df['name'] != nbs) ]

output:

>>> nms
  movie    name  rating
0   thg    John       3
1   thg     NaN       4
3   mol  Graham     NaN
4   lob     NaN     NaN
5   lob     NaN     NaN

How would I filter out NaN values so I can get results to work with like this:

  movie    name  rating
0   thg    John       3
3   mol  Graham     NaN

I am guessing I need something like ~np.isnan but the tilda does not work with strings.

ah bon
  • 9,293
  • 12
  • 65
  • 148
ccsv
  • 8,188
  • 12
  • 53
  • 97

7 Answers7

385

Simplest of all solutions:

filtered_df = df[df['name'].notnull()]

Thus, it filters out only rows that doesn't have NaN values in 'name' column.

For multiple columns:

filtered_df = df[df[['name', 'country', 'region']].notnull().all(1)]
Gil Baggio
  • 13,019
  • 3
  • 48
  • 37
337

Just drop them:

nms.dropna(thresh=2)

this will drop all rows where there are at least two non-NaN.

Then you could then drop where name is NaN:

In [87]:

nms
Out[87]:
  movie    name  rating
0   thg    John       3
1   thg     NaN       4
3   mol  Graham     NaN
4   lob     NaN     NaN
5   lob     NaN     NaN

[5 rows x 3 columns]
In [89]:

nms = nms.dropna(thresh=2)
In [90]:

nms[nms.name.notnull()]
Out[90]:
  movie    name  rating
0   thg    John       3
3   mol  Graham     NaN

[2 rows x 3 columns]

EDIT

Actually looking at what you originally want you can do just this without the dropna call:

nms[nms.name.notnull()]

UPDATE

Looking at this question 3 years later, there is a mistake, firstly thresh arg looks for at least n non-NaN values so in fact the output should be:

In [4]:
nms.dropna(thresh=2)

Out[4]:
  movie    name  rating
0   thg    John     3.0
1   thg     NaN     4.0
3   mol  Graham     NaN

It's possible that I was either mistaken 3 years ago or that the version of pandas I was running had a bug, both scenarios are entirely possible.

Cerberton
  • 376
  • 2
  • 7
  • 16
EdChum
  • 376,765
  • 198
  • 813
  • 562
15
df.dropna(subset=['columnName1', 'columnName2'])
JacoSolari
  • 1,226
  • 14
  • 28
10
df = pd.DataFrame({'movie': ['thg', 'thg', 'mol', 'mol', 'lob', 'lob'],'rating': [3., 4., 5., np.nan, np.nan, np.nan],'name': ['John','James', np.nan, np.nan, np.nan,np.nan]})

for col in df.columns:
    df = df[~pd.isnull(df[col])]
Bashar Mohammad
  • 111
  • 1
  • 4
2

You can also use query:

out = df.query("name.notna() & name !='N/A'", engine='python')

Output:

  movie  rating    name
0   thg     3.0    John
3   mol     NaN  Graham
1

Inside query() pass column_name == column_name to keep the rows where column_name is not NA.

For your case:

nms.query('name == name')
rachwa
  • 1,805
  • 1
  • 14
  • 17
1

You can filter negative to columns with na values: dt = dt[~dt[columns_to_filter].isna().all(1)]

Howdyouride
  • 99
  • 1
  • 7
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – shamnad sherief Jan 25 '23 at 03:05