Unpacking Json with nested Lists in Pandas

Question

I have a json file that I am trying to unpack that looks like this:

[{'batter': 'LA Marsh',
  'bowler': 'MJG Nielsen',
  'non_striker': 'M Kapp',
  'runs': {'batter': 0, 'extras': 0, 'total': 0}},
 {'batter': 'LA Marsh',
  'bowler': 'MJG Nielsen',
  'non_striker': 'M Kapp',
  'runs': {'batter': 0, 'extras': 0, 'total': 0},
  'wickets': [{'player_out': 'LA Marsh', 'kind': 'bowled'}]},
 {'batter': 'EA Perry',
  'bowler': 'MJG Nielsen',
  'non_striker': 'M Kapp',
  'runs': {'batter': 0, 'extras': 0, 'total': 0}}]

using the following code:

df = pd.json_normalize(data)

I get the following:

As you can see, the second entry has a nested list in it. In place of the column 'wickets' I would like to have two columns "player_out" and "kind". My preferred output looks like this:

does this answer your question? https://stackoverflow.com/questions/47242845/pandas-io-json-json-normalize-with-very-nested-json — nadine, Sep 06 '22 at 12:36
Does this answer your question? [Extract data from array - Python](https://stackoverflow.com/questions/73425151/extract-data-from-array-python) — ThePyGuy, Sep 06 '22 at 12:40

score 0 · Accepted Answer · answered Sep 06 '22 at 12:42

0

Use:

df = df.drop(columns=['wickets']).join(df['wickets'].explode().apply(pd.Series))

answered Sep 06 '22 at 12:42

Muhammad Hassan

4,079
1
13
27

This looks nice and simple, but it actually outputs a third columns titled '0'. Anyway to avoid this? – le_camerone Sep 06 '22 at 13:56

score 0 · Answer 2 · answered Sep 06 '22 at 12:43

You can try:

import pandas as pd
from collections import MutableMapping

def flatten(d, parent_key='', sep='.'):
    items = []
    for k, v in d.items():
        new_key = parent_key + sep + k if parent_key else k
        if isinstance(v, MutableMapping):
            items.extend(flatten(v, new_key, sep=sep).items())
        elif isinstance(v, list):
            for idx, value in enumerate(v):
                items.extend(flatten(value, new_key, sep).items())
        else:
            items.append((new_key, v))
    return dict(items)

data = [{'batter': 'LA Marsh',
  'bowler': 'MJG Nielsen',
  'non_striker': 'M Kapp',
  'runs': {'batter': 0, 'extras': 0, 'total': 0}},
 {'batter': 'LA Marsh',
  'bowler': 'MJG Nielsen',
  'non_striker': 'M Kapp',
  'runs': {'batter': 0, 'extras': 0, 'total': 0},
  'wickets': [{'player_out': 'LA Marsh', 'kind': 'bowled'}]},
 {'batter': 'EA Perry',
  'bowler': 'MJG Nielsen',
  'non_striker': 'M Kapp',
  'runs': {'batter': 0, 'extras': 0, 'total': 0}}]

output = []
for dict_data in data:
    output.append(flatten(dict_data))

df = pd.DataFrame(output)
print(df)

Output:

    batter       bowler non_striker  runs.batter  runs.extras  runs.total wickets.player_out wickets.kind
0  LA Marsh  MJG Nielsen      M Kapp            0            0           0                NaN          NaN
1  LA Marsh  MJG Nielsen      M Kapp            0            0           0           LA Marsh       bowled

2  EA Perry  MJG Nielsen      M Kapp            0            0           0                NaN          NaN

Lucas M. Uriarte · Answer 3 · 2022-09-06T22:40:09.433

if you want to keep using json normalize you need to fisrt homogenize the data

then apply json normalize

# homogenize data
nan_entries = [{'player_out': pd.NA, 'kind':  pd.NA}]
for entry in data:
    if 'wickets' not in entry.keys():
        entry['wickets'] = nan_entries
# use json normailze

pd.json_normalize(data, 
                  record_path='wickets', 
                  meta=['batter', 'bowler', 'non_striker', ['runs', 'batter'],
                         ['runs', 'extras'], ['runs', 'total'] ],
                  record_prefix='wickets.')

output

wickets.player_out  wickets.kind    batter  bowler  non_striker     runs.batter     runs.extras     runs.total
0        <NA>        <NA>         LA Marsh  MJG Nielsen     M Kapp  0   0   0
1      LA Marsh     bowled        LA Marsh  MJG Nielsen     M Kapp  0   0   0
2       <NA>         <NA>         EA Perry  MJG Nielsen     M Kapp  0   0   0

Unpacking Json with nested Lists in Pandas

3 Answers3