<jupyter_start><jupyter_text>Creating long dataframes from *puffy* tablesThis tutorial will show the basic features of using the `puffbird.puffy_to_long` method<jupyter_code>import numpy as np
import pandas as pd
import puffbird as pb<jupyter_output><empty_output><jupyter_text>A weirdly complex table First, we will create a *puffy* dataframe as an example:<jupyter_code>df = pd.DataFrame({
    # a and c have the same data repeated three times
    # b is just a bunch of numpy arrays of the same shapes
    # d is also just a bunch of numpy arrays of different shapes
    # e contains various pandas DataFrames with the same column structures
    #   and the same index format.
    # f contains various pandas DataFrames with different structures
    # g contains mixed data types
    # missing data is also included
    'a': [
        'aa', 'bb', 'cc', 'dd', 
        'aa', 'bb', 'cc', 'dd', 
        'aa', 'bb', 'cc', 'dd'
    ],
    'b': [
        np.random.random((10, 5)), 
        np.nan, 
        np.random.random((10, 5)), 
        np.random.random((10, 5)), 
        np.random.random((10, 5)),
        np.random.random((10, 5)),
        np.random.random((10, 5)),
        np.random.random((10, 5)),
        np.random.random((10, 5)),
        np.random.random((10, 5)),
        np.random.random((10, 5)), 
        np.random.random((10, 5))
    ],
    'c': [
        {'dicta':[1,2,3], 'dictb':3, 'dictc':{'key1':1, 'key2':2}}, 
        {'dicta':[52,3], 'dictb':[3,4], 'dictc':{'key4':1, 'key2':2}}, 
        {'dicta':[12,67], 'dictb':(4,5), 'dictc':{'key3':1, 'key2':77}}, 
        {'dicta':[1,23], 'dictb':3, 'dictc':{'key1':55, 'key2':33}}, 
        {'dicta':123, 'dictb':'words', 'dictc':{'key1':4, 'key2':2}}, 
        {'dicta':[1,2,3], 'dictb':3, 'dictc':{'key1':1, 'key2':2}}, 
        {'dicta':[52,3], 'dictb':[3,4], 'dictc':{'key4':1, 'key2':2}}, 
        {'dicta':[12,67], 'dictb':(4,5), 'dictc':{'key3':1, 'key2':77}}, 
        {'dicta':[1,23], 'dictb':3, 'dictc':{'key1':55, 'key2':33}}, 
        {'dicta':123, 'dictb':'words', 'dictc':{'key1':4, 'key2':2}}, 
        {'dicta':[1,2,3], 'dictb':3, 'dictc':{'key1':1, 'key2':2}}, 
        {'dicta':[52,3], 'dictb':[3,4], 'dictc':{'key4':1, 'key2':2}}, 
    ],
    'd': [
        np.random.random((16, 5)), 
        np.random.random((18, 5)), 
        np.random.random((19, 5)), 
        np.random.random((11, 5)), 
        np.random.random((12, 5)),
        np.random.random((14, 5)),
        np.random.random((17, 5)),
        np.random.random((110, 5)),
        None,
        np.random.random((2, 5)),
        np.random.random((4, 5)), 
        np.random.random((7, 5))
    ],
    'e': [
        pd.DataFrame(
            {'c1':[1,2,3], 'c2':[1,2,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['a', 'b']
            )
        ),
        pd.DataFrame(
            {'c1':[1,2,3,4], 'c2':[1,2,3,4]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c', 'd'], ['a', 'b', 'c', 'd']], 
                names=['a', 'b']
            )
        ), 
        pd.DataFrame(
            {'c1':[3,4,3], 'c2':[3,5,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['a', 'b']
            )
        ),
        np.nan, 
        pd.DataFrame(
            {'c1':[1,2,3], 'c2':[1,2,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['a', 'b']
            )
        ),
        pd.DataFrame(
            {'c1':[1,2,3,4], 'c2':[1,2,3,4]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c', 'd'], ['a', 'b', 'c', 'd']], 
                names=['a', 'b']
            )
        ), 
        pd.DataFrame(
            {'c1':[3,4,3], 'c2':[3,5,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['a', 'b']
            )
        ),
        np.nan, 
        pd.DataFrame(
            {'c1':[1,2,3], 'c2':[1,2,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['a', 'b']
            )
        ),
        pd.DataFrame(
            {'c1':[1,2,3,4], 'c2':[1,2,3,4], 'c3':[1,2,3,4]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c', 'd'], ['a', 'b', 'c', 'd']], 
                names=['a', 'b']
            )
        ), 
        pd.DataFrame(
            {'c1':[3,4,3], 'c2':[3,5,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['a', 'b']
            )
        ),
        np.nan,  
    ],
    'f': [
        pd.DataFrame(
            {'f1':[1,2,3], 'hh2':[1,2,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c'], ['f', 'f', 'f']], 
                names=['f', 'b', 'e']
            )
        ),
        pd.DataFrame(
            {'hh1':[1,2,3,4], 'qq2':[1,2,3,4]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c', 'd'], ['a', 'b', 'c', 'd']], 
                names=['a', 'b']
            )
        ), 
        pd.DataFrame(
            {'q1':[3,4,3], 'qq2':[3,5,3], 'c3':[1,2,3], 'c4':[1,2,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c'], ['t', 't', 't']], 
                names=['y', 'll', 'tt']
            )
        ),
        np.nan, 
        pd.DataFrame(
            {'qq1':[1,2,3], 'rr2':[1,2,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['a', 'b']
            )
        ),
        pd.DataFrame(
            [[1,2,3,4], [1,2,3,4]], 
            columns=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c', 'd'], ['a', 'b', 'c', 'd']], 
                names=['rr', 'b']
            ), 
            index=pd.MultiIndex.from_arrays(
                [[(1,2), (2,3)], ['a', 'b']], 
                names=['a', 'b']
            )
        ), 
        pd.DataFrame(
            {'cpp1':[3,4,3], 'c2':[3,5,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['a', 'rr']
            )
        ),
        np.nan, 
        pd.DataFrame(
            {'sr1':[1,2,3,4], 'c2':[1,2,3,4]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c', 'd'], ['a', 'b', 'c', 'd']], 
                names=['a', 'b']
            )
        ), 
        pd.DataFrame(
            {'cpp1':[3,4,3], 'c2':[3,5,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['a', 'b']
            )
        ),
        pd.DataFrame(
            {'c1':[3,4,3], 'c2':[3,5,3]}, 
            index=pd.MultiIndex.from_arrays(
                [['a', 'b', 'c'], ['a', 'b', 'c']], 
                names=['mm', 'b']
            )
        ),
        np.nan,  
    ],
    'g': [
        'a', 'b', {'ff':'gg'}, {'a', 'b', 'c'}, 
        ('r',), pd.Series({'a':'b'}), 'a', 'b', 
        1, 2, 3, 4
    ]
})

df<jupyter_output><empty_output><jupyter_text>So this dataframe is quite daunting, I like to call it a *puffy* table. *Exploding* the data out that are in *puffy* tablesNow with `puffy_to_long` you can easily unravel this dataframe. Since this dataframe is weirdly constructed `puffy_to_long` may take a while:<jupyter_code>long_df = pb.puffy_to_long(df)
long_df.head()<jupyter_output><empty_output><jupyter_text>Now we have a dataframe with only hashable elements. `puffy_to_long` iteratively exploded all cells and treated each column individually. For example, if a cell contains a numpy array that is two-dimensional then the column will be exploded twice and two new columns will be added that are called *\[COLUMN_NAME\]_level0* and *\[COLUMN_NAME\]_level1*. For our column `b`, we get two new columns called `b_level0` and `b_level1`. These levels will contain the index corresponding to the data point in the long-format of column `b`. Let's try `puffy_to_long` again but just on column `b`.  *Exploding* numpy.array-containing columns<jupyter_code>long_df = pb.puffy_to_long(df, 'b')
long_df.head()<jupyter_output><empty_output><jupyter_text>`index_level0` is the previous index from our dataframe. If this were a `pandas.MultiIndex`, we would have multiple columns instead of just one that corresponds to the old index of the dataframe. Now, we could take this normal dataframe objects and perform various operations that we would normally want to perform, e.g.:<jupyter_code>long_df.groupby('b_level0')['b'].mean()<jupyter_output><empty_output><jupyter_text>Let's say we want to *explode* both column `b` and column `d` that both contain numpy arrays. Imagine that we want to align the axis 1 of the data in `b` and `d`, and call this axes `aligned_axis`, we can do this with `puffy_to_long` keyword arguments:<jupyter_code>long_df = pb.puffy_to_long(df, 'b', 'd', aligned_axis={'b':1, 'd':1})
long_df.head()<jupyter_output><empty_output><jupyter_text>So now, axis 1 of `b` and `d` column data are aligned and those indices are defined in the `aligned_axis` column. Since both columns contain missing cells, some `index_level0` values have missing `b` and `b_level0` columns or missing `d_level0` and `d` columns. You can view these easily using standard pandas functionality:<jupyter_code>long_df.loc[long_df['b_level0'].isnull()].head()
long_df.loc[long_df['d_level0'].isnull()].head()<jupyter_output><empty_output><jupyter_text>*Exploding* pandas.DataFrame-containing columns Let's take a look at how `pandas.DataFrame` objects are handled within `puffy_to_long` by taking a look at column `e`:<jupyter_code>long_df = pb.puffy_to_long(df, 'e')
long_df.head()<jupyter_output><empty_output><jupyter_text>`pandas.DataFrame` objects are handled within one *explosion* iteration, unless the cell within that dataframe are non-hashable. This is why all new columns contain `level0`. The first two new columns `e_level0_a` and `e_level0_b` correspond to the `pandas.MultiIndex` index defined in all dataframes within this column. `e_level0_2` corresponds to all the columns names of the dataframe. `e` only contains the data within each cell of each dataframe. Let's say we don't want to unravel our columns in this way but instead just concatenate them all together. We can use the `expand_cols` argument for this, which expects a list of column names that contain only `pandas.DataFrame` or `pandas.Series` objects:<jupyter_code>long_df = pb.puffy_to_long(df, 'e', expand_cols=['e'])
long_df.head()<jupyter_output><empty_output><jupyter_text>Here we preserved the columns of each dataframe in each cell and simply concatenated the dataframes together with the `index_level0` information preserved. What if we use this method while also *exploding* column `a`, since this has the same column name as the column in the dataframes within column `e`:<jupyter_code>long_df = pb.puffy_to_long(df, 'a', 'e', expand_cols=['e'])
long_df.head()<jupyter_output><empty_output><jupyter_text>Since the column `a` already existed, the column `a` within each dataframe within column `e` was renamed to `a_e`.  Of course, similarly, this is handled with column `b`:<jupyter_code>long_df = pb.puffy_to_long(df, 'b', 'e', expand_cols=['e'])
long_df.head()<jupyter_output><empty_output><jupyter_text>Less structured dataframe-containing columns will result in more complex long-format dataframes:<jupyter_code>long_df = pb.puffy_to_long(df, 'f')
long_df.head()
long_df = pb.puffy_to_long(df, 'f', expand_cols=['f'])
long_df.head()<jupyter_output><empty_output><jupyter_text>*Exploding* dictionaries The column `c` contains dictionaries with various data types that are in them. The `puffy_to_long` algorithm iteratively *explodes* all objects within the dictionaries:<jupyter_code>long_df = pb.puffy_to_long(df, 'c')
long_df.head()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup