<jupyter_start><jupyter_text>converting jsonl files of the metadata and pdf_pases to data frames<jupyter_code>metadata_path = '20200705v1/sample/metadata/sample.jsonl'
df=pd.read_json(metadata_path, lines=True)
df.keys()
pdf_parses_path = '20200705v1/sample/pdf_parses/sample.jsonl'
df_pdf = pd.read_json(pdf_parses_path, lines=True)
df_pdf.keys()<jupyter_output><empty_output><jupyter_text>removing all columns except 'paper_id', 'abstract' and 'mag_field_of_study' and droping NaN value containging rows<jupyter_code>df_abstract = df.filter(['paper_id', 'abstract', 'mag_field_of_study'])
df_abstract.keys()
df_abstract.isna().sum()
df_abstract.dropna(inplace=True)
df_abstract.isna().sum()
df_abstract.columns
df_abstract.shape<jupyter_output><empty_output><jupyter_text>selecting rows with 'mag_field_of_study' columns values being only CS, Phy, Med, Bio, Chem and Mat<jupyter_code>df_abstract.mag_field_of_study.head(3)
df_abstract.mag_field_of_study=df_abstract.mag_field_of_study.str[0]
df_abstract.mag_field_of_study.head(3)
df_abstract.mag_field_of_study.unique()
df_abst_pmbccm = df_abstract.loc[df_abstract.mag_field_of_study.isin(['Computer Science', 'Physics', 'Medicine', 'Biology', 'Chemistry', 'Mathematics'])]
df_abst_pmbccm.mag_field_of_study.unique()<jupyter_output><empty_output><jupyter_text>merge metadata and pdf_parses<jupyter_code>df_merged = df_abst_pmbccm.merge(df_pdf, on='paper_id', how='left')
df_merged.columns
### droping columns of no relavance here
df_merged.drop(columns=['_pdf_hash', 'bib_entries', 'ref_entries', 'abstract_y'],  inplace=True)
df_merged.columns
df_merged.isna().sum()
df_merged.head(10)<jupyter_output><empty_output><jupyter_text>creating a function to filter body_text column to have only text key values<jupyter_code>df_merged.body_text.isnull()[0]
df_merged.body_text[1]!=df_merged.body_text[1]
df_merged.body_text[95]!=df_merged.body_text[95]
def get_text(y):
    if y!=y:
        return np.nan
    else:
        text = ''
        for dict_ in y:
            text = text+' '+dict_['text']
        return text
df_merged.body_text = df_merged['body_text'].apply(get_text)<jupyter_output><empty_output><jupyter_text>merged file is converted to a feather file<jupyter_code>df_merged.to_feather('20200705v1/sample/df_merged.feather')
ls 20200705v1/sample/<jupyter_output>df_merged.feather  [34mmetadata[m[m/          [34mpdf_parses[m[m/
StarcoderdataJupyterStructuredCleanDedup