Close parquet writer properly in `push_to_hub`
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-04-01T15:08:28Z
https://github.com/huggingface/datasets/pull/4081#issuecomment-1086017114
https://api.github.com/repos/huggingface/datasets/issues/4081
IC_kwDODunzps5Au05a
https://api.github.com/repos/huggingface/datasets/issues/comments/1086017114/reactions
2022-04-01T16:21:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1086017114
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
NONE
@lhoestq / @albertvillanova / @mariosasko 
I am facing the same scenario. Let me explain the situation point. I have a glue ETL job

1--> My files are in parquet format and stored in AWS s3.
2--> I am iterating a  loop for a data set where the same file name can occur with diffrent other data.
3--> I read the parquet and saved it in a pandas data frame.
4--> Done some operation on that data frame
5--> upload the updated data frame into the S3 parquet file.  Below are code snippet what I am using to save the updated 
         data frame into parquet format and load into S3
         `header_name_column_list = dict(data_frame)
          header_list = []
            for col_id, col_type in header_name_column_list.items():
                header_list.append(pyarrow.field(col_id, pyarrow.string()))
            table_schema = pyarrow.schema(header_list)
            table = pyarrow.Table.from_pandas(data_frame, schema=table_schema, preserve_index=False)
            writer = parquet.ParquetWriter(b_buffer, table.schema)
            writer.write_table(table)
            writer.close()
            b_buffer.seek(0)
            .....
            ....
           self.s3_client.upload_fileobj(
                b_buffer,
                self.bucket,
                file_key,
                ExtraArgs=extra_args)`

But when I executed the glue etl job, the first time it works properly and but in the next iteration, when I try to open the same file got that error.


<html>
<body>
<!--StartFragment-->

INFO:Iot-dsip-de-duplication-job:Dataframe uploaded: s3://abc/2022/07/12/file1_ft_20220714122108.3065_12345.parquet INFO:Iot-dsip-de-duplication-job:Sleep for 60 sec
INFO:Iot-dsip-de-duplication-job:start after sleep
.......................
..........................
..........................
ERROR:Iot-dsip-de-duplication-job:Failed to read data from parquet file s3://abc/2022/07/12/file1_ft_20220714122108.3065_12345.parquet, error is : Invalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.INFO:Iot-dsip-de-duplication-job:Empty dataframe found

<!--EndFragment-->
</body>
</html>

Any clue will be really helpful..I got stuck with this problem.
2022-07-14T13:23:17Z
https://github.com/huggingface/datasets/pull/4081#issuecomment-1184444428
https://api.github.com/repos/huggingface/datasets/issues/4081
IC_kwDODunzps5GmTAM
https://api.github.com/repos/huggingface/datasets/issues/comments/1184444428/reactions
2022-07-14T19:22:06Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1184444428
https://avatars.githubusercontent.com/u/65144657?v=4
https://api.github.com/users/krishnendudas1979/events{/privacy}
https://api.github.com/users/krishnendudas1979/followers
https://api.github.com/users/krishnendudas1979/following{/other_user}
https://api.github.com/users/krishnendudas1979/gists{/gist_id}
https://github.com/krishnendudas1979
krishnendudas1979
MDQ6VXNlcjY1MTQ0NjU3
https://api.github.com/users/krishnendudas1979/orgs
https://api.github.com/users/krishnendudas1979/received_events
https://api.github.com/users/krishnendudas1979/repos
https://api.github.com/users/krishnendudas1979/starred{/owner}{/repo}
https://api.github.com/users/krishnendudas1979/subscriptions
User
https://api.github.com/users/krishnendudas1979
public
We don’t call writer.close(), which causes https://github.com/huggingface/datasets/issues/4077. It can happen that we upload the file before the writer is garbage collected and writes the footer.

I fixed this by explicitly closing the parquet writer.

Close https://github.com/huggingface/datasets/issues/4077.