Support loading XML datasets
NONE
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5001). All of your documentation changes will be reflected on that endpoint.
2022-09-20T18:50:53Z
https://github.com/huggingface/datasets/pull/5001#issuecomment-1252774632
https://api.github.com/repos/huggingface/datasets/issues/5001
IC_kwDODunzps5Kq9Lo
https://api.github.com/repos/huggingface/datasets/issues/comments/1252774632/reactions
2022-09-20T18:50:53Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1252774632
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
MEMBER
> CC: @davanstrien

I should have some time to look at this on Friday :) 
2022-09-21T09:20:39Z
https://github.com/huggingface/datasets/pull/5001#issuecomment-1253436754
https://api.github.com/repos/huggingface/datasets/issues/5001
IC_kwDODunzps5Kte1S
https://api.github.com/repos/huggingface/datasets/issues/comments/1253436754/reactions
2022-09-21T09:20:39Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1253436754
https://avatars.githubusercontent.com/u/8995957?v=4
https://api.github.com/users/davanstrien/events{/privacy}
https://api.github.com/users/davanstrien/followers
https://api.github.com/users/davanstrien/following{/other_user}
https://api.github.com/users/davanstrien/gists{/gist_id}
https://github.com/davanstrien
davanstrien
MDQ6VXNlcjg5OTU5NTc=
https://api.github.com/users/davanstrien/orgs
https://api.github.com/users/davanstrien/received_events
https://api.github.com/users/davanstrien/repos
https://api.github.com/users/davanstrien/starred{/owner}{/repo}
https://api.github.com/users/davanstrien/subscriptions
User
https://api.github.com/users/davanstrien
public
MEMBER
@albertvillanova I've tried this with a few different XML datasets. One issue I've run into is getting a `KeyError` when the attributes of a field differ from the first parsed row. Unfortunately, this can come up in the ALTO XML format, for example, if you want to parse the 'string' field, which contains the text in the ALTO XML files. 

When parsing a file, this instance has no 'STYLE' attribute: 

```xml
<TextLine HEIGHT="39" WIDTH="295" VPOS="926" HPOS="247"><String WC="0.4600000083" CONTENT="jufqu’en" HEIGHT="39" WIDTH="117" VPOS="926" HPOS="247"/><SP WIDTH="14" VPOS="928" HPOS="365"/><String WC="0.6075000167" CONTENT="l’an" HEIGHT="26" WIDTH="50" VPOS="928" HPOS="380"/><SP WIDTH="24" VPOS="936" HPOS="431"/><String WC="0.4300000072" CONTENT="1" HEIGHT="16" WIDTH="9" VPOS="936" HPOS="456"/><String STYLE="italics" WC="0.5774999857" CONTENT="361." HEIGHT="25" WIDTH="68" VPOS="933" HPOS="474"/></TextLine>
```

Whereas this one which appears later in the file, does have this field: 

```xml
<TextLine HEIGHT="39" WIDTH="712" VPOS="966" HPOS="297"><String STYLE="italics" WC="0.6999999881" CONTENT="I" HEIGHT="17" WIDTH="9" VPOS="977" HPOS="297"/><String WC="0.5" CONTENT="I." HEIGHT="18" WIDTH="25" VPOS="976" HPOS="318"/><SP WIDTH="24" VPOS="971" HPOS="344"/><String STYLE="italics" WC="0.3359999955" CONTENT="Crade" HEIGHT="26" WIDTH="91" VPOS="967" HPOS="369"/><SP WIDTH="31" VPOS="971" HPOS="461"/><String STYLE="italics" WC="0.6060000062" CONTENT="Pétri" HEIGHT="26" WIDTH="71" VPOS="968" HPOS="493"/><SP WIDTH="23" VPOS="968" HPOS="565"/><String STYLE="italics" WC="0.612857163" CONTENT="Candidi" HEIGHT="27" WIDTH="111" VPOS="967" HPOS="589"/><SP WIDTH="19" VPOS="967" HPOS="701"/><String STYLE="italics" WC="0.4088888764" CONTENT="Decembrii" HEIGHT="28" WIDTH="144" VPOS="966" HPOS="721"/><SP WIDTH="10" VPOS="968" HPOS="866"/><String STYLE="italics" WC="0.4600000083" CONTENT="in" HEIGHT="25" WIDTH="27" VPOS="968" HPOS="877"/><SP WIDTH="9" VPOS="967" HPOS="905"/><String STYLE="italics" WC="0.5099999905" CONTENT="funere" HEIGHT="38" WIDTH="94" VPOS="967" HPOS="915"/></TextLine>
```

Since the first-seen fields define what is passed to `arrow_writer`, this causes a KeyError when the version with the extra attributes is encountered because it doesn't expect this column. 

Since it's important to support streaming, I'm not sure there is a nice way to detect attributes for the whole file easily in an automatic way. The two potential ways I can see of doing it.

- Do an initial pass on a batch of data to have a higher chance of encountering variations in attributes before doing the arrow write. 
- Do a full pass on one file (and assume that this won't change across files) 

I think the other way of doing this would be to allow users to define expected/wanted attributes as another loading argument. This could then be used to extract the described attributes (and make them None if not found). This requires a bit more work from the user but could be helpful. For example, in the XML above, likely, most users will only want the `WC` and `CONTENT` attributes. So they could specify this upfront and avoid loading extra data they don't need or want. I suspect this option would make more sense than making this operation automatic for the case where attributes might change. WDYT? 







2022-11-01T12:44:41Z
https://github.com/huggingface/datasets/pull/5001#issuecomment-1298456328
https://api.github.com/repos/huggingface/datasets/issues/5001
IC_kwDODunzps5NZN8I
https://api.github.com/repos/huggingface/datasets/issues/comments/1298456328/reactions
2022-11-01T12:44:41Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1298456328
https://avatars.githubusercontent.com/u/8995957?v=4
https://api.github.com/users/davanstrien/events{/privacy}
https://api.github.com/users/davanstrien/followers
https://api.github.com/users/davanstrien/following{/other_user}
https://api.github.com/users/davanstrien/gists{/gist_id}
https://github.com/davanstrien
davanstrien
MDQ6VXNlcjg5OTU5NTc=
https://api.github.com/users/davanstrien/orgs
https://api.github.com/users/davanstrien/received_events
https://api.github.com/users/davanstrien/repos
https://api.github.com/users/davanstrien/starred{/owner}{/repo}
https://api.github.com/users/davanstrien/subscriptions
User
https://api.github.com/users/davanstrien
public
CC: @davanstrien 