xPath to implement all operations for Path
COLLABORATOR
 I think https://github.com/fsspec/universal_pathlib is the project you are looking for.

`xPath` has the methods often used in dataset scripts, and `mkdir` is not one of them (`dl_manager`'s role is to "interact" with the file system, so using `mkdir` is discouraged).
2023-03-15T19:45:50Z
https://github.com/huggingface/datasets/issues/5638#issuecomment-1470736696
https://api.github.com/repos/huggingface/datasets/issues/5638
IC_kwDODunzps5Xqak4
https://api.github.com/repos/huggingface/datasets/issues/comments/1470736696/reactions
2023-03-15T19:45:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1470736696
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Right is there a difference between UPath and xPath? Typically is xPath less well implemented compared to Upath, ie missing some implementations of some methods? Or are there methods in xPath that are not implemented with UPath?
2023-03-16T08:54:27Z
https://github.com/huggingface/datasets/issues/5638#issuecomment-1471545785
https://api.github.com/repos/huggingface/datasets/issues/5638
IC_kwDODunzps5XtgG5
https://api.github.com/repos/huggingface/datasets/issues/comments/1471545785/reactions
2023-03-16T08:54:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1471545785
https://avatars.githubusercontent.com/u/24695242?v=4
https://api.github.com/users/thomasw21/events{/privacy}
https://api.github.com/users/thomasw21/followers
https://api.github.com/users/thomasw21/following{/other_user}
https://api.github.com/users/thomasw21/gists{/gist_id}
https://github.com/thomasw21
thomasw21
MDQ6VXNlcjI0Njk1MjQy
https://api.github.com/users/thomasw21/orgs
https://api.github.com/users/thomasw21/received_events
https://api.github.com/users/thomasw21/repos
https://api.github.com/users/thomasw21/starred{/owner}{/repo}
https://api.github.com/users/thomasw21/subscriptions
User
https://api.github.com/users/thomasw21
public
COLLABORATOR
`xPath` is an internal component (it doesn't have a leading underscore in the name, but it should) not meant to be used outside of `datasets`, and it's only tested on HTTP URLs, not S3.


2023-03-16T17:49:25Z
https://github.com/huggingface/datasets/issues/5638#issuecomment-1472441748
https://api.github.com/repos/huggingface/datasets/issues/5638
IC_kwDODunzps5Xw62U
https://api.github.com/repos/huggingface/datasets/issues/comments/1472441748/reactions
2023-03-16T17:49:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1472441748
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
Okay I understand that xPath won't support my usecase. What I was perhaps getting to is why not use UPath in `datasets` instead of `xPath` if UPath seems to have strictly more robust implementations.
2023-03-16T17:59:58Z
https://github.com/huggingface/datasets/issues/5638#issuecomment-1472458349
https://api.github.com/repos/huggingface/datasets/issues/5638
IC_kwDODunzps5Xw-5t
https://api.github.com/repos/huggingface/datasets/issues/comments/1472458349/reactions
2023-03-16T17:59:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1472458349
https://avatars.githubusercontent.com/u/24695242?v=4
https://api.github.com/users/thomasw21/events{/privacy}
https://api.github.com/users/thomasw21/followers
https://api.github.com/users/thomasw21/following{/other_user}
https://api.github.com/users/thomasw21/gists{/gist_id}
https://github.com/thomasw21
thomasw21
MDQ6VXNlcjI0Njk1MjQy
https://api.github.com/users/thomasw21/orgs
https://api.github.com/users/thomasw21/received_events
https://api.github.com/users/thomasw21/repos
https://api.github.com/users/thomasw21/starred{/owner}{/repo}
https://api.github.com/users/thomasw21/subscriptions
User
https://api.github.com/users/thomasw21
public
COLLABORATOR
It seems like `universal_pathlib` does not support `fsspec` URL chaining (`::` is the chaining symbol) and "compression" filesystems (e.g., `zip`), but this is what we need to access and stream files from within an archive (e.g., we want to stream URLs such as this one: `zip://data.parquet::https://www.dummyurl.com/archive.zip`)
2023-03-16T18:46:45Z
https://github.com/huggingface/datasets/issues/5638#issuecomment-1472565844
https://api.github.com/repos/huggingface/datasets/issues/5638
IC_kwDODunzps5XxZJU
https://api.github.com/repos/huggingface/datasets/issues/comments/1472565844/reactions
2023-03-16T18:46:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1472565844
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
### Feature request

Current xPath implementation is a great extension of Path in order to work with remote objects. However some methods such as `mkdir` are not implemented correctly. It should instead rely on `fsspec` methods, instead of defaulting do `Path` methods which only work locally.

### Motivation

I'm using xPath to interact with remote objects.

### Your contribution

I could try to make a PR. I'm a bit unfamiliar with chaining right now.