nlp downloads to its module path
MEMBER
Indeed this is a known issue arising from the fact that we try to be compatible with cloupickle.

Does this also happen if you are installing in a virtual environment?
2020-09-01T14:10:28Z
https://github.com/huggingface/datasets/issues/554#issuecomment-684883298
https://api.github.com/repos/huggingface/datasets/issues/554
MDEyOklzc3VlQ29tbWVudDY4NDg4MzI5OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/684883298/reactions
2020-09-01T14:18:57Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684883298
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
> Indeed this is a know issue with the fact that we try to be compatible with cloupickle.
> 
> Does this also happen if you are installing in a virtual environment?

Then it would work, because the package is in a writable path.
2020-09-01T14:11:30Z
https://github.com/huggingface/datasets/issues/554#issuecomment-684884070
https://api.github.com/repos/huggingface/datasets/issues/554
MDEyOklzc3VlQ29tbWVudDY4NDg4NDA3MA==
https://api.github.com/repos/huggingface/datasets/issues/comments/684884070/reactions
2020-09-01T14:11:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684884070
https://avatars.githubusercontent.com/u/49398?v=4
https://api.github.com/users/danieldk/events{/privacy}
https://api.github.com/users/danieldk/followers
https://api.github.com/users/danieldk/following{/other_user}
https://api.github.com/users/danieldk/gists{/gist_id}
https://github.com/danieldk
danieldk
MDQ6VXNlcjQ5Mzk4
https://api.github.com/users/danieldk/orgs
https://api.github.com/users/danieldk/received_events
https://api.github.com/users/danieldk/repos
https://api.github.com/users/danieldk/starred{/owner}{/repo}
https://api.github.com/users/danieldk/subscriptions
User
https://api.github.com/users/danieldk
public
MEMBER
If it's fine for you then this is the recommended way to solve this issue.
2020-09-01T14:19:32Z
https://github.com/huggingface/datasets/issues/554#issuecomment-684889910
https://api.github.com/repos/huggingface/datasets/issues/554
MDEyOklzc3VlQ29tbWVudDY4NDg4OTkxMA==
https://api.github.com/repos/huggingface/datasets/issues/comments/684889910/reactions
2020-09-01T14:19:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684889910
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
> If it's fine for you then this is the recommended way to solve this issue.

I don't want to use a virtual environment, because Nix is fully reproducible, and virtual environments are not. And I am the maintainer of the `transformers` in nixpkgs, so sooner or later I will have to package `nlp`, since it is becoming a dependency of `transformers` ;).
2020-09-01T14:24:19Z
https://github.com/huggingface/datasets/issues/554#issuecomment-684893087
https://api.github.com/repos/huggingface/datasets/issues/554
MDEyOklzc3VlQ29tbWVudDY4NDg5MzA4Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/684893087/reactions
2020-09-01T14:26:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/684893087
https://avatars.githubusercontent.com/u/49398?v=4
https://api.github.com/users/danieldk/events{/privacy}
https://api.github.com/users/danieldk/followers
https://api.github.com/users/danieldk/following{/other_user}
https://api.github.com/users/danieldk/gists{/gist_id}
https://github.com/danieldk
danieldk
MDQ6VXNlcjQ5Mzk4
https://api.github.com/users/danieldk/orgs
https://api.github.com/users/danieldk/received_events
https://api.github.com/users/danieldk/repos
https://api.github.com/users/danieldk/starred{/owner}{/repo}
https://api.github.com/users/danieldk/subscriptions
User
https://api.github.com/users/danieldk
public
MEMBER
Ok interesting. We could have another check to see if it's possible to download and import the datasets script at another location than the module path. I think this would probably involve tweaking the python system path dynamically.

I don't know anything about Nix so if you want to give this a try your self we can guide you or you can give us more information on your general project and how this works.

Regarding `nlp` and `transformers`, we are not sure `nlp` will become a required dependency for `transformers`.  It will probably be used a lot in the examples but I think it probably won't be a required dependency for the main package since we try to keep it as light as possible in terms of deps.

Happy to help you make all these things work better for your use-case 
2020-09-02T08:01:45Z
https://github.com/huggingface/datasets/issues/554#issuecomment-685426812
https://api.github.com/repos/huggingface/datasets/issues/554
MDEyOklzc3VlQ29tbWVudDY4NTQyNjgxMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/685426812/reactions
2020-09-02T08:02:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/685426812
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
MEMBER
@danieldk modules are now installed in a different location (by default in the cache directory of the lib, in `~/.cache/huggingface/modules`). You can also change that using the environment variable `HF_MODULES_PATH`

Feel free to play with this change from the master branch for now, and let us know if it sounds good for you :)
We plan to do a release in the next coming days
2020-09-08T09:30:27Z
https://github.com/huggingface/datasets/issues/554#issuecomment-688744993
https://api.github.com/repos/huggingface/datasets/issues/554
MDEyOklzc3VlQ29tbWVudDY4ODc0NDk5Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/688744993/reactions
2020-09-08T09:30:27Z
https://api.github.com/repos/huggingface/datasets/issues/comments/688744993
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
Awesome! I’ll hopefully have some time in the coming days to try this.
2020-09-10T06:45:21Z
https://github.com/huggingface/datasets/issues/554#issuecomment-690025788
https://api.github.com/repos/huggingface/datasets/issues/554
MDEyOklzc3VlQ29tbWVudDY5MDAyNTc4OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/690025788/reactions
2020-09-10T06:45:21Z
https://api.github.com/repos/huggingface/datasets/issues/comments/690025788
https://avatars.githubusercontent.com/u/49398?v=4
https://api.github.com/users/danieldk/events{/privacy}
https://api.github.com/users/danieldk/followers
https://api.github.com/users/danieldk/following{/other_user}
https://api.github.com/users/danieldk/gists{/gist_id}
https://github.com/danieldk
danieldk
MDQ6VXNlcjQ5Mzk4
https://api.github.com/users/danieldk/orgs
https://api.github.com/users/danieldk/received_events
https://api.github.com/users/danieldk/repos
https://api.github.com/users/danieldk/starred{/owner}{/repo}
https://api.github.com/users/danieldk/subscriptions
User
https://api.github.com/users/danieldk
public
MEMBER
> Feel free to play with this change from the master branch for now, and let us know if it sounds good for you :)
> We plan to do a release in the next coming days

Thanks for making this change! I just packaged the latest commit on master and it works like a charm now! :partying_face: 
2020-09-10T17:45:43Z
https://github.com/huggingface/datasets/issues/554#issuecomment-690567488
https://api.github.com/repos/huggingface/datasets/issues/554
MDEyOklzc3VlQ29tbWVudDY5MDU2NzQ4OA==
https://api.github.com/repos/huggingface/datasets/issues/comments/690567488/reactions
2020-09-10T17:45:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/690567488
https://avatars.githubusercontent.com/u/49398?v=4
https://api.github.com/users/danieldk/events{/privacy}
https://api.github.com/users/danieldk/followers
https://api.github.com/users/danieldk/following{/other_user}
https://api.github.com/users/danieldk/gists{/gist_id}
https://github.com/danieldk
danieldk
MDQ6VXNlcjQ5Mzk4
https://api.github.com/users/danieldk/orgs
https://api.github.com/users/danieldk/received_events
https://api.github.com/users/danieldk/repos
https://api.github.com/users/danieldk/starred{/owner}{/repo}
https://api.github.com/users/danieldk/subscriptions
User
https://api.github.com/users/danieldk
public
I am trying to package `nlp` for Nix, because it is now an optional dependency for `transformers`. The problem that I encounter is that the `nlp` library downloads to the module path, which is typically not writable in most package management systems:

```>>> import nlp
>>> squad_dataset = nlp.load_dataset('squad')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 530, in load_dataset
    module_path, hash = prepare_module(path, download_config=download_config, dataset=True)
  File "/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/load.py", line 329, in prepare_module
    os.makedirs(main_folder_path, exist_ok=True)
  File "/nix/store/685kq8pyhrvajah1hdsfn4q7gm3j4yd4-python3-3.8.5/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/nix/store/2yhik0hhqayksmkkfb0ylqp8cf5wa5wp-python3-3.8.5-env/lib/python3.8/site-packages/nlp/datasets/squad'
```

Do you have any suggested workaround for this issue?

Perhaps overriding the default value for `force_local_path` of `prepare_module`?