Add CMU MoCap Dataset
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-04-24T17:41:26Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1107885980
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5CCP-c
https://api.github.com/repos/huggingface/datasets/issues/comments/1107885980/reactions
2022-10-03T09:38:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1107885980
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
CONTRIBUTOR
- Updated the readme.
- Added dummy_data.zip and ran the all the tests.

The dataset works for "asf/amc" and "avi" formats which have a single download link for the complete dataset. But "c3d" and "mpg" have multiple download links, can we combine and host these links on the Hub since the dataset is free to use ?
2022-05-01T15:15:16Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1114263414
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5Cak92
https://api.github.com/repos/huggingface/datasets/issues/comments/1114263414/reactions
2022-05-01T15:15:30Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1114263414
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
"c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?
Can we combine and host these links on the Hub since the dataset is free to use ?
2022-05-10T05:40:17Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1121953735
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5C36fH
https://api.github.com/repos/huggingface/datasets/issues/comments/1121953735/reactions
2022-05-10T17:12:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1121953735
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
COLLABORATOR
> "c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?

We store downloaded data under `~/.cache/huggingface/datasets/downloads` (by default), so these downloads are "hidden" and won't clutter one's filesystem in an "obvious way".
2022-05-16T13:15:29Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1127659774
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5DNrj-
https://api.github.com/repos/huggingface/datasets/issues/comments/1127659774/reactions
2022-05-16T13:15:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1127659774
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
> We store downloaded data under ~/.cache/huggingface/datasets/downloads (by default), so these downloads are "hidden" and won't clutter one's filesystem in an "obvious way".

Yes, the filesystem won't be clustered, but the problem is processing the dataset becomes cumbersome. For eg, for the c3d format has 5 part-downloads, so the folders will be as follows : 
```
['~/.cache/huggingface/datasets/downloads/extracted/0e6bf028f490bf18c23ce572d1437c4ef32a74f630e33c26a806250d35cfcdd1', '~/.cache/huggingface/datasets/downloads/extracted/1b44fc5c7a6e031c904545422d449fd964f8ee795b9d1dcb0b6a76d03b50ebe6', '~/.cache/huggingface/datasets/downloads/extracted/137595188e96187c24ce1aa5c78200c7f78816fbd9d6c62354c01b3e6ec550c7', '~/.cache/huggingface/datasets/downloads/extracted/6c0c893e435f36fd79aa0f199f58fe16f01985f039644a7cb094a8c43a15ffd4', '~/.cache/huggingface/datasets/downloads/extracted/45e4703354cbc975e6add66f1b17b716c882b56f44575b033c5926aa5fcfb17f']
```
Each of these folders have a given set of subjects, so we'll be need to write extra code to fetch data from each of these folders, and the mpg format has 12 part-downloads which will lead to 12 folders having certain set of subjects, so it is cumbersome to process them.
2022-05-16T16:58:42Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1127912243
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5DOpMz
https://api.github.com/repos/huggingface/datasets/issues/comments/1127912243/reactions
2022-05-16T16:58:42Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1127912243
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
I have added all the changes that were suggested. We just need to handle the multi-part download for c3d and mpg formats. Easiest way would be to have just one zip for these formats.
2022-05-18T15:01:24Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1130129075
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5DXGaz
https://api.github.com/repos/huggingface/datasets/issues/comments/1130129075/reactions
2022-05-18T15:01:24Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1130129075
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
COLLABORATOR
But we can handle this with a simple mapping that stores the id ranges (for each config), no? And an actual file path is not important during processing.
2022-05-20T17:04:46Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1133128559
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5Diitv
https://api.github.com/repos/huggingface/datasets/issues/comments/1133128559/reactions
2022-05-20T17:05:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1133128559
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
CONTRIBUTOR
I have added code to handle c3d, mpg formats as well.  The data for the mpg format seems incomplete as it contains only 53 rows. I have added a note regarding this in the Data Splits section.
2022-05-22T10:58:50Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1133869958
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5DlXuG
https://api.github.com/repos/huggingface/datasets/issues/comments/1133869958/reactions
2022-05-22T10:58:50Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1133869958
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
The real data test works fine and dummy_data test work fine. There were few missing files which was causing issues, I have fixed it now.

2022-05-23T17:37:50Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1134956013
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5Dpg3t
https://api.github.com/repos/huggingface/datasets/issues/comments/1134956013/reactions
2022-05-24T15:27:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1134956013
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
CONTRIBUTOR
- Reduced the dummy_data size.
- Added sample dataset preprocessing code, it is not complete though.
- Added all changes suggested.

Let me know if anything else is required. Thank you. :)
2022-05-29T18:43:31Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1140503605
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5D-rQ1
https://api.github.com/repos/huggingface/datasets/issues/comments/1140503605/reactions
2022-05-29T18:43:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1140503605
https://avatars.githubusercontent.com/u/17746528?v=4
https://api.github.com/users/dnaveenr/events{/privacy}
https://api.github.com/users/dnaveenr/followers
https://api.github.com/users/dnaveenr/following{/other_user}
https://api.github.com/users/dnaveenr/gists{/gist_id}
https://github.com/dnaveenr
dnaveenr
MDQ6VXNlcjE3NzQ2NTI4
https://api.github.com/users/dnaveenr/orgs
https://api.github.com/users/dnaveenr/received_events
https://api.github.com/users/dnaveenr/repos
https://api.github.com/users/dnaveenr/starred{/owner}{/repo}
https://api.github.com/users/dnaveenr/subscriptions
User
https://api.github.com/users/dnaveenr
public
MEMBER
Thanks for your contribution, @dnaveenr.

We are removing the dataset scripts from this GitHub repo and moving them to the Hugging Face Hub: https://huggingface.co/datasets

We would suggest you create this dataset there. Please, feel free to tell us if you need some help.
2022-09-23T15:10:02Z
https://github.com/huggingface/datasets/pull/4208#issuecomment-1256340557
https://api.github.com/repos/huggingface/datasets/issues/4208
IC_kwDODunzps5K4jxN
https://api.github.com/repos/huggingface/datasets/issues/comments/1256340557/reactions
2022-09-23T15:10:02Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1256340557
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
Resolves #3457 

Dataset Request : Add CMU Graphics Lab Motion Capture dataset [#3457](https://github.com/huggingface/datasets/issues/3457)
This PR adds the CMU MoCap Dataset.

The authors didn't respond even after multiple follow ups, so I ended up crawling the website to get categories, subcategories and description information. Some of the subjects do not have category/subcategory/description as well. I am using a subject to categories, subcategories and description map (metadata file).

Currently the loading of the dataset works for "asf/amc" and "avi" formats since they have a single download link. But "c3d" and "mpg" have multiple download links (part archives) and dl_manager.download_and_extract() extracts the files to multiple paths, is there a way to extract these multiple archives into one folder ? Any other way to go about this ?
Any suggestions/inputs on this would be helpful. Thank you.


