Update `_format_columns` in `remove_columns`
MEMBER
🤗 This PR closes https://github.com/huggingface/datasets/issues/4398
2022-05-26T11:42:09Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1138443035
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5D20Mb
https://api.github.com/repos/huggingface/datasets/issues/comments/1138443035/reactions
2022-05-26T11:42:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1138443035
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
NONE
_The documentation is not available anymore as the PR was closed or merged._
2022-05-26T11:44:41Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1138444863
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5D20o_
https://api.github.com/repos/huggingface/datasets/issues/comments/1138444863/reactions
2022-06-14T16:11:43Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1138444863
https://avatars.githubusercontent.com/u/99929124?v=4
https://api.github.com/users/HuggingFaceDocBuilderDev/events{/privacy}
https://api.github.com/users/HuggingFaceDocBuilderDev/followers
https://api.github.com/users/HuggingFaceDocBuilderDev/following{/other_user}
https://api.github.com/users/HuggingFaceDocBuilderDev/gists{/gist_id}
https://github.com/HuggingFaceDocBuilderDev
HuggingFaceDocBuilderDev
U_kgDOBfTMJA
https://api.github.com/users/HuggingFaceDocBuilderDev/orgs
https://api.github.com/users/HuggingFaceDocBuilderDev/received_events
https://api.github.com/users/HuggingFaceDocBuilderDev/repos
https://api.github.com/users/HuggingFaceDocBuilderDev/starred{/owner}{/repo}
https://api.github.com/users/HuggingFaceDocBuilderDev/subscriptions
User
https://api.github.com/users/HuggingFaceDocBuilderDev
public
COLLABORATOR
Hi! Thanks for reporting and providing a fix. I made a small change to make the fix easier to understand.
2022-05-26T14:57:47Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1138670051
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5D3rnj
https://api.github.com/repos/huggingface/datasets/issues/comments/1138670051/reactions
2022-05-26T15:01:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1138670051
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Hi, @mariosasko thanks! It makes sense, sorry I'm not that familiar with `datasets` code 😩 
2022-05-27T06:14:12Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1139316205
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5D6JXt
https://api.github.com/repos/huggingface/datasets/issues/comments/1139316205/reactions
2022-05-27T06:14:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1139316205
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
Sure @albertvillanova I'll do that later today and ping you once done, thanks! :hugs:
2022-05-30T10:27:26Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1140983878
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EAghG
https://api.github.com/repos/huggingface/datasets/issues/comments/1140983878/reactions
2022-05-30T10:27:26Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1140983878
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
Hi again @albertvillanova! Let me know if those tests are fine 🤗 
2022-05-30T11:56:38Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1141066242
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EA0oC
https://api.github.com/repos/huggingface/datasets/issues/comments/1141066242/reactions
2022-05-30T11:56:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1141066242
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
Hi @alvarobartt,

I think your tests are failing. I don't know why previously, after your last commit, the CI tests were not triggered. 

In order to force the re-running of the CI tests, I had to edit your file using the GitHub UI.

First I tried to do it using my terminal, but I don't have push right to your PR branch: I would ask you next time you open a PR, please mark the checkbox "Allow edits from maintainers": https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/allowing-changes-to-a-pull-request-branch-created-from-a-fork#enabling-repository-maintainer-permissions-on-existing-pull-requests
2022-05-30T15:15:25Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1141268655
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EBmCv
https://api.github.com/repos/huggingface/datasets/issues/comments/1141268655/reactions
2022-05-30T15:19:51Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1141268655
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
Hi @albertvillanova, let me check those again! And regarding that checkbox I thought it was already checked so my bad there 😩 
2022-05-30T15:25:45Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1141278853
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EBoiF
https://api.github.com/repos/huggingface/datasets/issues/comments/1141278853/reactions
2022-05-30T15:25:45Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1141278853
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
@albertvillanova  again it seems that the tests were not automatically triggered, but I tested those locally and now they work, as previously those were failing as I created an assertion as `self.assertEqual` over an empty list that was compared as `None` while the value was `[]` so I updated it to be `self.assertListEqual` and changed the comparison value to `[]`.
2022-05-30T15:40:15Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1141291299
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EBrkj
https://api.github.com/repos/huggingface/datasets/issues/comments/1141291299/reactions
2022-05-30T15:40:15Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1141291299
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
@lhoestq any idea why the CI is not triggered?
2022-05-30T15:47:44Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1141297360
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EBtDQ
https://api.github.com/repos/huggingface/datasets/issues/comments/1141297360/reactions
2022-05-30T15:47:44Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1141297360
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
@alvarobartt I have tested locally and the tests continue failing.

I think there is a basis error: `new_dset._format_columns` is always `None` in those cases.

2022-05-30T15:55:09Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1141303344
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EBugw
https://api.github.com/repos/huggingface/datasets/issues/comments/1141303344/reactions
2022-05-30T15:56:54Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1141303344
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
MEMBER
You're right @albertvillanova I was indeed running the tests with `datasets==2.2.0` rather than with the branch version, I'll check it again! Sorry for the inconvenience...
2022-05-30T16:01:17Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1141308146
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EBvry
https://api.github.com/repos/huggingface/datasets/issues/comments/1141308146/reactions
2022-05-30T16:01:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1141308146
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
> @alvarobartt I have tested locally and the tests continue failing.
> 
> I think there is a basis error: `new_dset._format_columns` is always `None` in those cases.

In order to have some regressions tests for the fixed scenario, I've manually updated the value of `_format_columns` in the `ArrowDataset` so as to check whether it's updated or not right after calling `remove_columns`, and it does behave as expected, so with the latest version of this branch the reported issue doesn't occur anymore.
2022-05-30T16:19:40Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1141323779
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EBzgD
https://api.github.com/repos/huggingface/datasets/issues/comments/1141323779/reactions
2022-05-30T16:19:40Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1141323779
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
Hi again @albertvillanova sorry I was on leave! I'll do that ASAP :hugs:
2022-06-09T07:29:37Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1150771613
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5El2Gd
https://api.github.com/repos/huggingface/datasets/issues/comments/1150771613/reactions
2022-06-09T07:29:37Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1150771613
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
@albertvillanova, does it make sense to add regression tests for `DatasetDict`? As `DatasetDict` doesn't have the attribute `_format_columns`, when we call `remove_columns` over a `DatasetDict` it removes the columns and updates the attributes of each split which is an `ArrowDataset`.

So on, we can either:
- Update first the `_format_columns` attribute of each split and then remove the columns over the `DatasetDict`
- Loop over the splits of `DatasetDict` and remove the columns right after updating `_format_columns` of each `ArrowDataset`.

I assume that the best regression test is the one implemented (mentioned first above), let me know if there's a better way to do that 👍🏻 
2022-06-09T09:30:38Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1150892418
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5EmTmC
https://api.github.com/repos/huggingface/datasets/issues/comments/1150892418/reactions
2022-06-09T09:31:09Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1150892418
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
I think there's already a decorator to support transmitting the right `_format_columns`: `@transmit_format`, have you tried adding this decorator to `remove_columns` ?
2022-06-14T14:49:19Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1155298536
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5E3HTo
https://api.github.com/repos/huggingface/datasets/issues/comments/1155298536/reactions
2022-06-14T14:49:19Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1155298536
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
> I think there's already a decorator to support transmitting the right `_format_columns`: `@transmit_format`, have you tried adding this decorator to `remove_columns` ?

Hi @lhoestq I can check now!
2022-06-14T15:12:31Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1155335187
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5E3QQT
https://api.github.com/repos/huggingface/datasets/issues/comments/1155335187/reactions
2022-06-14T15:12:31Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1155335187
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
MEMBER
It worked indeed @lhoestq, thanks for the proposal and the review! 🤗 
2022-06-14T15:25:35Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1155351076
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5E3UIk
https://api.github.com/repos/huggingface/datasets/issues/comments/1155351076/reactions
2022-06-14T15:25:35Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1155351076
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
COLLABORATOR
Oops, I forgot about `@transmit_format`'s existence. From what I see, we should also use this decorator in `flatten`, `rename_column` and  `rename_columns`. 

@alvarobartt Let me know if you'd like to work on this (in a subsequent PR).
2022-06-14T15:53:11Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1155388648
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5E3dTo
https://api.github.com/repos/huggingface/datasets/issues/comments/1155388648/reactions
2022-06-14T15:53:11Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1155388648
https://avatars.githubusercontent.com/u/47462742?v=4
https://api.github.com/users/mariosasko/events{/privacy}
https://api.github.com/users/mariosasko/followers
https://api.github.com/users/mariosasko/following{/other_user}
https://api.github.com/users/mariosasko/gists{/gist_id}
https://github.com/mariosasko
mariosasko
MDQ6VXNlcjQ3NDYyNzQy
https://api.github.com/users/mariosasko/orgs
https://api.github.com/users/mariosasko/received_events
https://api.github.com/users/mariosasko/repos
https://api.github.com/users/mariosasko/starred{/owner}{/repo}
https://api.github.com/users/mariosasko/subscriptions
User
https://api.github.com/users/mariosasko
public
MEMBER
Sure @mariosasko I can prepare another PR to add those too, thanks! 
2022-06-14T15:56:08Z
https://github.com/huggingface/datasets/pull/4411#issuecomment-1155391844
https://api.github.com/repos/huggingface/datasets/issues/4411
IC_kwDODunzps5E3eFk
https://api.github.com/repos/huggingface/datasets/issues/comments/1155391844/reactions
2022-06-14T15:56:08Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1155391844
https://avatars.githubusercontent.com/u/36760800?v=4
https://api.github.com/users/alvarobartt/events{/privacy}
https://api.github.com/users/alvarobartt/followers
https://api.github.com/users/alvarobartt/following{/other_user}
https://api.github.com/users/alvarobartt/gists{/gist_id}
https://github.com/alvarobartt
alvarobartt
MDQ6VXNlcjM2NzYwODAw
https://api.github.com/users/alvarobartt/orgs
https://api.github.com/users/alvarobartt/received_events
https://api.github.com/users/alvarobartt/repos
https://api.github.com/users/alvarobartt/starred{/owner}{/repo}
https://api.github.com/users/alvarobartt/subscriptions
User
https://api.github.com/users/alvarobartt
public
As explained at #4398, when calling `dataset.add_faiss_index` under certain conditions when calling a sequence of operations `cast_column`, `map`, and `remove_columns`, it fails as it's trying to look for already removed columns.

So on, after testing some possible fixes, it seems that setting the dataset format right after removing the columns seems to be working fine, so I had to add a call to `.set_format` in the `remove_columns` function.

Hope this helps!