ArrowCapacityError: List array cannot contain more than 2147483646 child elements, have 2147483648
MEMBER
Can you give us stats/information on your pandas DataFrame?
2020-09-11T08:19:13Z
https://github.com/huggingface/datasets/issues/611#issuecomment-690949805
https://api.github.com/repos/huggingface/datasets/issues/611
MDEyOklzc3VlQ29tbWVudDY5MDk0OTgwNQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/690949805/reactions
2020-09-11T08:19:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/690949805
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
```
<class 'pandas.core.frame.DataFrame'>
Int64Index: 17136104 entries, 0 to 17136103
Data columns (total 6 columns):
 #   Column        Dtype  
---  ------        -----  
 0   item_id       int64  
 1   item_titl     object 
 2   start_price   float64
 3   shipping_fee  float64
 4   picture_url   object 
 5   embeddings    object 
dtypes: float64(2), int64(1), object(3)
memory usage: 915.2+ MB
```
2020-09-11T08:34:12Z
https://github.com/huggingface/datasets/issues/611#issuecomment-690956685
https://api.github.com/repos/huggingface/datasets/issues/611
MDEyOklzc3VlQ29tbWVudDY5MDk1NjY4NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/690956685/reactions
2020-09-11T08:34:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/690956685
https://avatars.githubusercontent.com/u/32364921?v=4
https://api.github.com/users/sangyx/events{/privacy}
https://api.github.com/users/sangyx/followers
https://api.github.com/users/sangyx/following{/other_user}
https://api.github.com/users/sangyx/gists{/gist_id}
https://github.com/sangyx
sangyx
MDQ6VXNlcjMyMzY0OTIx
https://api.github.com/users/sangyx/orgs
https://api.github.com/users/sangyx/received_events
https://api.github.com/users/sangyx/repos
https://api.github.com/users/sangyx/starred{/owner}{/repo}
https://api.github.com/users/sangyx/subscriptions
User
https://api.github.com/users/sangyx
public
MEMBER
Thanks and some more on the `embeddings` and `picture_url` would be nice as well (type and max lengths of the elements)
2020-09-11T08:35:34Z
https://github.com/huggingface/datasets/issues/611#issuecomment-690957344
https://api.github.com/repos/huggingface/datasets/issues/611
MDEyOklzc3VlQ29tbWVudDY5MDk1NzM0NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/690957344/reactions
2020-09-11T08:35:34Z
https://api.github.com/repos/huggingface/datasets/issues/comments/690957344
https://avatars.githubusercontent.com/u/7353373?v=4
https://api.github.com/users/thomwolf/events{/privacy}
https://api.github.com/users/thomwolf/followers
https://api.github.com/users/thomwolf/following{/other_user}
https://api.github.com/users/thomwolf/gists{/gist_id}
https://github.com/thomwolf
thomwolf
MDQ6VXNlcjczNTMzNzM=
https://api.github.com/users/thomwolf/orgs
https://api.github.com/users/thomwolf/received_events
https://api.github.com/users/thomwolf/repos
https://api.github.com/users/thomwolf/starred{/owner}{/repo}
https://api.github.com/users/thomwolf/subscriptions
User
https://api.github.com/users/thomwolf
public
NONE
`embedding` is `np.array` of shape `(128,)`. `picture_url` is url, such as 'https://i.ebayimg.com/00/s/MTE5OVgxNjAw/z/ZOsAAOSwAG9fHQq5/$_12.JPG?set_id=880000500F;https://i.ebayimg.com/00/s/MTE5OVgxNjAw/z/OSgAAOSwokBfHQq8/$_12.JPG?set_id=880000500F'
2020-09-11T08:41:00Z
https://github.com/huggingface/datasets/issues/611#issuecomment-690959851
https://api.github.com/repos/huggingface/datasets/issues/611
MDEyOklzc3VlQ29tbWVudDY5MDk1OTg1MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/690959851/reactions
2020-09-11T08:41:00Z
https://api.github.com/repos/huggingface/datasets/issues/comments/690959851
https://avatars.githubusercontent.com/u/32364921?v=4
https://api.github.com/users/sangyx/events{/privacy}
https://api.github.com/users/sangyx/followers
https://api.github.com/users/sangyx/following{/other_user}
https://api.github.com/users/sangyx/gists{/gist_id}
https://github.com/sangyx
sangyx
MDQ6VXNlcjMyMzY0OTIx
https://api.github.com/users/sangyx/orgs
https://api.github.com/users/sangyx/received_events
https://api.github.com/users/sangyx/repos
https://api.github.com/users/sangyx/starred{/owner}{/repo}
https://api.github.com/users/sangyx/subscriptions
User
https://api.github.com/users/sangyx
public
MEMBER
It looks like a Pyarrow limitation.
I was able to reproduce the error with 

```python
import pandas as pd
import numpy as np
import pyarrow as pa

 n = 1713614
df = pd.DataFrame.from_dict({"a": list(np.zeros((n, 128))), "b": range(n)})
pa.Table.from_pandas(df)
```

I also tried with 50% of the dataframe and it actually works.
I created an issue on Apache Arrow's JIRA [here](https://issues.apache.org/jira/browse/ARROW-9976)

One way to fix that would be to chunk the dataframe and concatenate arrow tables.
2020-09-11T15:25:03Z
https://github.com/huggingface/datasets/issues/611#issuecomment-691160914
https://api.github.com/repos/huggingface/datasets/issues/611
MDEyOklzc3VlQ29tbWVudDY5MTE2MDkxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/691160914/reactions
2020-09-11T15:25:03Z
https://api.github.com/repos/huggingface/datasets/issues/comments/691160914
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
It looks like it's going to be fixed in pyarrow 2.0.0 :)

In the meantime I suggest to chunk big dataframes to create several small datasets, and then concatenate them using [concatenate_datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets)
2020-09-25T15:14:55Z
https://github.com/huggingface/datasets/issues/611#issuecomment-698987641
https://api.github.com/repos/huggingface/datasets/issues/611
MDEyOklzc3VlQ29tbWVudDY5ODk4NzY0MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/698987641/reactions
2020-09-25T15:14:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/698987641
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
Hi, I'm trying to load a dataset from Dataframe, but I get the error:
```bash
---------------------------------------------------------------------------
ArrowCapacityError                        Traceback (most recent call last)
<ipython-input-7-146b6b495963> in <module>
----> 1 dataset = Dataset.from_pandas(emb)

~/miniconda3/envs/dev/lib/python3.7/site-packages/nlp/arrow_dataset.py in from_pandas(cls, df, features, info, split)
    223         info.features = features
    224         pa_table: pa.Table = pa.Table.from_pandas(
--> 225             df=df, schema=pa.schema(features.type) if features is not None else None
    226         )
    227         return cls(pa_table, info=info, split=split)

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.from_pandas()

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/pandas_compat.py in dataframe_to_arrays(df, schema, preserve_index, nthreads, columns, safe)
    591         for i, maybe_fut in enumerate(arrays):
    592             if isinstance(maybe_fut, futures.Future):
--> 593                 arrays[i] = maybe_fut.result()
    594 
    595     types = [x.type for x in arrays]

~/miniconda3/envs/dev/lib/python3.7/concurrent/futures/_base.py in result(self, timeout)
    426                 raise CancelledError()
    427             elif self._state == FINISHED:
--> 428                 return self.__get_result()
    429 
    430             self._condition.wait(timeout)

~/miniconda3/envs/dev/lib/python3.7/concurrent/futures/_base.py in __get_result(self)
    382     def __get_result(self):
    383         if self._exception:
--> 384             raise self._exception
    385         else:
    386             return self._result

~/miniconda3/envs/dev/lib/python3.7/concurrent/futures/thread.py in run(self)
     55 
     56         try:
---> 57             result = self.fn(*self.args, **self.kwargs)
     58         except BaseException as exc:
     59             self.future.set_exception(exc)

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/pandas_compat.py in convert_column(col, field)
    557 
    558         try:
--> 559             result = pa.array(col, type=type_, from_pandas=True, safe=safe)
    560         except (pa.ArrowInvalid,
    561                 pa.ArrowNotImplementedError,

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib.array()

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/array.pxi in pyarrow.lib._ndarray_to_array()

~/miniconda3/envs/dev/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowCapacityError: List array cannot contain more than 2147483646 child elements, have 2147483648
```
My code is :
```python
from nlp import Dataset
dataset = Dataset.from_pandas(emb)
```