MemoryError when computing WER metric
MEMBER
Hi ! Thanks for reporting.
We're indeed using `jiwer` to compute the WER.

Maybe instead of calling `jiwer.wer` once for all the preditions/references we can compute the WER iteratively to avoid memory issues ? I'm not too familial with `jiwer` but this must be possible.

Currently the code to compute the WER is defined here:

https://github.com/huggingface/nlp/blob/349ac4398a3bcae6356f14c5754483383a60e8a4/metrics/wer/wer.py#L93-L94
2021-03-24T10:01:55Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-805665494
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgwNTY2NTQ5NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/805665494/reactions
2021-03-24T10:01:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/805665494
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Hi,

I've just pushed a pull request that is related to this issue https://github.com/huggingface/datasets/pull/2169. It's not iterative, but it should avoid memory errors. It's based on the editdistance python library. An iterative implementation should be as easy as storing scores and words stepwise and dividing at the end.  
2021-04-05T15:47:08Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-813464082
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgxMzQ2NDA4Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/813464082/reactions
2021-04-05T15:49:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/813464082
https://avatars.githubusercontent.com/u/5707233?v=4
https://api.github.com/users/diego-fustes/events{/privacy}
https://api.github.com/users/diego-fustes/followers
https://api.github.com/users/diego-fustes/following{/other_user}
https://api.github.com/users/diego-fustes/gists{/gist_id}
https://github.com/diego-fustes
diego-fustes
MDQ6VXNlcjU3MDcyMzM=
https://api.github.com/users/diego-fustes/orgs
https://api.github.com/users/diego-fustes/received_events
https://api.github.com/users/diego-fustes/repos
https://api.github.com/users/diego-fustes/starred{/owner}{/repo}
https://api.github.com/users/diego-fustes/subscriptions
User
https://api.github.com/users/diego-fustes
public
NONE
I see, this was solved by other thread. Ok, let me know if you want to switch the implementation for any reason :)
2021-04-06T07:42:55Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-813903683
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgxMzkwMzY4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/813903683/reactions
2021-04-06T07:42:55Z
https://api.github.com/repos/huggingface/datasets/issues/comments/813903683
https://avatars.githubusercontent.com/u/5707233?v=4
https://api.github.com/users/diego-fustes/events{/privacy}
https://api.github.com/users/diego-fustes/followers
https://api.github.com/users/diego-fustes/following{/other_user}
https://api.github.com/users/diego-fustes/gists{/gist_id}
https://github.com/diego-fustes
diego-fustes
MDQ6VXNlcjU3MDcyMzM=
https://api.github.com/users/diego-fustes/orgs
https://api.github.com/users/diego-fustes/received_events
https://api.github.com/users/diego-fustes/repos
https://api.github.com/users/diego-fustes/starred{/owner}{/repo}
https://api.github.com/users/diego-fustes/subscriptions
User
https://api.github.com/users/diego-fustes
public
MEMBER
Thanks for diving into this anyway ^^'
As you said this actually got solved a few days ago
2021-04-06T15:04:17Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-814194656
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgxNDE5NDY1Ng==
https://api.github.com/repos/huggingface/datasets/issues/comments/814194656/reactions
2021-04-06T15:04:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/814194656
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
Someone created an issue https://github.com/jitsi/jiwer/issues/40 at jiwer which shows that this is still a problem in the current version. Would be curious to figure out how this can be fixed by jiwer... :) I assume that it runs of out memory because it's trying to compute the WER over (too many) test samples?
2021-04-29T14:47:13Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-829300942
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgyOTMwMDk0Mg==
https://api.github.com/repos/huggingface/datasets/issues/comments/829300942/reactions
2021-04-29T14:47:13Z
https://api.github.com/repos/huggingface/datasets/issues/comments/829300942
https://avatars.githubusercontent.com/u/7225987?v=4
https://api.github.com/users/nikvaessen/events{/privacy}
https://api.github.com/users/nikvaessen/followers
https://api.github.com/users/nikvaessen/following{/other_user}
https://api.github.com/users/nikvaessen/gists{/gist_id}
https://github.com/nikvaessen
nikvaessen
MDQ6VXNlcjcyMjU5ODc=
https://api.github.com/users/nikvaessen/orgs
https://api.github.com/users/nikvaessen/received_events
https://api.github.com/users/nikvaessen/repos
https://api.github.com/users/nikvaessen/starred{/owner}{/repo}
https://api.github.com/users/nikvaessen/subscriptions
User
https://api.github.com/users/nikvaessen
public
MEMBER
Hi !

It's computed iteratively so not sure what could go wrong

https://github.com/huggingface/datasets/blob/8afd0ba8c27800a55ea69d9fcd702dc97d9c16d8/metrics/wer/wer.py#L100-L106

@NiklasHoltmeyer what version of `datasets` are you running ?

2021-04-29T17:03:18Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-829434771
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgyOTQzNDc3MQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/829434771/reactions
2021-04-29T17:03:18Z
https://api.github.com/repos/huggingface/datasets/issues/comments/829434771
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
MEMBER
One possible explanation might be that it is the user who is passing all the sentences in a single element to `wer.compute`?

As current implementation iterates over the elements of `predictions` and `references`, this can be problematic if `predictions` and `references` contain a single huge element each. 

This could be the case, for example, with a single string with all sentences:
```python
result["predicted"] = "One sentence. Other sentence."
```
or with a __double__ nested list of sentence lists
```python
result["predicted"] = [[ ["One sentence."], ["Other sentence"] ]]
```

The user should check the dimensions of the data structure passed to `predictions` and `references`.
2021-04-30T07:13:22Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-829892913
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgyOTg5MjkxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/829892913/reactions
2021-04-30T07:17:23Z
https://api.github.com/repos/huggingface/datasets/issues/comments/829892913
https://avatars.githubusercontent.com/u/8515462?v=4
https://api.github.com/users/albertvillanova/events{/privacy}
https://api.github.com/users/albertvillanova/followers
https://api.github.com/users/albertvillanova/following{/other_user}
https://api.github.com/users/albertvillanova/gists{/gist_id}
https://github.com/albertvillanova
albertvillanova
MDQ6VXNlcjg1MTU0NjI=
https://api.github.com/users/albertvillanova/orgs
https://api.github.com/users/albertvillanova/received_events
https://api.github.com/users/albertvillanova/repos
https://api.github.com/users/albertvillanova/starred{/owner}{/repo}
https://api.github.com/users/albertvillanova/subscriptions
User
https://api.github.com/users/albertvillanova
public
NONE
Hi all,

in my case I was using and older version of datasets and, as @albertvillanova points out, passing the full list of sentences for the metric calculation. The problem was in the way jiwer implements WER, as it tries to compute WER for the full list at once instead of doing it element-wise. I think that with the latest implementation of datasets, or by using the alternative WER function that I've contributed on this [pull request](https://github.com/huggingface/datasets/pull/2169) there shouldn't be memory errors.
2021-04-30T07:49:06Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-829911913
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgyOTkxMTkxMw==
https://api.github.com/repos/huggingface/datasets/issues/comments/829911913/reactions
2021-04-30T07:49:33Z
https://api.github.com/repos/huggingface/datasets/issues/comments/829911913
https://avatars.githubusercontent.com/u/5707233?v=4
https://api.github.com/users/diego-fustes/events{/privacy}
https://api.github.com/users/diego-fustes/followers
https://api.github.com/users/diego-fustes/following{/other_user}
https://api.github.com/users/diego-fustes/gists{/gist_id}
https://github.com/diego-fustes
diego-fustes
MDQ6VXNlcjU3MDcyMzM=
https://api.github.com/users/diego-fustes/orgs
https://api.github.com/users/diego-fustes/received_events
https://api.github.com/users/diego-fustes/repos
https://api.github.com/users/diego-fustes/starred{/owner}{/repo}
https://api.github.com/users/diego-fustes/subscriptions
User
https://api.github.com/users/diego-fustes
public
NONE
@lhoestq i was using Datasets==1.5.0 with 1.6.1 it worked (atleast the first run) but 1.5.0 is not compatible with my preprocessing. i cant save my dataset to a parquet file while using the latest datasets version

-> 
```
  File "../preprocess_dataset.py", line 132, in <module>
    pq.write_table(train_dataset.data, f'{resampled_data_dir}/{data_args.dataset_config_name}.train.parquet')
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 1674, in write_table
    writer.write_table(table, row_group_size=row_group_size)
  File "/usr/local/lib/python3.8/dist-packages/pyarrow/parquet.py", line 588, in write_table
    self.writer.write_table(table, row_group_size=row_group_size)
TypeError: Argument 'table' has incorrect type (expected pyarrow.lib.Table, got ConcatenationTable)
``` 

if i do 
```
import pyarrow.parquet as pq
...
...
pq.write_table(train_dataset.data, 'train.parquet')
pq.write_table(eval_dataset.data, 'eval.parquet')
```

while using 1.6.1. and its working with 1.5.0

2021-04-30T19:37:49Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-830336154
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgzMDMzNjE1NA==
https://api.github.com/repos/huggingface/datasets/issues/comments/830336154/reactions
2021-04-30T19:37:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/830336154
https://avatars.githubusercontent.com/u/18324772?v=4
https://api.github.com/users/NiklasHoltmeyer/events{/privacy}
https://api.github.com/users/NiklasHoltmeyer/followers
https://api.github.com/users/NiklasHoltmeyer/following{/other_user}
https://api.github.com/users/NiklasHoltmeyer/gists{/gist_id}
https://github.com/NiklasHoltmeyer
NiklasHoltmeyer
MDQ6VXNlcjE4MzI0Nzcy
https://api.github.com/users/NiklasHoltmeyer/orgs
https://api.github.com/users/NiklasHoltmeyer/received_events
https://api.github.com/users/NiklasHoltmeyer/repos
https://api.github.com/users/NiklasHoltmeyer/starred{/owner}{/repo}
https://api.github.com/users/NiklasHoltmeyer/subscriptions
User
https://api.github.com/users/NiklasHoltmeyer
public
MEMBER
Hi ! You can pass dataset.data.table instead of dataset.data to pq.write_table
2021-04-30T20:59:25Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-830387297
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgzMDM4NzI5Nw==
https://api.github.com/repos/huggingface/datasets/issues/comments/830387297/reactions
2021-04-30T20:59:25Z
https://api.github.com/repos/huggingface/datasets/issues/comments/830387297
https://avatars.githubusercontent.com/u/42851186?v=4
https://api.github.com/users/lhoestq/events{/privacy}
https://api.github.com/users/lhoestq/followers
https://api.github.com/users/lhoestq/following{/other_user}
https://api.github.com/users/lhoestq/gists{/gist_id}
https://github.com/lhoestq
lhoestq
MDQ6VXNlcjQyODUxMTg2
https://api.github.com/users/lhoestq/orgs
https://api.github.com/users/lhoestq/received_events
https://api.github.com/users/lhoestq/repos
https://api.github.com/users/lhoestq/starred{/owner}{/repo}
https://api.github.com/users/lhoestq/subscriptions
User
https://api.github.com/users/lhoestq
public
NONE
This seems to be working so far! Thanks!
2021-05-01T08:31:49Z
https://github.com/huggingface/datasets/issues/2078#issuecomment-830580665
https://api.github.com/repos/huggingface/datasets/issues/2078
MDEyOklzc3VlQ29tbWVudDgzMDU4MDY2NQ==
https://api.github.com/repos/huggingface/datasets/issues/comments/830580665/reactions
2021-05-01T08:31:49Z
https://api.github.com/repos/huggingface/datasets/issues/comments/830580665
https://avatars.githubusercontent.com/u/18324772?v=4
https://api.github.com/users/NiklasHoltmeyer/events{/privacy}
https://api.github.com/users/NiklasHoltmeyer/followers
https://api.github.com/users/NiklasHoltmeyer/following{/other_user}
https://api.github.com/users/NiklasHoltmeyer/gists{/gist_id}
https://github.com/NiklasHoltmeyer
NiklasHoltmeyer
MDQ6VXNlcjE4MzI0Nzcy
https://api.github.com/users/NiklasHoltmeyer/orgs
https://api.github.com/users/NiklasHoltmeyer/received_events
https://api.github.com/users/NiklasHoltmeyer/repos
https://api.github.com/users/NiklasHoltmeyer/starred{/owner}{/repo}
https://api.github.com/users/NiklasHoltmeyer/subscriptions
User
https://api.github.com/users/NiklasHoltmeyer
public
Hi, I'm trying to follow the ASR example to try Wav2Vec. This is the code that I use for WER calculation:

```
wer = load_metric("wer")
print(wer.compute(predictions=result["predicted"], references=result["target"]))
```

However, I receive the following exception:

`Traceback (most recent call last):
  File "/home/diego/IpGlobal/wav2vec/test_wav2vec.py", line 51, in <module>
    print(wer.compute(predictions=result["predicted"], references=result["target"]))
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/datasets/metric.py", line 403, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "/home/diego/.cache/huggingface/modules/datasets_modules/metrics/wer/73b2d32b723b7fb8f204d785c00980ae4d937f12a65466f8fdf78706e2951281/wer.py", line 94, in _compute
    return wer(references, predictions)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 81, in wer
    truth, hypothesis, truth_transform, hypothesis_transform, **kwargs
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 192, in compute_measures
    H, S, D, I = _get_operation_counts(truth, hypothesis)
  File "/home/diego/miniconda3/envs/wav2vec3.6/lib/python3.6/site-packages/jiwer/measures.py", line 273, in _get_operation_counts
    editops = Levenshtein.editops(source_string, destination_string)
MemoryError`

My system has more than 10GB of available RAM. Looking at the code, I think that it could be related to the way jiwer does the calculation, as it is pasting all the sentences in a single string before calling Levenshtein editops function.



