[Feature Request/Help] BLEURT model -> PyTorch
NONE
Is there any update on this? 

Thanks!
2020-10-06T16:40:12Z
https://github.com/huggingface/datasets/issues/224#issuecomment-704404132
https://api.github.com/repos/huggingface/datasets/issues/224
MDEyOklzc3VlQ29tbWVudDcwNDQwNDEzMg==
https://api.github.com/repos/huggingface/datasets/issues/comments/704404132/reactions
2020-10-06T16:40:12Z
https://api.github.com/repos/huggingface/datasets/issues/comments/704404132
https://avatars.githubusercontent.com/u/17298127?v=4
https://api.github.com/users/manikbhandari/events{/privacy}
https://api.github.com/users/manikbhandari/followers
https://api.github.com/users/manikbhandari/following{/other_user}
https://api.github.com/users/manikbhandari/gists{/gist_id}
https://github.com/manikbhandari
manikbhandari
MDQ6VXNlcjE3Mjk4MTI3
https://api.github.com/users/manikbhandari/orgs
https://api.github.com/users/manikbhandari/received_events
https://api.github.com/users/manikbhandari/repos
https://api.github.com/users/manikbhandari/starred{/owner}{/repo}
https://api.github.com/users/manikbhandari/subscriptions
User
https://api.github.com/users/manikbhandari
public
NONE
Hitting this error when using bleurt with PyTorch ...

```
UnrecognizedFlagError: Unknown command line flag 'f'
```
... and I'm assuming because it was built for TF specifically.  Is there a way to use this metric in PyTorch?
2020-12-27T21:31:38Z
https://github.com/huggingface/datasets/issues/224#issuecomment-751518383
https://api.github.com/repos/huggingface/datasets/issues/224
MDEyOklzc3VlQ29tbWVudDc1MTUxODM4Mw==
https://api.github.com/repos/huggingface/datasets/issues/comments/751518383/reactions
2020-12-27T21:31:38Z
https://api.github.com/repos/huggingface/datasets/issues/comments/751518383
https://avatars.githubusercontent.com/u/14000?v=4
https://api.github.com/users/ohmeow/events{/privacy}
https://api.github.com/users/ohmeow/followers
https://api.github.com/users/ohmeow/following{/other_user}
https://api.github.com/users/ohmeow/gists{/gist_id}
https://github.com/ohmeow
ohmeow
MDQ6VXNlcjE0MDAw
https://api.github.com/users/ohmeow/orgs
https://api.github.com/users/ohmeow/received_events
https://api.github.com/users/ohmeow/repos
https://api.github.com/users/ohmeow/starred{/owner}{/repo}
https://api.github.com/users/ohmeow/subscriptions
User
https://api.github.com/users/ohmeow
public
MEMBER
We currently provide a wrapper on the TensorFlow implementation: https://huggingface.co/metrics/bleurt

We have long term plans to better handle model-based metrics, but they probably won't be implemented right away

@adamwlev it would still be cool to add the BLEURT checkpoints to the transformers repo if you're interested, but that would best be discussed there :) 

closing for now
2021-01-04T09:53:32Z
https://github.com/huggingface/datasets/issues/224#issuecomment-753876014
https://api.github.com/repos/huggingface/datasets/issues/224
MDEyOklzc3VlQ29tbWVudDc1Mzg3NjAxNA==
https://api.github.com/repos/huggingface/datasets/issues/comments/753876014/reactions
2021-01-04T09:53:32Z
https://api.github.com/repos/huggingface/datasets/issues/comments/753876014
https://avatars.githubusercontent.com/u/10469459?v=4
https://api.github.com/users/yjernite/events{/privacy}
https://api.github.com/users/yjernite/followers
https://api.github.com/users/yjernite/following{/other_user}
https://api.github.com/users/yjernite/gists{/gist_id}
https://github.com/yjernite
yjernite
MDQ6VXNlcjEwNDY5NDU5
https://api.github.com/users/yjernite/orgs
https://api.github.com/users/yjernite/received_events
https://api.github.com/users/yjernite/repos
https://api.github.com/users/yjernite/starred{/owner}{/repo}
https://api.github.com/users/yjernite/subscriptions
User
https://api.github.com/users/yjernite
public
NONE
Hi there. We ran into the same problem this year (converting BLEURT to PyTorch) and thanks to @adamwlev found his colab notebook which didn't work but served as a good starting point. Finally, we **made it work** by doing just two simple conceptual fixes: 

1. Transposing 'kernel' layers instead of 'dense' ones when copying params from the original model;
2. Taking pooler_output as a cls_state in forward function of the BleurtModel class.

Plus few minor syntactical fixes for the outdated parts. The result is still not exactly the same, but is very close to the expected one (1.0483 vs 1.0474).

Find the fixed version here (fixes are commented): https://colab.research.google.com/drive/1KsCUkFW45d5_ROSv2aHtXgeBa2Z98r03?usp=sharing  

2021-09-02T15:02:17Z
https://github.com/huggingface/datasets/issues/224#issuecomment-911782388
https://api.github.com/repos/huggingface/datasets/issues/224
IC_kwDODunzps42WLH0
https://api.github.com/repos/huggingface/datasets/issues/comments/911782388/reactions
2021-09-02T15:02:17Z
https://api.github.com/repos/huggingface/datasets/issues/comments/911782388
https://avatars.githubusercontent.com/u/9405872?v=4
https://api.github.com/users/LoraIpsum/events{/privacy}
https://api.github.com/users/LoraIpsum/followers
https://api.github.com/users/LoraIpsum/following{/other_user}
https://api.github.com/users/LoraIpsum/gists{/gist_id}
https://github.com/LoraIpsum
LoraIpsum
MDQ6VXNlcjk0MDU4NzI=
https://api.github.com/users/LoraIpsum/orgs
https://api.github.com/users/LoraIpsum/received_events
https://api.github.com/users/LoraIpsum/repos
https://api.github.com/users/LoraIpsum/starred{/owner}{/repo}
https://api.github.com/users/LoraIpsum/subscriptions
User
https://api.github.com/users/LoraIpsum
public
CONTRIBUTOR
I created a new model based on `transformers` that can load every BLEURT checkpoints released so far. https://github.com/lucadiliello/bleurt-pytorch
2023-01-19T15:46:58Z
https://github.com/huggingface/datasets/issues/224#issuecomment-1397191434
https://api.github.com/repos/huggingface/datasets/issues/224
IC_kwDODunzps5TR3MK
https://api.github.com/repos/huggingface/datasets/issues/comments/1397191434/reactions
2023-01-19T15:46:58Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1397191434
https://avatars.githubusercontent.com/u/23355969?v=4
https://api.github.com/users/lucadiliello/events{/privacy}
https://api.github.com/users/lucadiliello/followers
https://api.github.com/users/lucadiliello/following{/other_user}
https://api.github.com/users/lucadiliello/gists{/gist_id}
https://github.com/lucadiliello
lucadiliello
MDQ6VXNlcjIzMzU1OTY5
https://api.github.com/users/lucadiliello/orgs
https://api.github.com/users/lucadiliello/received_events
https://api.github.com/users/lucadiliello/repos
https://api.github.com/users/lucadiliello/starred{/owner}{/repo}
https://api.github.com/users/lucadiliello/subscriptions
User
https://api.github.com/users/lucadiliello
public
NONE
@LoraIpsum Thanks for sharing your work here. However, I'm unable to reproduce the results. That's strange because you are. FYI, I am trying to convert a finetuned BLEURT to PyTorch. Any suggestions on how I can reproduce results?
2023-08-26T17:38:48Z
https://github.com/huggingface/datasets/issues/224#issuecomment-1694419541
https://api.github.com/repos/huggingface/datasets/issues/224
IC_kwDODunzps5k_spV
https://api.github.com/repos/huggingface/datasets/issues/comments/1694419541/reactions
2023-08-26T17:38:48Z
https://api.github.com/repos/huggingface/datasets/issues/comments/1694419541
https://avatars.githubusercontent.com/u/99115073?v=4
https://api.github.com/users/vaiibhavgupta/events{/privacy}
https://api.github.com/users/vaiibhavgupta/followers
https://api.github.com/users/vaiibhavgupta/following{/other_user}
https://api.github.com/users/vaiibhavgupta/gists{/gist_id}
https://github.com/vaiibhavgupta
vaiibhavgupta
U_kgDOBehgQQ
https://api.github.com/users/vaiibhavgupta/orgs
https://api.github.com/users/vaiibhavgupta/received_events
https://api.github.com/users/vaiibhavgupta/repos
https://api.github.com/users/vaiibhavgupta/starred{/owner}{/repo}
https://api.github.com/users/vaiibhavgupta/subscriptions
User
https://api.github.com/users/vaiibhavgupta
public
Hi, I am interested in porting google research's new BLEURT learned metric to PyTorch (because I wish to do something experimental with language generation and backpropping through BLEURT). I noticed that you guys don't have it yet so I am partly just asking if you plan to add it (@thomwolf said you want to do so on Twitter).

I had a go of just like manually using the checkpoint that they publish which includes the weights. It seems like the architecture is exactly aligned with the out-of-the-box BertModel in transformers just with a single linear layer on top of the CLS embedding. I loaded all the weights to the PyTorch model but I am not able to get the same numbers as the BLEURT package's python api. Here is my colab notebook where I tried  https://colab.research.google.com/drive/1Bfced531EvQP_CpFvxwxNl25Pj6ptylY?usp=sharing . If you have any pointers on what might be going wrong that would be much appreciated!

Thank you muchly!