| --- |
| language: |
| - ru |
| tags: |
| - PyTorch |
| - Text2Image |
| thumbnail: "https://github.com/sberbank-ai/ru-clip" |
| --- |
| |
| # Model Card: ruCLIP |
| Disclaimer: The code for using model you can found [here](https://github.com/sberbank-ai/ru-clip). |
| # Model Details |
| The ruCLIP model was developed by researchers at SberDevices and Sber AI based on origin OpenAI paper. |
| # Model Type |
| The model uses a ViT-B/32 Transformer architecture (initialized from OpenAI checkpoint and freezed while training) as an image encoder and uses [ruGPT3Small](https://github.com/sberbank-ai/ru-gpts) as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss. |
| # Documents |
| Our habr [post](https://habr.com/ru/company/sberdevices/blog/564440/). |
| # Usage |
| Code for using model you can obtain in our [repo](https://github.com/sberbank-ai/ru-clip). |
|
|
| ``` |
| from clip.evaluate.utils import ( |
| get_text_batch, get_image_batch, get_tokenizer, |
| show_test_images, load_weights_only |
| ) |
| import torch |
| |
| # Load model and tokenizer |
| model, args = load_weights_only("ViT-B/32-small") |
| model = model.cuda().float().eval() |
| tokenizer = get_tokenizer() |
| # Load test images and prepare for model |
| images, texts = show_test_images(args) |
| input_ids, attention_mask = get_text_batch(["Это " + desc for desc in texts], tokenizer, args) |
| img_input = get_image_batch(images, args.img_transform, args) |
| # Call model |
| with torch.no_grad(): |
| logits_per_image, logits_per_text = model( |
| img_input={"x": img_input}, |
| text_input={"x": input_ids, "attention_mask": attention_mask} |
| ) |
| ``` |
|
|
| # Performance |
| We evaluate our model on CIFAR100 and CIFAR10 datasets. |
|
|
| zero-shot classification CIFAR100 top1 accuracy 0.4057; top5 accuracy 0.6975. |
|
|
| zero-shot classification CIFAR10 top1 accuracy 0.7803; top5 accuracy 0.9834. |
