Improve model card: add paper info, update license metadata, clean up content

This PR enhances the model card for InternVL2_5-78B-MPO by:

- Updating the `license` metadata tag from `other` to `mit` to reflect the project's primary license as declared in the README content. The specific Qwen license for the language model component remains clarified via `license_name` and `license_link`.
- Removing the `

README.md

@@ -1,23 +1,31 @@
 ---
-license: other
-license_name: qwen
-license_link: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE
-pipeline_tag: image-text-to-text
-library_name: transformers
 base_model:
-  - OpenGVLab/InternVL2_5-78B
-base_model_relation: finetune
+- OpenGVLab/InternVL2_5-78B
 datasets:
-  - OpenGVLab/MMPR-v1.1
+- OpenGVLab/MMPR-v1.1
 language:
-  - multilingual
+- multilingual
+library_name: transformers
+license: mit
+license_name: qwen
+license_link: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/blob/main/LICENSE
+pipeline_tag: image-text-to-text
 tags:
-  - internvl
-  - custom_code
+- internvl
+- custom_code
+base_model_relation: finetune
 ---
 
 # InternVL2_5-78B-MPO
 
+## Paper
+The model was presented in the paper [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://huggingface.co/papers/2412.05271).
+
+### Abstract
+We introduce InternVL 2.5, an advanced multimodal large language model (MLLM) series that builds upon InternVL 2.0, maintaining its core model architecture while introducing significant enhancements in training and testing strategies as well as data quality. In this work, we delve into the relationship between model scaling and performance, systematically exploring the performance trends in vision encoders, language models, dataset sizes, and test-time configurations. Through extensive evaluations on a wide range of benchmarks, including multi-discipline reasoning, document understanding, multi-image / video understanding, real-world comprehension, multimodal hallucination detection, visual grounding, multilingual capabilities, and pure language processing, InternVL 2.5 exhibits competitive performance, rivaling leading commercial models such as GPT-4o and Claude-3.5-Sonnet. Notably, our model is the first open-source MLLMs to surpass 70% on the MMMU benchmark, achieving a 3.7-point improvement through Chain-of-Thought (CoT) reasoning and showcasing strong potential for test-time scaling. We hope this model contributes to the open-source community by setting new standards for developing and applying multimodal AI systems. HuggingFace demo see this https URL
+
+---
+
 [\[📂 GitHub\]](https://github.com/OpenGVLab/InternVL)  [\[📜 InternVL 1.0\]](https://huggingface.co/papers/2312.14238)  [\[📜 InternVL 1.5\]](https://huggingface.co/papers/2404.16821)  [\[📜 InternVL 2.5\]](https://huggingface.co/papers/2412.05271)  [\[📜 InternVL2.5-MPO\]](https://huggingface.co/papers/2411.10442)
 
 [\[🆕 Blog\]](https://internvl.github.io/blog/)  [\[🗨️ Chat Demo\]](https://internvl.opengvlab.com/)  [\[🤗 HF Demo\]](https://huggingface.co/spaces/OpenGVLab/InternVL)  [\[🚀 Quick Start\]](#quick-start)  [\[📖 Documents\]](https://internvl.readthedocs.io/en/latest/)
@@ -238,6 +246,7 @@ device_map = split_model('InternVL2_5-78B')
 model = AutoModel.from_pretrained(
     path,
     torch_dtype=torch.bfloat16,
+    load_in_8bit=False,
     low_cpu_mem_usage=True,
     use_flash_attn=True,
     trust_remote_code=True,
@@ -378,40 +387,50 @@ generation_config = dict(max_new_tokens=1024, do_sample=True)
 # pure-text conversation (纯文本对话)
 question = 'Hello, who are you?'
 response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 question = 'Can you tell me a story?'
 response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 # single-image single-round conversation (单图单轮对话)
-question = '<image>\nPlease describe the image shortly.'
+question = '<image>
+Please describe the image shortly.'
 response = model.chat(tokenizer, pixel_values, question, generation_config)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 # single-image multi-round conversation (单图多轮对话)
-question = '<image>\nPlease describe the image in detail.'
+question = '<image>
+Please describe the image in detail.'
 response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 question = 'Please write a poem according to the image.'
 response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 # multi-image multi-round conversation, combined images (多图多轮对话，拼接图像)
 pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
 pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()
 pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
 
-question = '<image>\nDescribe the two images in detail.'
+question = '<image>
+Describe the two images in detail.'
 response, history = model.chat(tokenizer, pixel_values, question, generation_config,
                                history=None, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 question = 'What are the similarities and differences between these two images.'
 response, history = model.chat(tokenizer, pixel_values, question, generation_config,
                                history=history, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 # multi-image multi-round conversation, separate images (多图多轮对话，独立图像)
 pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
@@ -419,17 +438,21 @@ pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat1
 pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
 num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]
 
-question = 'Image-1: <image>\nImage-2: <image>\nDescribe the two images in detail.'
+question = 'Image-1: <image>
+Image-2: <image>
+Describe the two images in detail.'
 response, history = model.chat(tokenizer, pixel_values, question, generation_config,
                                num_patches_list=num_patches_list,
                                history=None, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 question = 'What are the similarities and differences between these two images.'
 response, history = model.chat(tokenizer, pixel_values, question, generation_config,
                                num_patches_list=num_patches_list,
                                history=history, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 # batch inference, single image per sample (单图批处理)
 pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
@@ -437,13 +460,15 @@ pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat1
 num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]
 pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
 
-questions = ['<image>\nDescribe the image in detail.'] * len(num_patches_list)
+questions = ['<image>
+Describe the image in detail.'] * len(num_patches_list)
 responses = model.batch_chat(tokenizer, pixel_values,
                              num_patches_list=num_patches_list,
                              questions=questions,
                              generation_config=generation_config)
 for question, response in zip(questions, responses):
-    print(f'User: {question}\nAssistant: {response}')
+    print(f'User: {question}
+Assistant: {response}')
 
 # video multi-round conversation (视频多轮对话)
 def get_index(bound, fps, max_frame, first_idx=0, num_segments=32):
@@ -481,17 +506,24 @@ def load_video(video_path, bound=None, input_size=448, max_num=1, num_segments=3
 video_path = './examples/red-panda.mp4'
 pixel_values, num_patches_list = load_video(video_path, num_segments=8, max_num=1)
 pixel_values = pixel_values.to(torch.bfloat16).cuda()
-video_prefix = ''.join([f'Frame{i+1}: <image>\n' for i in range(len(num_patches_list))])
+video_prefix = ''.join([f'Frame{i+1}: <image>
+' for i in range(len(num_patches_list))])
 question = video_prefix + 'What is the red panda doing?'
-# Frame1: <image>\nFrame2: <image>\n...\nFrame8: <image>\n{question}
+# Frame1: <image>
+Frame2: <image>
+...
+Frame8: <image>
+{question}
 response, history = model.chat(tokenizer, pixel_values, question, generation_config,
                                num_patches_list=num_patches_list, history=None, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 
 question = 'Describe this video in detail.'
 response, history = model.chat(tokenizer, pixel_values, question, generation_config,
                                num_patches_list=num_patches_list, history=history, return_history=True)
-print(f'User: {question}\nAssistant: {response}')
+print(f'User: {question}
+Assistant: {response}')
 ```
 
 #### Streaming Output
@@ -573,7 +605,9 @@ image_urls=[
 
 images = [load_image(img_url) for img_url in image_urls]
 # Numbering images improves multi-image conversations
-response = pipe((f'Image-1: {IMAGE_TOKEN}\nImage-2: {IMAGE_TOKEN}\ndescribe these two images', images))
+response = pipe((f'Image-1: {IMAGE_TOKEN}
+Image-2: {IMAGE_TOKEN}
+describe these two images', images))
 print(response.text)
 ```
 
@@ -682,9 +716,699 @@ If you find this project useful in your research, please consider citing:
 @article{chen2024far,
   title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
   author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
-  journal={arXiv preprint arXiv:2404.16821},
+  journal={Science China Information Sciences},
+  volume={67},
+  number={12},
+  pages={220101},
+  year={2024},
+  publisher={Springer}
+}
+@inproceedings{chen2024internvl,
+  title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},
+  author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},
+  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+  pages={24185--24198},
   year={2024}
 }
+```
+
+## What can InternVL do?
+
+<details>
+  <summary>Visual Perception (click to expand)</summary>
+
+- Linear-Probe Image Classification [[see details]](./classification#-evaluation)
+
+  ViT-22B uses the private JFT-3B dataset.
+
+  | method              | #param | IN-1K | IN-ReaL | IN-V2 | IN-A  | IN-R  | IN-Sketch |
+  | ------------------- | :----: | :---: | :-----: | :---: | :---: | :---: | :-------: |
+  | OpenCLIP-G          |  1.8B  | 86.2  |  89.4   | 77.2  | 63.8  | 87.8  |   66.4    |
+  | DINOv2-g            |  1.1B  | 86.5  |  89.6   | 78.4  | 75.9  | 78.8  |   62.5    |
+  | EVA-01-CLIP-g       |  1.1B  | 86.5  |  89.3   | 77.4  | 70.5  | 87.7  |   63.1    |
+  | MAWS-ViT-6.5B       |  6.5B  | 87.8  |    -    |   -   |   -   |   -   |     -     |
+  | ViT-22B\*           | 21.7B  | 89.5  |  90.9   | 83.2  | 83.8  | 87.4  |     -     |
+  | InternViT-6B (ours) |  5.9B  | 88.2  |  90.4   | 79.9  | 77.5  | 89.8  |   69.1    |
+
+- Semantic Segmentation [[see details]](./segmentation#-evaluation)
+
+  | method                | decoder | #param (train/total) | crop size | mIoU         |
+  | --------------------- | :-----: | :------------------: | :-------: | ------------ |
+  | OpenCLIP-G (frozen)   | Linear  |     0.3M / 1.8B      |    512    | 39.3         |
+  | ViT-22B (frozen)      | Linear  |     0.9M / 21.7B     |    504    | 34.6         |
+  | InternViT-6B (frozen) | Linear  |     0.5M / 5.9B      |    504    | 47.2 (+12.6) |
+  | ViT-22B (frozen)      | UperNet |     0.8B / 22.5B     |    504    | 52.7         |
+  | InternViT-6B (frozen) | UperNet |     0.4B / 6.3B      |    504    | 54.9 (+2.2)  |
+  | ViT-22B               | UperNet |    22.5B / 22.5B     |    504    | 55.3         |
+  | InternViT-6B          | UperNet |     6.3B / 6.3B      |    504    | 58.9 (+3.6)  |
+
+- Zero-Shot Image Classification [[see details]](./clip_benchmark#imagenet-variants-and-objectnet)
+
+  | method            | IN-1K | IN-A  | IN-R  | IN-V2 | IN-Sketch | ObjectNet |
+  | ----------------- | :---: | :---: | :---: | :---: | :-------: | :-------: |
+  | OpenCLIP-G        | 80.1  | 69.3  | 92.1  | 73.6  |   68.9    |   73.0    |
+  | EVA-02-CLIP-E+    | 82.0  | 82.1  | 94.5  | 75.7  |   71.6    |   79.6    |
+  | ViT-22B\*         | 85.9  | 90.1  | 96.0  | 80.9  |     -     |   87.6    |
+  | InternVL-C (ours) | 83.2  | 83.8  | 95.5  | 77.3  |   73.9    |   80.6    |
+
+- Multilingual Zero-Shot Image Classification [[see details]](./clip_benchmark#multilingual-imagenet-1k)
+
+  EN: English, ZH: Chinese, JP: Japanese, Ar: Arabic, IT: Italian
+
+  | method            | IN-1K (EN) | IN-1K (ZH) | IN-1K (JP) | IN-1K (AR) | IN-1K (IT) |
+  | ----------------- | :--------: | :--------: | :--------: | :--------: | :--------: |
+  | Taiyi-CLIP-ViT-H  |     -      |    54.4    |     -      |     -      |     -      |
+  | WuKong-ViT-L-G    |     -      |    57.5    |     -      |     -      |     -      |
+  | CN-CLIP-ViT-H     |     -      |    59.6    |     -      |     -      |     -      |
+  | AltCLIP-ViT-L     |    74.5    |    59.6    |     -      |     -      |     -      |
+  | EVA-02-CLIP-E+    |    82.0    |     -      |     -      |     -      |    41.2    |
+  | OpenCLIP-XLM-R-H  |    77.0    |    55.7    |    53.1    |    37.0    |    56.8    |
+  | InternVL-C (ours) |    83.2    |    64.5    |    61.5    |    44.9    |    65.7    |
+
+- Zero-Shot Video Classification
+
+  | method            | #frame | K400  | K600  | K700  |
+  | ----------------- | :----: | :---: | :---: | :---: |
+  | OpenCLIP-G        |   1    | 65.9  | 66.1  | 59.2  |
+  | EVA-02-CLIP-E+    |   1    | 69.8  | 69.3  | 63.4  |
+  | InternVL-C (ours) |   1    | 71.0  | 71.3  | 65.7  |
+  | ViCLIP            |   8    | 75.7  | 73.5  | 66.4  |
+  | InternVL-C (ours) |   8    | 79.4  | 78.8  | 71.5  |
+
+</details>
+
+<details>
+  <summary>Cross-Modal Retrieval (click to expand)</summary>
+
+- English Zero-Shot Image-Text Retrieval [[see details]](./clip_benchmark#flickr30k--coco)
+
+  <table>
+    <tr align=center>
+        <td rowspan="3" align=left><b>model</b></td>
+        <td colspan="6" align=center><b>Flickr30K</b></td>
+        <td colspan="6" align=center><b>COCO</b></td>
+        <td rowspan="3" align=center><b>avg</b></td>
+    </tr>
+     <tr align=center>
+        <td colspan="3" align=center><b>image-to-text</b></td>
+        <td colspan="3" align=center><b>text-to-image</b></td>
+         <td colspan="3" align=center><b>image-to-text</b></td>
+        <td colspan="3" align=center><b>text-to-image</b></td>
+     </tr>
+     <tr>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+     </tr>
+  <tr align=center>
+        <td align=left>OpenCLIP-G</td>
+        <td>92.9</td>
+        <td>99.3</td>
+        <td>99.8</td>
+        <td>79.5</td>
+        <td>95.0</td>
+        <td>97.1</td>
+        <td>67.3</td>
+        <td>86.9</td>
+        <td>92.6</td>
+        <td>51.4</td>
+        <td>74.9</td>
+        <td>83.0</td>
+        <td>85.0</td>
+     </tr>
+  <tr align=center>
+        <td align=left>EVA-02-CLIP-E+</td>
+        <td>93.9</td>
+        <td>99.4</td>
+        <td>99.8</td>
+        <td>78.8</td>
+        <td>94.2</td>
+        <td>96.8</td>
+        <td>68.8</td>
+        <td>87.8</td>
+        <td>92.8</td>
+        <td>51.1</td>
+        <td>75.0</td>
+        <td>82.7</td>
+        <td>85.1</td>
+     </tr>
+    <tr align=center>
+        <td align=left>EVA-CLIP-8B</td>
+        <td>95.6</td>
+        <td>99.6</td>
+        <td>99.9</td>
+        <td>80.8</td>
+        <td>95.5</td>
+        <td>97.6</td>
+        <td>70.3</td>
+        <td>89.3</td>
+        <td>93.9</td>
+        <td>53.0</td>
+        <td>76.0</td>
+        <td>83.4</td>
+        <td>86.2</td>
+     </tr>
+  <tr align=center>
+        <td align=left>InternVL-C (ours)</td>
+        <td>94.7</td>
+        <td>99.6</td>
+        <td>99.9</td>
+        <td>81.7</td>
+        <td>96.0</td>
+        <td>98.2</td>
+        <td>70.6</td>
+        <td>89.0</td>
+        <td>93.5</td>
+        <td>54.1</td>
+        <td>77.3</td>
+        <td>84.6</td>
+        <td>86.6</td>
+     </tr>
+  <tr align=center>
+        <td align=left>InternVL-G (ours)</td>
+        <td>95.7</td>
+        <td>99.7</td>
+        <td>99.9</td>
+        <td>85.0</td>
+        <td>97.0</td>
+        <td>98.6</td>
+        <td>74.9</td>
+        <td>91.3</td>
+        <td>95.2</td>
+        <td>58.6</td>
+        <td>81.3</td>
+        <td>88.0</td>
+        <td>88.8</td>
+     </tr>
+
+  </table>
+
+- Chinese Zero-Shot Image-Text Retrieval [[see details]](./clip_benchmark#flickr30k-cn--coco-cn)
+
+  <table>
+    <tr  align=center>
+        <td rowspan="3" align=left><b>model</b></td>
+        <td colspan="6" align=center><b>Flickr30K-CN</b></td>
+        <td colspan="6" align=center><b>COCO-CN</b></td>
+        <td rowspan="3" align=center><b>avg</b></td>
+
+  </tr>
+     <tr  align=center>
+        <td colspan="3" align=center><b>image-to-text</b></td>
+        <td colspan="3" align=center><b>text-to-image</b></td>
+         <td colspan="3" align=center><b>image-to-text</b></td>
+        <td colspan="3" align=center><b>text-to-image</b></td>
+     </tr>
+     <tr>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+     </tr>
+
+  <tr align=center>
+        <td align=left>CN-CLIP-ViT-H</td>
+        <td>81.6</td>
+        <td>97.5</td>
+        <td>98.8</td>
+        <td>71.2</td>
+        <td>91.4</td>
+        <td>95.5</td>
+        <td>63.0</td>
+        <td>86.6</td>
+        <td>92.9</td>
+        <td>69.2</td>
+        <td>89.9</td>
+        <td>96.1</td>
+        <td>86.1</td>
+     </tr>
+
+  <tr align=center>
+        <td align=left>OpenCLIP-XLM-R-H</td>
+        <td>86.1</td>
+        <td>97.5</td>
+        <td>99.2</td>
+        <td>71.0</td>
+        <td>90.5</td>
+        <td>94.9</td>
+        <td>70.0</td>
+        <td>91.5</td>
+        <td>97.0</td>
+        <td>66.1</td>
+        <td>90.8</td>
+        <td>96.0</td>
+        <td>87.6</td>
+     </tr>
+
+  <tr align=center>
+        <td align=left>InternVL-C (ours)</td>
+        <td>90.3</td>
+        <td>98.8</td>
+        <td>99.7</td>
+        <td>75.1</td>
+        <td>92.9</td>
+        <td>96.4</td>
+        <td>68.8</td>
+        <td>92.0</td>
+        <td>96.7</td>
+        <td>68.9</td>
+        <td>91.9</td>
+        <td>96.5</td>
+        <td>89.0</td>
+     </tr>
+  <tr align=center>
+        <td align=left>InternVL-G (ours)</td>
+        <td>92.9</td>
+        <td>99.4</td>
+        <td>99.8</td>
+        <td>77.7</td>
+        <td>94.8</td>
+        <td>97.3</td>
+        <td>71.4</td>
+        <td>93.9</td>
+        <td>97.7</td>
+        <td>73.8</td>
+        <td>94.4</td>
+        <td>98.1</td>
+        <td>90.9</td>
+     </tr>
+
+  </table>
+
+- Multilingual Zero-Shot Image-Text Retrieval on XTD [[see details]](./clip_benchmark#xtd)
+
+  | method            |  EN   |  ES   |  FR   |  ZH   |  IT   |  KO   |  RU   |  JP   | average |
+  | ----------------- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :-----: |
+  | AltCLIP           | 95.4  | 94.1  | 92.9  | 95.1  | 94.2  | 94.4  | 91.8  | 91.7  |  93.7   |
+  | OpenCLIP-XLM-R-H  | 97.3  | 96.1  | 94.5  | 94.7  | 96.0  | 90.2  | 93.9  | 94.0  |  94.6   |
+  | InternVL-C (ours) | 97.3  | 95.7  | 95.1  | 95.6  | 96.0  | 92.2  | 93.3  | 95.5  |  95.1   |
+  | InternVL-G (ours) | 98.6  | 97.7  | 96.5  | 96.7  | 96.9  | 95.1  | 94.8  | 96.1  |  96.6   |
+
+</details>
+
+<details>
+  <summary>Multimodal Dialogue</summary>
+
+</details>
+
+## Quick Start with HuggingFace
+
+<details>
+  <summary>using InternViT-6B for visual feature extraction (click to expand)</summary>
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModel, CLIPImageProcessor
+
+model = AutoModel.from_pretrained(
+    'OpenGVLab/InternViT-6B-448px-V2_5',
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True).cuda().eval()
+
+image = Image.open('./examples/image1.jpg').convert('RGB')
+
+image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-5')
+
+pixel_values = image_processor(images=image, return_tensors='pt').pixel_values
+pixel_values = pixel_values.to(torch.bfloat16).cuda()
+
+outputs = model(pixel_values)
+```
+
+</details>
+
+<details>
+  <summary>using InternVL-C(ontrastive) and InternVL-G(enerative) for cross-modal retrieval (click to expand)</summary>
+
+```python
+import torch
+from PIL import Image
+from transformers import AutoModel, CLIPImageProcessor
+from transformers import AutoTokenizer
+
+
+model = AutoModel.from_pretrained(
+    'OpenGVLab/InternVL-14B-224px',
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True).cuda().eval()
+
+image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternVL-14B-224px')
+
+tokenizer = AutoTokenizer.from_pretrained(
+    'OpenGVLab/InternVL-14B-224px', use_fast=False, add_eos_token=True)
+tokenizer.pad_token_id = 0  # set pad_token_id to 0
+
+images = [
+    Image.open('./examples/image1.jpg').convert('RGB'),
+    Image.open('./examples/image2.jpg').convert('RGB'),
+    Image.open('./examples/image3.jpg').convert('RGB')
+]
+prefix = 'summarize:'
+texts = [
+    prefix + 'a photo of a red panda',  # English
+    prefix + '一张熊猫的照片',  # Chinese
+    prefix + '二匹の猫の写真'  # Japanese
+]
+
+pixel_values = image_processor(images=images, return_tensors='pt').pixel_values
+pixel_values = pixel_values.to(torch.bfloat16).cuda()
+input_ids = tokenizer(texts, return_tensors='pt', max_length=80,
+                      truncation=True, padding='max_length').input_ids.cuda()
+
+# InternVL-C
+logits_per_image, logits_per_text = model(
+    image=pixel_values, text=input_ids, mode='InternVL-C')
+probs = logits_per_image.softmax(dim=-1)
+# tensor([[9.9609e-01, 5.2185e-03, 6.0070e-08],
+#         [2.2949e-02, 9.7656e-01, 5.9903e-06],
+#         [3.2932e-06, 7.4863e-05, 1.0000e+00]], device='cuda:0',
+#        dtype=torch.bfloat16, grad_fn=<SoftmaxBackward0>)
+
+# InternVL-G
+logits_per_image, logits_per_text = model(
+    image=pixel_values, text=input_ids, mode='InternVL-G')
+probs = logits_per_image.softmax(dim=-1)
+# tensor([[9.9609e-01, 3.1738e-03, 3.6322e-08],
+#         [8.6060e-03, 9.9219e-01, 2.8759e-06],
+#         [1.7583e-06, 3.1233e-05, 1.0000e+00]], device='cuda:0',
+#        dtype=torch.bfloat16, grad_fn=<SoftmaxBackward0>)
+
+# please set add_eos_token to False for generation
+tokenizer.add_eos_token = False
+image = Image.open('./examples/image1.jpg').convert('RGB')
+pixel_values = image_processor(images=image, return_tensors='pt').pixel_values
+pixel_values = pixel_values.to(torch.bfloat16).cuda()
+
+tokenized = tokenizer("English caption:", return_tensors='pt')
+pred = model.generate(
+    pixel_values=pixel_values,
+    input_ids=tokenized.input_ids.cuda(),
+    attention_mask=tokenized.attention_mask.cuda(),
+    num_beams=5,
+    min_new_tokens=8,
+)
+caption = tokenizer.decode(pred[0].cpu(), skip_special_tokens=True).strip()
+# English caption: a red panda sitting on top of a wooden platform
+```
+
+</details>
+
+<details>
+  <summary>using InternVL 2.5 for multimodal chat (click to expand)</summary>
+
+Here, we take the smaller `OpenGVLab/InternVL2_5-8B` as an example:
+
+```python
+import numpy as np
+import torch
+import torchvision.transforms as T
+from decord import VideoReader, cpu
+from PIL import Image
+from torchvision.transforms.functional import InterpolationMode
+from transformers import AutoModel, AutoTokenizer
+
+IMAGENET_MEAN = (0.485, 0.456, 0.406)
+IMAGENET_STD = (0.229, 0.224, 0.225)
+
+def build_transform(input_size):
+    MEAN, STD = IMAGENET_MEAN, IMAGENET_STD
+    transform = T.Compose([
+        T.Lambda(lambda img: img.convert('RGB') if img.mode != 'RGB' else img),
+        T.Resize((input_size, input_size), interpolation=InterpolationMode.BICUBIC),
+        T.ToTensor(),
+        T.Normalize(mean=MEAN, std=STD)
+    ])
+    return transform
+
+def find_closest_aspect_ratio(aspect_ratio, target_ratios, width, height, image_size):
+    best_ratio_diff = float('inf')
+    best_ratio = (1, 1)
+    area = width * height
+    for ratio in target_ratios:
+        target_aspect_ratio = ratio[0] / ratio[1]
+        ratio_diff = abs(aspect_ratio - target_aspect_ratio)
+        if ratio_diff < best_ratio_diff:
+            best_ratio_diff = ratio_diff
+            best_ratio = ratio
+        elif ratio_diff == best_ratio_diff:
+            if area > 0.5 * image_size * image_size * ratio[0] * ratio[1]:
+                best_ratio = ratio
+    return best_ratio
+
+def dynamic_preprocess(image, min_num=1, max_num=12, image_size=448, use_thumbnail=False):
+    orig_width, orig_height = image.size
+    aspect_ratio = orig_width / orig_height
+
+    # calculate the existing image aspect ratio
+    target_ratios = set(
+        (i, j) for n in range(min_num, max_num + 1) for i in range(1, n + 1) for j in range(1, n + 1) if
+        i * j <= max_num and i * j >= min_num)
+    target_ratios = sorted(target_ratios, key=lambda x: x[0] * x[1])
+
+    # find the closest aspect ratio to the target
+    target_aspect_ratio = find_closest_aspect_ratio(
+        aspect_ratio, target_ratios, orig_width, orig_height, image_size)
+
+    # calculate the target width and height
+    target_width = image_size * target_aspect_ratio[0]
+    target_height = image_size * target_aspect_ratio[1]
+    blocks = target_aspect_ratio[0] * target_aspect_ratio[1]
+
+    # resize the image
+    resized_img = image.resize((target_width, target_height))
+    processed_images = []
+    for i in range(blocks):
+        box = (
+            (i % (target_width // image_size)) * image_size,
+            (i // (target_width // image_size)) * image_size,
+            ((i % (target_width // image_size)) + 1) * image_size,
+            ((i // (target_width // image_size)) + 1) * image_size
+        )
+        # split the image
+        split_img = resized_img.crop(box)
+        processed_images.append(split_img)
+    assert len(processed_images) == blocks
+    if use_thumbnail and len(processed_images) != 1:
+        thumbnail_img = image.resize((image_size, image_size))
+        processed_images.append(thumbnail_img)
+    return processed_images
+
+def load_image(image_file, input_size=448, max_num=12):
+    image = Image.open(image_file).convert('RGB')
+    transform = build_transform(input_size=input_size)
+    images = dynamic_preprocess(image, image_size=input_size, use_thumbnail=True, max_num=max_num)
+    pixel_values = [transform(image) for image in images]
+    pixel_values = torch.stack(pixel_values)
+    return pixel_values
+
+# If you have an 80G A100 GPU, you can put the entire model on a single GPU.
+# Otherwise, you need to load a model using multiple GPUs, please refer to the `Multiple GPUs` section.
+path = 'OpenGVLab/InternVL2_5-8B'
+model = AutoModel.from_pretrained(
+    path,
+    torch_dtype=torch.bfloat16,
+    low_cpu_mem_usage=True,
+    trust_remote_code=True).eval().cuda()
+tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True, use_fast=False)
+
+# set the max number of tiles in `max_num`
+pixel_values = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
+generation_config = dict(max_new_tokens=1024, do_sample=False)
+
+# pure-text conversation (纯文本对话)
+question = 'Hello, who are you?'
+response, history = model.chat(tokenizer, None, question, generation_config, history=None, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+question = 'Can you tell me a story?'
+response, history = model.chat(tokenizer, None, question, generation_config, history=history, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+# single-image single-round conversation (单图单轮对话)
+question = '<image>
+Please describe the image shortly.'
+response = model.chat(tokenizer, pixel_values, question, generation_config)
+print(f'User: {question}
+Assistant: {response}')
+
+# single-image multi-round conversation (单图多轮对话)
+question = '<image>
+Please describe the image in detail.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=None, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+question = 'Please write a poem according to the image.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config, history=history, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+# multi-image multi-round conversation, combined images (多图多轮对话，拼接图像)
+pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
+pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()
+pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
+
+question = '<image>
+Describe the two images in detail.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config,
+                               history=None, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+question = 'What are the similarities and differences between these two images.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config,
+                               history=history, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+# multi-image multi-round conversation, separate images (多图多轮对话，独立图像)
+pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
+pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()
+pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
+num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]
+
+question = 'Image-1: <image>
+Image-2: <image>
+Describe the two images in detail.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config,
+                               num_patches_list=num_patches_list,
+                               history=None, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+question = 'What are the similarities and differences between these two images.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config,
+                               num_patches_list=num_patches_list, history=history, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+# batch inference, single image per sample (单图批处理)
+pixel_values1 = load_image('./examples/image1.jpg', max_num=12).to(torch.bfloat16).cuda()
+pixel_values2 = load_image('./examples/image2.jpg', max_num=12).to(torch.bfloat16).cuda()
+num_patches_list = [pixel_values1.size(0), pixel_values2.size(0)]
+pixel_values = torch.cat((pixel_values1, pixel_values2), dim=0)
+
+questions = ['<image>
+Describe the image in detail.'] * len(num_patches_list)
+responses = model.batch_chat(tokenizer, pixel_values,
+                             num_patches_list=num_patches_list,
+                             questions=questions,
+                             generation_config=generation_config)
+for question, response in zip(questions, responses):
+    print(f'User: {question}
+Assistant: {response}')
+
+# video multi-round conversation (视频多轮对话)
+def get_index(bound, fps, max_frame, first_idx=0, num_segments=32):
+    if bound:
+        start, end = bound[0], bound[1]
+    else:
+        start, end = -100000, 100000
+    start_idx = max(first_idx, round(start * fps))
+    end_idx = min(round(end * fps), max_frame)
+    seg_size = float(end_idx - start_idx) / num_segments
+    frame_indices = np.array([
+        int(start_idx + (seg_size / 2) + np.round(seg_size * idx))
+        for idx in range(num_segments)
+    ])
+    return frame_indices
+
+def load_video(video_path, bound=None, input_size=448, max_num=1, num_segments=32):
+    vr = VideoReader(video_path, ctx=cpu(0), num_threads=1)
+    max_frame = len(vr) - 1
+    fps = float(vr.get_avg_fps())
+
+    pixel_values_list, num_patches_list = [], []
+    transform = build_transform(input_size=input_size)
+    frame_indices = get_index(bound, fps, max_frame, first_idx=0, num_segments=num_segments)
+    for frame_index in frame_indices:
+        img = Image.fromarray(vr[frame_index].asnumpy()).convert('RGB')
+        img = dynamic_preprocess(img, image_size=input_size, use_thumbnail=True, max_num=max_num)
+        pixel_values = [transform(tile) for tile in img]
+        pixel_values = torch.stack(pixel_values)
+        num_patches_list.append(pixel_values.shape[0])
+        pixel_values_list.append(pixel_values)
+    pixel_values = torch.cat(pixel_values_list)
+    return pixel_values, num_patches_list
+
+video_path = './examples/red-panda.mp4'
+pixel_values, num_patches_list = load_video(video_path, num_segments=8, max_num=1)
+pixel_values = pixel_values.to(torch.bfloat16).cuda()
+video_prefix = ''.join([f'Frame-{i+1}: <image>
+' for i in range(len(num_patches_list))])
+question = video_prefix + 'What is the red panda doing?'
+# Frame1: <image>
+Frame2: <image>
+...
+Frame8: <image>
+{question}
+response, history = model.chat(tokenizer, pixel_values, question, generation_config,
+                               num_patches_list=num_patches_list, history=None, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+
+question = 'Describe this video in detail.'
+response, history = model.chat(tokenizer, pixel_values, question, generation_config,
+                               num_patches_list=num_patches_list, history=history, return_history=True)
+print(f'User: {question}
+Assistant: {response}')
+```
+
+</details>
+
+## License
+
+This project is released under the MIT License. This project uses the pre-trained Qwen2.5-72B-Instruct as a component, which is licensed under the Qwen License.
+
+## Citation
+
+If you find this project useful in your research, please consider citing:
+
+```BibTeX
+@article{wang2024mpo,
+  title={Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization},
+  author={Wang, Weiyun and Chen, Zhe and Wang, Wenhai and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Zhu, Jinguo and Zhu, Xizhou and Lu, Lewei and Qiao, Yu and Dai, Jifeng},
+  journal={arXiv preprint arXiv:2411.10442},
+  year={2024}
+}
+@article{chen2024expanding,
+  title={Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling},
+  author={Chen, Zhe and Wang, Weiyun and Cao, Yue and Liu, Yangzhou and Gao, Zhangwei and Cui, Erfei and Zhu, Jinguo and Ye, Shenglong and Tian, Hao and Liu, Zhaoyang and others},
+  journal={arXiv preprint arXiv:2412.05271},
+  year={2024}
+}
+@article{chen2024far,
+  title={How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites},
+  author={Chen, Zhe and Wang, Weiyun and Tian, Hao and Ye, Shenglong and Gao, Zhangwei and Cui, Erfei and Tong, Wenwen and Hu, Kongzhi and Luo, Jiapeng and Ma, Zheng and others},
+  journal={Science China Information Sciences},
+  volume={67},
+  number={12},
+  pages={220101},
+  year={2024},
+  publisher={Springer}
+}
 @inproceedings{chen2024internvl,
   title={Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks},
   author={Chen, Zhe and Wu, Jiannan and Wang, Wenhai and Su, Weijie and Chen, Guo and Xing, Sen and Zhong, Muyan and Zhang, Qinglong and Zhu, Xizhou and Lu, Lewei and others},
@@ -693,3 +1417,299 @@ If you find this project useful in your research, please consider citing:
   year={2024}
 }
 ```
+
+## What can InternVL do?
+
+<details>
+  <summary>Visual Perception (click to expand)</summary>
+
+- Linear-Probe Image Classification [[see details]](./classification#-evaluation)
+
+  ViT-22B uses the private JFT-3B dataset.
+
+  | method              | #param | IN-1K | IN-ReaL | IN-V2 | IN-A  | IN-R  | IN-Sketch |
+  | ------------------- | :----: | :---: | :-----: | :---: | :---: | :---: | :-------: |
+  | OpenCLIP-G          |  1.8B  | 86.2  |  89.4   | 77.2  | 63.8  | 87.8  |   66.4    |
+  | DINOv2-g            |  1.1B  | 86.5  |  89.6   | 78.4  | 75.9  | 78.8  |   62.5    |
+  | EVA-01-CLIP-g       |  1.1B  | 86.5  |  89.3   | 77.4  | 70.5  | 87.7  |   63.1    |
+  | MAWS-ViT-6.5B       |  6.5B  | 87.8  |    -    |   -   |   -   |   -   |     -     |
+  | ViT-22B\*           | 21.7B  | 89.5  |  90.9   | 83.2  | 83.8  | 87.4  |     -     |
+  | InternViT-6B (ours) |  5.9B  | 88.2  |  90.4   | 79.9  | 77.5  | 89.8  |   69.1    |
+
+- Semantic Segmentation [[see details]](./segmentation#-evaluation)
+
+  | method                | decoder | #param (train/total) | crop size | mIoU         |
+  | --------------------- | :-----: | :------------------: | :-------: | ------------ |
+  | OpenCLIP-G (frozen)   | Linear  |     0.3M / 1.8B      |    512    | 39.3         |
+  | ViT-22B (frozen)      | Linear  |     0.9M / 21.7B     |    504    | 34.6         |
+  | InternViT-6B (frozen) | Linear  |     0.5M / 5.9B      |    504    | 47.2 (+12.6) |
+  | ViT-22B (frozen)      | UperNet |     0.8B / 22.5B     |    504    | 52.7         |
+  | InternViT-6B (frozen) | UperNet |     0.4B / 6.3B      |    504    | 54.9 (+2.2)  |
+  | ViT-22B               | UperNet |    22.5B / 22.5B     |    504    | 55.3         |
+  | InternViT-6B          | UperNet |     6.3B / 6.3B      |    504    | 58.9 (+3.6)  |
+
+- Zero-Shot Image Classification [[see details]](./clip_benchmark#imagenet-variants-and-objectnet)
+
+  | method            | IN-1K | IN-A  | IN-R  | IN-V2 | IN-Sketch | ObjectNet |
+  | ----------------- | :---: | :---: | :---: | :---: | :-------: | :-------: |
+  | OpenCLIP-G        | 80.1  | 69.3  | 92.1  | 73.6  |   68.9    |   73.0    |
+  | EVA-02-CLIP-E+    | 82.0  | 82.1  | 94.5  | 75.7  |   71.6    |   79.6    |
+  | ViT-22B\*         | 85.9  | 90.1  | 96.0  | 80.9  |     -     |   87.6    |
+  | InternVL-C (ours) | 83.2  | 83.8  | 95.5  | 77.3  |   73.9    |   80.6   |
+
+- Multilingual Zero-Shot Image Classification [[see details]](./clip_benchmark#multilingual-imagenet-1k)
+
+  EN: English, ZH: Chinese, JP: Japanese, Ar: Arabic, IT: Italian
+
+  | method            | IN-1K (EN) | IN-1K (ZH) | IN-1K (JP) | IN-1K (AR) | IN-1K (IT) |
+  | ----------------- | :--------: | :--------: | :--------: | :--------: | :--------: |
+  | Taiyi-CLIP-ViT-H  |     -      |    54.4    |     -      |     -      |     -      |
+  | WuKong-ViT-L-G    |     -      |    57.5    |     -      |     -      |     -      |
+  | CN-CLIP-ViT-H     |     -      |    59.6    |     -      |     -      |     -      |
+  | AltCLIP-ViT-L     |    74.5    |    59.6    |     -      |     -      |     -      |
+  | EVA-02-CLIP-E+    |    82.0    |     -      |     -      |     -      |    41.2    |
+  | OpenCLIP-XLM-R-H  |    77.0    |    55.7    |    53.1    |    37.0    |    56.8    |
+  | InternVL-C (ours) |    83.2    |    64.5    |    61.5    |    44.9    |    65.7    |
+
+- Zero-Shot Video Classification
+
+  | method            | #frame | K400  | K600  | K700  |
+  | ----------------- | :----: | :---: | :---: | :---: |
+  | OpenCLIP-G        |   1    | 65.9  | 66.1  | 59.2  |
+  | EVA-02-CLIP-E+    |   1    | 69.8  | 69.3  | 63.4  |
+  | InternVL-C (ours) |   1    | 71.0  | 71.3  | 65.7  |
+  | ViCLIP            |   8    | 75.7  | 73.5  | 66.4  |
+  | InternVL-C (ours) |   8    | 79.4  | 78.8  | 71.5  |
+
+</details>
+
+<details>
+  <summary>Cross-Modal Retrieval (click to expand)</summary>
+
+- English Zero-Shot Image-Text Retrieval [[see details]](./clip_benchmark#flickr30k--coco)
+
+  <table>
+    <tr align=center>
+        <td rowspan="3" align=left><b>model</b></td>
+        <td colspan="6" align=center><b>Flickr30K</b></td>
+        <td colspan="6" align=center><b>COCO</b></td>
+        <td rowspan="3" align=center><b>avg</b></td>
+    </tr>
+     <tr align=center>
+        <td colspan="3" align=center><b>image-to-text</b></td>
+        <td colspan="3" align=center><b>text-to-image</b></td>
+         <td colspan="3" align=center><b>image-to-text</b></td>
+        <td colspan="3" align=center><b>text-to-image</b></td>
+     </tr>
+     <tr>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+     </tr>
+  <tr align=center>
+        <td align=left>OpenCLIP-G</td>
+        <td>92.9</td>
+        <td>99.3</td>
+        <td>99.8</td>
+        <td>79.5</td>
+        <td>95.0</td>
+        <td>97.1</td>
+        <td>67.3</td>
+        <td>86.9</td>
+        <td>92.6</td>
+        <td>51.4</td>
+        <td>74.9</td>
+        <td>83.0</td>
+        <td>85.0</td>
+     </tr>
+  <tr align=center>
+        <td align=left>EVA-02-CLIP-E+</td>
+        <td>93.9</td>
+        <td>99.4</td>
+        <td>99.8</td>
+        <td>78.8</td>
+        <td>94.2</td>
+        <td>96.8</td>
+        <td>68.8</td>
+        <td>87.8</td>
+        <td>92.8</td>
+        <td>51.1</td>
+        <td>75.0</td>
+        <td>82.7</td>
+        <td>85.1</td>
+     </tr>
+    <tr align=center>
+        <td align=left>EVA-CLIP-8B</td>
+        <td>95.6</td>
+        <td>99.6</td>
+        <td>99.9</td>
+        <td>80.8</td>
+        <td>95.5</td>
+        <td>97.6</td>
+        <td>70.3</td>
+        <td>89.3</td>
+        <td>93.9</td>
+        <td>53.0</td>
+        <td>76.0</td>
+        <td>83.4</td>
+        <td>86.2</td>
+     </tr>
+  <tr align=center>
+        <td align=left>InternVL-C (ours)</td>
+        <td>94.7</td>
+        <td>99.6</td>
+        <td>99.9</td>
+        <td>81.7</td>
+        <td>96.0</td>
+        <td>98.2</td>
+        <td>70.6</td>
+        <td>89.0</td>
+        <td>93.5</td>
+        <td>54.1</td>
+        <td>77.3</td>
+        <td>84.6</td>
+        <td>86.6</td>
+     </tr>
+  <tr align=center>
+        <td align=left>InternVL-G (ours)</td>
+        <td>95.7</td>
+        <td>99.7</td>
+        <td>99.9</td>
+        <td>85.0</td>
+        <td>97.0</td>
+        <td>98.6</td>
+        <td>74.9</td>
+        <td>91.3</td>
+        <td>95.2</td>
+        <td>58.6</td>
+        <td>81.3</td>
+        <td>88.0</td>
+        <td>88.8</td>
+     </tr>
+
+  </table>
+
+- Chinese Zero-Shot Image-Text Retrieval [[see details]](./clip_benchmark#flickr30k-cn--coco-cn)
+
+  <table>
+    <tr  align=center>
+        <td rowspan="3" align=left><b>model</b></td>
+        <td colspan="6" align=center><b>Flickr30K-CN</b></td>
+        <td colspan="6" align=center><b>COCO-CN</b></td>
+        <td rowspan="3" align=center><b>avg</b></td>
+
+  </tr>
+     <tr  align=center>
+        <td colspan="3" align=center><b>image-to-text</b></td>
+        <td colspan="3" align=center><b>text-to-image</b></td>
+         <td colspan="3" align=center><b>image-to-text</b></td>
+        <td colspan="3" align=center><b>text-to-image</b></td>
+     </tr>
+     <tr>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+        <td>R@1</td>
+        <td>R@5</td>
+        <td>R@10</td>
+     </tr>
+
+  <tr align=center>
+        <td align=left>CN-CLIP-ViT-H</td>
+        <td>81.6</td>
+        <td>97.5</td>
+        <td>98.8</td>
+        <td>71.2</td>
+        <td>91.4</td>
+        <td>95.5</td>
+        <td>63.0</td>
+        <td>86.6</td>
+        <td>92.9</td>
+        <td>69.2</td>
+        <td>89.9</td>
+        <td>96.1</td>
+        <td>86.1</td>
+     </tr>
+
+  <tr align=center>
+        <td align=left>OpenCLIP-XLM-R-H</td>
+        <td>86.1</td>
+        <td>97.5</td>
+        <td>99.2</td>
+        <td>71.0</td>
+        <td>90.5</td>
+        <td>94.9</td>
+        <td>70.0</td>
+        <td>91.5</td>
+        <td>97.0</td>
+        <td>66.1</td>
+        <td>90.8</td>
+        <td>96.0</td>
+        <td>87.6</td>
+     </tr>
+
+  <tr align=center>
+        <td align=left>InternVL-C (ours)</td>
+        <td>90.3</td>
+        <td>98.8</td>
+        <td>99.7</td>
+        <td>75.1</td>
+        <td>92.9</td>
+        <td>96.4</td>
+        <td>68.8</td>
+        <td>92.0</td>
+        <td>96.7</td>
+        <td>68.9</td>
+        <td>91.9</td>
+        <td>96.5</td>
+        <td>89.0</td>
+     </tr>
+  <tr align=center>
+        <td align=left>InternVL-G (ours)</td>
+        <td>92.9</td>
+        <td>99.4</td>
+        <td>99.8</td>
+        <td>77.7</td>
+        <td>94.8</td>
+        <td>97.3</td>
+        <td>71.4</td>
+        <td>93.9</td>
+        <td>97.7</td>
+        <td>73.8</td>
+        <td>94.4</td>
+        <td>98.1</td>
+        <td>90.9</td>
+     </tr>
+
+  </table>
+
+- Multilingual Zero-Shot Image-Text Retrieval on XTD [[see details]](./clip_benchmark#xtd)
+
+  | method            |  EN   |  ES   |  FR   |  ZH   |  IT   |  KO   |  RU   |  JP   | average |
+  | ----------------- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :-----: |
+  | AltCLIP           | 95.4  | 94.1  | 92.9  | 95.1  | 94.2  | 94.4  | 91.8  | 91.7  |  93.7   |
+  | OpenCLIP-XLM-R-H  | 97.3  | 96.1  | 94.5  | 94.7  | 96.0  | 90.2  | 93.9  | 94.0  |  94.6   |
+  | InternVL-C (ours) | 97.3  | 95.7  | 95.1  | 95.6  | 96.0  | 92.2  | 93.3  | 95.5  |  95.1   |
+  | InternVL-G (ours) | 98.6  | 97.7  | 96.5  | 96.7  | 96.9  | 95.1  | 94.8  | 96.1  |  96.6   |
+
+</details>
+
+<details>
+  <summary>Multimodal Dialogue</summary>
+
+</details>