google
/

videoprism-large-f8r288

@@ -6,52 +6,43 @@ tags:
 ---
 # VideoPrism Model Card
-**Paper page**: https://huggingface.co/papers/2402.13217
-**Arxiv**: https://arxiv.org/pdf/2402.13217
-**Github link**: https://github.com/google-deepmind/videoprism
-**Blogpost**: https://research.google/blog/videoprism-a-foundational-visual-encoder-for-video-understanding/
 VideoPrism is a foundational video encoder that enables state-of-the-art performance on a large variety of video understanding tasks. It takes video frames as input and outputs compact embeddings of the frames, which one can conveniently feed into classifiers, LLMs, retrieval models, etc. When tested on 33 public video understanding benchmarks over four task categories, a single frozen VideoPrism checkpoint outperforms previous best-performing foundation models on 31 of them, with no fine-tuning on target task datasets.
-## Model Details
-### Model Description
 VideoPrism-B/L are the composition of a Vision Transformer image encoder and four temporal-attention Transformer layers. The image encoder and text encoder are initialized from [CoCa](https://arxiv.org/abs/2205.01917), which is trained on WebLI following the CoCa recipes. VideoPrism is based on the [ViViT](https://arxiv.org/abs/2103.15691) factorized video encoder architecture.
 ### Inputs and outputs
 The models take videos with shape (num_frames, 288, 288, 3) as inputs and outputs embeddings with shape (num_frames * 16 * 16, feature_channels) which could be reshaped into (num_frames, 16, 16, feature_channels) for spatiotemporal representations. During model training, num_frames is set to 16 and 8 for VideoPrism-B and VideoPrism-L, respectively. Both models are expected to work with arbitrary num_frames by interpolating the temporal positional embeddings.
 ## Uses
 VideoPrism has a wide range of applications across various video understanding scenarios. The following lists some primary use cases and yet is not comprehensive. The purpose of this list is to provide contextual information the model creators considered as part of model training and development.
 *    **Video classification**: By feeding the video embeddings to a lightweight classifier, we can tackle video action recognition, a fundamental task in video understanding, under various scenarios.
-*    **Temporal and spatiotemporal localization**: We can also use the model to localize actions of interest spatially across time by equipping it with a bounding box proposal.
 *    **Video retrieval and open-set classification**: By pairing up the video embeddings with a text encoder in the CLIP fashion, we can do text-video retrieval and open-set video classification.
-## Ethical Considerations and Risks
 The model inherits the safety benefits and safety risks associated with the image encoder CoCa and the training datasets described above. We recommend that the model should not be used for downstream applications without prior assessment and mitigation of downstream application-specific security and fairness concerns.
-*    Data Bias: Large datasets scraped from the internet can contain inherent biases, leading to skewed model performance and potentially discriminatory outputs. The presence of "noisy parallel text" like ASR transcripts introduces potential inaccuracies and biases from the speech-to-text process.
-*    Content Moderation: The sheer volume of data (36M video-caption pairs and 582M video clips) raises concerns about the presence of objectionable or inappropriate content within the training data, which could lead to harmful model outputs.
-*    Ethical Use: As with any powerful video understanding model, there are risks of misuse, such as in surveillance or the propagation of misinformation.
-*    Limitations: The reliance on potentially noisy text data can limit the models understanding of the true video content. Further research is needed to refine the models ability to understand long form videos, geometric information in videos, and non-semantic cues.
-## How to Get Started with the Model
-Use the code at our Github to get started with the model: https://github.com/google-deepmind/videoprism.
-## Training Details
-### Training Data
 VideoPrism is pre-trained on a wide range of videos (36M video-caption pairs and 582M video clips), including the datasets below. Note that the number of clips are subject to change due to wipeout according to policy.
 | Pretraining datasets | Public | Domain | Caption source | Caption quality | # of videos | # of clips |
@@ -67,7 +58,6 @@ VideoPrism is pre-trained on a wide range of videos (36M video-caption pairs and
 ## Evaluation
 ### Results on video-focused tasks with frozen backbones
 | Dataset | K400 | MiT | SSv2 | D48 | Charades | ActivityNet | AVA | AVA-K |
 | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
 | VideoPrism-B (public) | 82.9 | 39.7 | 62.2 | 64.3 | 43.5 | 36.5 | 28.3 | 30.8 |
@@ -80,27 +70,19 @@ VideoPrism is pre-trained on a wide range of videos (36M video-caption pairs and
 "Public" denotes models we released in this repository. "Paper" and "Prior SOTA" denote our models and previous best-performing models reported in the paper, respectively. Our public models perform slightly worse than the paper models due to different pre-training image-text data we used subject to data policy.
-## Implementation Information
-Details about the model internals.
-### Model Architecture
 Vision model is a [ViViT](https://arxiv.org/abs/2103.15691) factorized video encoder architecture, initialized from the Vision Transformer image encoder ([CoCa](https://arxiv.org/abs/2205.01917)) followed by four temporal-attention Transformer layers.
 ### Hardware
 VideoPrism was trained using [Tensor Processing Unit
 (TPU)](https://cloud.google.com/tpu/docs/intro-to-tpu) hardware.
 #### Software
 JAX, Flax
 ## Citation
 VideoPrism:
 ```
 @inproceedings{zhao2024videoprism,

 ---
 # VideoPrism Model Card
+**Paper**: https://huggingface.co/papers/2402.13217
+**arXiv**: https://arxiv.org/pdf/2402.13217
+**GitHub**: https://github.com/google-deepmind/videoprism
+**Blog**: https://research.google/blog/videoprism-a-foundational-visual-encoder-for-video-understanding/
 VideoPrism is a foundational video encoder that enables state-of-the-art performance on a large variety of video understanding tasks. It takes video frames as input and outputs compact embeddings of the frames, which one can conveniently feed into classifiers, LLMs, retrieval models, etc. When tested on 33 public video understanding benchmarks over four task categories, a single frozen VideoPrism checkpoint outperforms previous best-performing foundation models on 31 of them, with no fine-tuning on target task datasets.
+## Model details
+### Model description
 VideoPrism-B/L are the composition of a Vision Transformer image encoder and four temporal-attention Transformer layers. The image encoder and text encoder are initialized from [CoCa](https://arxiv.org/abs/2205.01917), which is trained on WebLI following the CoCa recipes. VideoPrism is based on the [ViViT](https://arxiv.org/abs/2103.15691) factorized video encoder architecture.
 ### Inputs and outputs
 The models take videos with shape (num_frames, 288, 288, 3) as inputs and outputs embeddings with shape (num_frames * 16 * 16, feature_channels) which could be reshaped into (num_frames, 16, 16, feature_channels) for spatiotemporal representations. During model training, num_frames is set to 16 and 8 for VideoPrism-B and VideoPrism-L, respectively. Both models are expected to work with arbitrary num_frames by interpolating the temporal positional embeddings.
 ## Uses
 VideoPrism has a wide range of applications across various video understanding scenarios. The following lists some primary use cases and yet is not comprehensive. The purpose of this list is to provide contextual information the model creators considered as part of model training and development.
 *    **Video classification**: By feeding the video embeddings to a lightweight classifier, we can tackle video action recognition, a fundamental task in video understanding, under various scenarios.
+*    **Temporal and spatiotemporal localization**: We can also use the model to localize actions of interest spatially across time by equipping it with a bounding box proposal.
 *    **Video retrieval and open-set classification**: By pairing up the video embeddings with a text encoder in the CLIP fashion, we can do text-video retrieval and open-set video classification.
+## Ethical considerations and risks
 The model inherits the safety benefits and safety risks associated with the image encoder CoCa and the training datasets described above. We recommend that the model should not be used for downstream applications without prior assessment and mitigation of downstream application-specific security and fairness concerns.
+*    Data bias: Large datasets scraped from the internet can contain inherent biases, leading to skewed model performance and potentially discriminatory outputs. The presence of "noisy parallel text" like ASR transcripts introduces potential inaccuracies and biases from the speech-to-text process.
+*    Content moderation: The sheer volume of data (36M video-caption pairs and 582M video clips) raises concerns about the presence of objectionable or inappropriate content within the training data, which could lead to harmful model outputs.
+*    Ethical use: As with any powerful video understanding model, there are risks of misuse, such as in surveillance or the propagation of misinformation.
+*    Limitations: The reliance on potentially noisy text data can limit the models understanding of the true video content. Further research is needed to refine the models ability to understand long form videos, geometric information in videos, and non-semantic cues.
+## How to get started with the model
+Use the code at our GitHub to get started with the model: https://github.com/google-deepmind/videoprism.
+## Training details
+### Training data
 VideoPrism is pre-trained on a wide range of videos (36M video-caption pairs and 582M video clips), including the datasets below. Note that the number of clips are subject to change due to wipeout according to policy.
 | Pretraining datasets | Public | Domain | Caption source | Caption quality | # of videos | # of clips |
 ## Evaluation
 ### Results on video-focused tasks with frozen backbones
 | Dataset | K400 | MiT | SSv2 | D48 | Charades | ActivityNet | AVA | AVA-K |
 | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |
 | VideoPrism-B (public) | 82.9 | 39.7 | 62.2 | 64.3 | 43.5 | 36.5 | 28.3 | 30.8 |
 "Public" denotes models we released in this repository. "Paper" and "Prior SOTA" denote our models and previous best-performing models reported in the paper, respectively. Our public models perform slightly worse than the paper models due to different pre-training image-text data we used subject to data policy.
+## Implementation information
+### Model architecture
 Vision model is a [ViViT](https://arxiv.org/abs/2103.15691) factorized video encoder architecture, initialized from the Vision Transformer image encoder ([CoCa](https://arxiv.org/abs/2205.01917)) followed by four temporal-attention Transformer layers.
 ### Hardware
 VideoPrism was trained using [Tensor Processing Unit
 (TPU)](https://cloud.google.com/tpu/docs/intro-to-tpu) hardware.
 #### Software
 JAX, Flax
 ## Citation
 VideoPrism:
 ```
 @inproceedings{zhao2024videoprism,