t pairs from existing datasets. A pipeline was constructed to extract and link text spans (like noun phrases and referring expressions) in the caption to the spatial locations (like bounding boxes) of its corresponding objects or regions in the image.

The model was trained to establish a mapping between image regions and their corresponding location tokens and connect the image regions with their associated text spans. This allows KOSMOS-2 to not only perceive general modalities, follow instructions, and p