Add new model

README.md

@@ -1,3 +1,112 @@
 ---
+language: en
+tags:
+- bridgetower
+- gaudi
 license: mit
+datasets:
+- conceptual_captions
+- conceptual_12m
+- sbu_captions
+- visual_genome
+- mscoco_captions
 ---
+
+# BridgeTower large-itm-mlm-gaudi model
+
+The BridgeTower model was proposed in "BridgeTower: Building Bridges Between Encoders in Vision-Language Representative Learning" by Xiao Xu, Chenfei Wu, Shachar Rosenman, Vasudev Lal, Wanxiang Che, Nan Duan. 
+The model was pretrained on English language using masked language modeling (MLM) and image text matching (ITM)objectives. It was introduced in
+[this paper](https://arxiv.org/pdf/2206.08657.pdf) and first released in
+[this repository](https://github.com/microsoft/BridgeTower). 
+
+BridgeTower got accepted to [AAAI'23](https://aaai.org/Conferences/AAAI-23/).
+
+## Model description
+
+The abstract from the paper is the following:
+Vision-Language (VL) models with the Two-Tower architecture have dominated visual-language representation learning in recent years. Current VL models either use lightweight uni-modal encoders and learn to extract, align and fuse both modalities simultaneously in a deep cross-modal encoder, or feed the last-layer uni-modal representations from the deep pre-trained uni-modal encoders into the top cross-modal encoder. Both approaches potentially restrict vision-language representation learning and limit model performance. In this paper, we propose BridgeTower, which introduces multiple bridge layers that build a connection between the top layers of uni-modal encoders and each layer of the cross-modal encoder. This enables effective bottom-up cross-modal alignment and fusion between visual and textual representations of different semantic levels of pre-trained uni-modal encoders in the cross-modal encoder. Pre-trained with only 4M images, BridgeTower achieves state-of-the-art performance on various downstream vision-language tasks. In particular, on the VQAv2 test-std set, BridgeTower achieves an accuracy of 78.73%, outperforming the previous state-of-the-art model METER by 1.09% with the same pre-training data and almost negligible additional parameters and computational costs. Notably, when further scaling the model, BridgeTower achieves an accuracy of 81.15%, surpassing models that are pre-trained on orders-of-magnitude larger datasets.
+
+## Intended uses & limitations
+
+
+### How to use
+
+Here is how to use this model to perform image and text matching:
+
+```python
+from transformers import BridgeTowerProcessor, BridgeTowerForImageAndTextRetrieval
+import requests
+from PIL import Image
+
+url = "http://images.cocodataset.org/val2017/000000039769.jpg"
+image = Image.open(requests.get(url, stream=True).raw)
+texts = ["An image of two cats chilling on a couch", "A football player scoring a goal"]
+
+processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-gaudi")
+model = BridgeTowerForImageAndTextRetrieval.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-gaudi")
+
+# forward pass
+scores = dict()
+for text in texts:
+    # prepare inputs
+    encoding = processor(image, text, return_tensors="pt")
+    outputs = model(**encoding)
+    scores[text] = outputs.logits[0,1].item()
+```
+
+Here is how to use this model to perform masked language modeling:
+
+```python
+from transformers import BridgeTowerProcessor, BridgeTowerForMaskedLM
+from PIL import Image
+import requests
+
+url = "http://images.cocodataset.org/val2017/000000360943.jpg"
+image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
+text = "a <mask> looking out of the window"
+
+processor = BridgeTowerProcessor.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-gaudi")
+model = BridgeTowerForMaskedLM.from_pretrained("BridgeTower/bridgetower-large-itm-mlm-gaudi")
+
+# prepare inputs
+encoding = processor(image, text, return_tensors="pt")
+
+# forward pass
+outputs = model(**encoding)
+
+results = processor.decode(outputs.logits.argmax(dim=-1).squeeze(0).tolist())
+
+print(results)
+#.a cat looking out of the window.
+```
+
+## Training data
+
+The BridgeTower model was pretrained on four public image-caption datasets:
+- [Conceptual Captions (CC3M)](https://ai.google.com/research/ConceptualCaptions/)
+- [Conceptual 12M (CC12M)](https://github.com/google-research-datasets/conceptual-12m)
+- [SBU Captions](https://www.cs.rice.edu/~vo9/sbucaptions/)
+- [MSCOCO Captions](https://arxiv.org/pdf/1504.00325.pdf)
+- [Visual Genome](https://visualgenome.org/)
+  
+The total number of unique images in the combined data is around 16M. 
+
+## Training procedure
+
+### Pretraining
+
+The model was pre-trained for 10 epochs on an Intel AI supercomputing cluster using 512 Gaudis and 128 Xeons with a batch size of 2048. 
+The optimizer used was AdamW with a learning rate of 1e-7. No data augmentation was used except for center-crop. The image resolution in pre-training is set to 294 x 294. 
+
+## Evaluation results
+Please refer to [Table 5](https://arxiv.org/pdf/2206.08657.pdf) for BridgeTower's performance on Image Retrieval and other downstream tasks. 
+
+### BibTeX entry and citation info
+```bibtex
+@article{xu2022bridge,
+  title={BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learning},
+  author={Xu, Xiao and Wu, Chenfei and Rosenman, Shachar and Lal, Vasudev and Che, Wanxiang and Duan, Nan},
+  journal={arXiv preprint arXiv:2206.08657},
+  year={2022}
+}
+```

config.json

@@ -0,0 +1,56 @@
+{
+  "contrastive_hidden_size": 512,
+  "share_cross_modal_transformer_layers": true,
+  "drop_rate": 0.1,
+  "head_hidden_scale": 2,
+  "hidden_act": "gelu",
+  "hidden_size": 1024,
+  "initializer_factor": 1,
+  "is_encoder_decoder": false,
+  "layer_norm_eps": 1e-05,
+  "share_link_tower_layers": false,
+  "link_tower_type": "add",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 6,
+  "tie_word_embeddings": false,
+  "text_config_dict": null,
+  "init_layernorm_from_vision_encoder": false,
+  "text_config": {
+    "architectures": [
+      "BridgeTowerTextModel"
+    ],
+    "vocab_size": 50265,
+    "hidden_size": 1024,
+    "num_hidden_layers": 24,
+    "num_attention_heads": 16,
+    "intermediate_size": 4096,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "attention_probs_dropout_prob": 0.1,
+    "max_position_embeddings": 514,
+    "type_vocab_size": 1,
+    "initializer_factor": 1,
+    "initializer_range": 0.02,
+    "layer_norm_eps": 1e-05,
+    "pad_token_id": 1,
+    "bos_token_id": 0,
+    "eos_token_id": 2,
+    "position_embedding_type": "absolute",
+    "use_cache": true,
+    "classifier_dropout": null
+  },
+  "vision_config_dict": null,
+  "vision_config": {
+    "architectures": [
+      "BridgeTowerVisionModel"
+    ],
+    "hidden_size": 1024,
+    "num_hidden_layers": 24,
+    "patch_size": 14,
+    "image_size": 294,
+    "initializer_factor": 1,
+    "stop_gradient": false,
+    "share_layernorm": true,
+    "remove_last_layer": false
+  }
+}

preprocessor_config.json

@@ -0,0 +1,6 @@
+{
+    "max_text_len":50,
+    "size":294,
+    "tokenizer":"roberta-large",
+    "vocab_size":50265
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c29278f0447e2d3139382eb658be43f297b1ec492b298ef7a0ee65863e88e743
+size 3677121923