Merge branch 'main' of https://huggingface.co/allenai/MolmoE-1B-0924 into main

README.md

@@ -2,35 +2,95 @@
 license: apache-2.0
 language:
 - en
-tags:
-- moe
-- olmo
-- olmoe
-- molmo
-- molmoe
-co2_eq_emissions: 1
+base_model:
+- openai/clip-vit-large-patch14-336
+- allenai/OLMoE-1B-7B-0924
 datasets:
 - allenai/OLMoE-mix-0924
-library_name: transformers
+pipeline_tag: image-text-to-text
+tags:
+  - multimodal
+  - moe
+  - olmo
+  - olmoe
+  - molmo
+  - molmoe
 ---
 
-<img alt="Molmo Logo." src="molmo_logo.png" width="250px">
+<img src="molmo_logo.png" alt="Logo for the Molmo Project" style="width: auto; height: 50px;">
+
+# MolmoE 1B
+
+Molmo is an open vision-language model developed by the Allen Institute for AI. Molmo models are trained on PixMo, a dataset of 1 million, highly-curated image-text pairs. It has state-of-the-art performance among multimodal models with a similar size while being fully open-source. You can find all models in the Molmo family [here](https://huggingface.co/collections/allenai/molmo-66f379e6fe3b8ef090a8ca19).
+
+MolmoE-1B is a multimodal Mixture-of-Experts LLM with 1.5B active and 7.2B total parameters released in September 2024 (0924) based on [OLMoE-1B-7B-0924](https://huggingface.co/allenai/OLMoE-1B-7B-0924). 
+It nearly matches the performance of GPT-4V on both academic benchmarks and human evaluation, and achieves state-of-the-art performance among similarly-sized open multimodal models.
+
+This checkpoint is a **preview** of the Molmo release. All artifacts used in creating Molmo (PixMo dataset, training code, evaluations, intermediate checkpoints) will be made available at a later date, furthering our commitment to open-source AI development and reproducibility.
+
+**[Sign up here](https://docs.google.com/forms/d/e/1FAIpQLSdML1MhNNBDsCHpgWG65Oydg2SjZzVasyqlP08nBrWjZp_c7A/viewform)** to be the first to know when artifacts are released.
+
+
+
+## Quick Start
+
+To run MolmoE, first install dependencies:
+
+```bash
+pip install einops tensorflow torchvision
+```
+
+Then, follow these steps:
+
+```python
+from transformers import AutoModelForCausalLM, AutoProcessor, GenerationConfig
+from PIL import Image
+import requests
+
+# load the processor
+processor = AutoProcessor.from_pretrained(
+    'allenai/MolmoE-1B-0924',
+    trust_remote_code=True,
+    torch_dtype='auto',
+    device_map='auto'
+)
 
-# Model Summary
+# load the model
+model = AutoModelForCausalLM.from_pretrained(
+    'allenai/MolmoE-1B-0924',
+    trust_remote_code=True,
+    torch_dtype='auto',
+    device_map='auto'
+)
 
-> MolmoE-1B is a multimodal Mixture-of-Experts LLM with 1.5B active and 7.2B total parameters released in September 2024 (0924) based on [OLMoE-1B-7B-0924](https://huggingface.co/allenai/OLMoE-1B-7B-0924). It yields state-of-the-art performance among multimodal models with a similar size while being fully open-source.
+# process the image and text
+inputs = processor.process(
+    images=[Image.open(requests.get("https://picsum.photos/id/237/536/354", stream=True).raw)],
+    text="Describe this image."
+)
 
-- **Paper:** WIP
-- **Code:** WIP
+# move inputs to the correct device and make a batch of size 1
+inputs = {k: v.to(model.device).unsqueeze(0) for k, v in inputs.items()}
 
-# Use
+# generate output; maximum 200 new tokens; stop generation when <|endoftext|> is generated
+output = model.generate_from_batch(
+    inputs,
+    GenerationConfig(max_new_tokens=200, stop_strings="<|endoftext|>"),
+    tokenizer=processor.tokenizer
+)
 
-WIP
+# only get generated tokens; decode them to text
+generated_tokens = output[0,inputs['input_ids'].size(1):]
+generated_text = processor.tokenizer.decode(generated_tokens, skip_special_tokens=True)
 
-# Evaluation Snapshot
+# print the generated text
+print(generated_text)
 
-WIP
+# >>> This photograph captures an adorable black Labrador puppy sitting on a weathered
+#     wooden deck. The deck's planks, which are a mix of light and dark brown with ...
+```
 
-# Citation
+## License and Use
 
-WIP
+This model is licensed under Apache 2.0. It is intended for research and educational use.
+For more information, please see our [Responsible Use Guidelines](https://allenai.org/responsible-use).