Instructions to use llava-hf/llava-1.5-7b-hf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use llava-hf/llava-1.5-7b-hf with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="llava-hf/llava-1.5-7b-hf")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("llava-hf/llava-1.5-7b-hf")
model = AutoModelForMultimodalLM.from_pretrained("llava-hf/llava-1.5-7b-hf", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use llava-hf/llava-1.5-7b-hf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "llava-hf/llava-1.5-7b-hf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "llava-hf/llava-1.5-7b-hf",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/llava-hf/llava-1.5-7b-hf

SGLang

How to use llava-hf/llava-1.5-7b-hf with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "llava-hf/llava-1.5-7b-hf" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "llava-hf/llava-1.5-7b-hf",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "llava-hf/llava-1.5-7b-hf" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "llava-hf/llava-1.5-7b-hf",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use llava-hf/llava-1.5-7b-hf with Docker Model Runner:
```
docker model run hf.co/llava-hf/llava-1.5-7b-hf
```

Error in Colab

by sudhir2016 - opened Dec 9, 2023

Discussion

sudhir2016

Dec 9, 2023

Getting this error in Colab.
--> 761 raise KeyError(key)
762 value = self._mapping[key]
763 module_name = model_type_to_module_name(key)

KeyError: 'llava'

nielsr

Llava Hugging Face org Dec 9, 2023

•

edited Dec 9, 2023

Hi,

Make sure to have Transformers installed from the main branch: pip install git+https://github.com/huggingface/transformers@main.

sudhir2016

Dec 9, 2023

Tried it. Now it gives this error.
PyTorch SDPA requirements in Transformers are not met. Please install torch>=2.1.1.

Error persists even after installing torch >=2-1.1

ArthurZ

Llava Hugging Face org Dec 9, 2023

Okay I'll have a look this is not expected

ArthurZ

Llava Hugging Face org Dec 9, 2023

•

edited Dec 9, 2023

:hug:

ArthurZ

Llava Hugging Face org Dec 9, 2023

I could not reproduce on main:


In [1]: from transformers import pipeline
   ...: from PIL import Image
   ...: import requests
   ...: 
   ...: 
   ...: model_id = "llava-hf/bakLlava-v1-hf"
   ...: pipe = pipeline("image-to-text", model=model_id)
   ...: url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"
   ...: 
   ...: image = Image.open(requests.get(url, stream=True).raw)
   ...: prompt = "USER: <image>\nWhat does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud\nASSISTANT:"
   ...: 
   ...: outputs = pipe(image, prompt=prompt, generate_kwargs={"max_new_tokens": 200})
   ...: print(outputs)
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00,  2.29it/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[{'generated_text': 'USER:  \nWhat does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud\nASSISTANT: Lava'}]

In [2]: import torch

In [3]: torch.__version__
Out[3]: '1.13.1'

sudhir2016

Dec 10, 2023

•

edited Dec 10, 2023

Thanks. I was trying the llava-hf/llava-1.5-7b-hf model and not llava-hf/bakLlava-v1-hf. I was trying the 4 bit quantized version as per the example Colab provided on the model page. Issue remains.

nielsr

Llava Hugging Face org Dec 10, 2023

If you have that issue, it means that you don't have the main version of Transformers installed in your environment (which you can verify by doing pip show transformers). Make sure to have v4.36.dev

ybelkada

Llava Hugging Face org Dec 10, 2023

@sudhir2016 can you try to upgrade torch? pip install -U torch

sudhir2016

Dec 11, 2023

Many thanks to the incredible support provided by the awesome Hugging Face team !!. It works now. pip install -U torch did the trick.

sudhir2016 changed discussion status to closed Dec 11, 2023

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment