Instructions to use mistralai/Mistral-7B-v0.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use mistralai/Mistral-7B-v0.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="mistralai/Mistral-7B-v0.1")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")
model = AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", device_map="auto")

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use mistralai/Mistral-7B-v0.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Install mistral-common:
pip install --upgrade mistral-common
# Start the vLLM server:
vllm serve "mistralai/Mistral-7B-v0.1" --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral --enable-auto-tool-choice
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/mistralai/Mistral-7B-v0.1

SGLang

How to use mistralai/Mistral-7B-v0.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "mistralai/Mistral-7B-v0.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "mistralai/Mistral-7B-v0.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "mistralai/Mistral-7B-v0.1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use mistralai/Mistral-7B-v0.1 with Docker Model Runner:
```
docker model run hf.co/mistralai/Mistral-7B-v0.1
```

KeyError: 'mistral'

#14

by GbrlOl - opened Sep 28, 2023

Discussion

GbrlOl

Sep 28, 2023

KeyError Traceback (most recent call last)
in <cell line: 19>()
17 #device_map={"": 0}
18
---> 19 model = AutoModelForCausalLM.from_pretrained(
20 model_name,
21 quantization_config=bnb_config,

2 frames
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py in getitem(self, key)
732 return self._extra_content[key]
733 if key not in self._mapping:
--> 734 raise KeyError(key)
735 value = self._mapping[key]
736 module_name = model_type_to_module_name(key)

how do I fix this error?

jselvam

Sep 28, 2023

install transformers from source: pip install git+https://github.com/huggingface/transformers

CREET01

Sep 29, 2023

@jselvam This solution works.

atineoSE

Oct 2, 2023

•

This comment has been hidden

lerela

Oct 5, 2023

Closing as this is indeed the solution. Mistral is now part of Transformers 4.34.0 so pip install "transformers>=4.34.0" is enough.

lerela changed discussion status to closed Oct 5, 2023

samprita

Feb 22, 2024

•

edited Feb 22, 2024

still its not working for CPU(im using google colab CPU)
i tried all the above mentioned methods,still its giving
KeyError Traceback (most recent call last)
in <cell line: 3>()
1 tokenizer = AutoTokenizer.from_pretrained(model_name)
2
----> 3 model = AutoModelForCausalLM.from_pretrained(model_name,
4 device_map="auto",
5 #device_map = device_map,

2 frames
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py in getitem(self, key)
669 ("reformer", "Reformer"),
670 ("regnet", "RegNet"),
--> 671 ("rembert", "RemBERT"),
672 ("resnet", "ResNet"),
673 ("retribert", "RetriBERT"),

KeyError: 'mistral'
this error

songogeta31

Mar 21, 2024

File /opt/conda/envs/py39/lib/python3.9/ctypes/__init__.py:382, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    379 self._FuncPtr = _FuncPtr
    381 if handle is None:
--> 382     self._handle = _dlopen(self._name, mode)
    383 else:
    384     self._handle = handle

OSError: libiomp5.so: cannot open shared object file: No such file or directory

I am getting this error after solving transformers error

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment