Update readme
#5
by anmolg-nvidia - opened
README.md
CHANGED
|
@@ -340,6 +340,8 @@ For BF16, use `vllm==0.25.0` for `/v2/embed` serving. NVIDIA also validated `vll
|
|
| 340 |
pip install --upgrade "vllm==0.25.0" openai requests numpy
|
| 341 |
```
|
| 342 |
|
|
|
|
|
|
|
| 343 |
### vLLM Offline Python
|
| 344 |
|
| 345 |
Use the offline Python API when you want local vLLM inference without running an HTTP server. `LLM.embed` accepts formatted strings, so add the `query: ` and `passage: ` prefixes manually.
|
|
|
|
| 340 |
pip install --upgrade "vllm==0.25.0" openai requests numpy
|
| 341 |
```
|
| 342 |
|
| 343 |
+
**FP8 acceleration:** On NVIDIA Hopper and Ada Lovelace GPUs, enable FP8 in vLLM online with `--quantization fp8_per_tensor`, or offline with `quantization="fp8_per_tensor"` in `LLM(...)`. Validated with vLLM `0.25.0` on H100; accuracy matched BF16.
|
| 344 |
+
|
| 345 |
### vLLM Offline Python
|
| 346 |
|
| 347 |
Use the offline Python API when you want local vLLM inference without running an HTTP server. `LLM.embed` accepts formatted strings, so add the `query: ` and `passage: ` prefixes manually.
|