Instructions to use simpledirect/Vinci-MLE-8B-1.0-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use simpledirect/Vinci-MLE-8B-1.0-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M # Run inference directly in the terminal: llama cli -hf simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
Use Docker
docker model run hf.co/simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use simpledirect/Vinci-MLE-8B-1.0-GGUF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "simpledirect/Vinci-MLE-8B-1.0-GGUF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "simpledirect/Vinci-MLE-8B-1.0-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
- Ollama
How to use simpledirect/Vinci-MLE-8B-1.0-GGUF with Ollama:
ollama run hf.co/simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
- Unsloth Desktop
- Pi
How to use simpledirect/Vinci-MLE-8B-1.0-GGUF with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use simpledirect/Vinci-MLE-8B-1.0-GGUF with Docker Model Runner:
docker model run hf.co/simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
- Lemonade
How to use simpledirect/Vinci-MLE-8B-1.0-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
Run and chat with the model
lemonade run user.Vinci-MLE-8B-1.0-GGUF-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use simpledirect/Vinci-MLE-8B-1.0-GGUF with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use simpledirect/Vinci-MLE-8B-1.0-GGUF with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "simpledirect/Vinci-MLE-8B-1.0-GGUF:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Vinci-MLE-8B-1.0-GGUF
Open-weight ML-engineering research in GGUF: evidence-based diagnosis, bounded changes, and justified no-change or stop decisions.
Canadian-developed by SimpleDirect Β· Apache-2.0 Β· Research baseline
Available now: F16, Q8_0, Q5_K_M, and Q4_K_M. Each variant has recorded generation and tool-call-format smoke checks; see runtime validation for their scope.
This is the GGUF distribution of Vinci MLE 8B 1.0, an open-weight research model trained around evidence-based diagnosis, bounded workspace changes, and justified no-change or stop decisions. Its source model is a fine-tuned derivative of IBM Granite 4.1 8B.
Model at a glance
| Property | Value |
|---|---|
| Developer | SimpleDirect, Canada |
| Source model | simpledirect/Vinci-MLE-8B-1.0 |
| Original parent | ibm-granite/granite-4.1-8b |
| Distribution format | GGUF |
| Available model files | F16, Q8_0, Q5_K_M, Q4_K_M |
| Conversion runtime | llama.cpp, revision 6f4f53f2 |
| Validation scope | CUDA full-offload runtime smoke; no minimum-memory certification |
| License | Apache-2.0, with NOTICE |
ML engineering specialization
The source model was trained on 91 trajectories across 64 task roots in eight ML-engineering families: data duplication and quality; experiment design and interpretation; failed resume; memory, batching and gradient accumulation; metric/evaluator defects; NaNs and divergence; packaging, serving and reproducibility; and tokenizer/model configuration.
| Trained response | Intended behavior |
|---|---|
| Act | Make a bounded change when evidence supports a defect and intervention |
| No Change | Preserve a correct setup and explain the evidence |
| Stop / Needs Evidence | Identify missing evidence or a blocker instead of inventing a repair |
These describe the source model's training scope, not measured success rates. Both source-model sizes consumed the same tokenized examples but used different training recipes. See the source card for the 8B-specific recipe, selective-supervision design, and development-selection limitations.
Files and choosing a variant
| File | Bytes | GB | GiB |
|---|---|---|---|
| Q4_K_M | 5,347,914,336 | 5.35 | 4.98 |
| Q5_K_M | 6,253,884,000 | 6.25 | 5.82 |
| Q8_0 | 9,345,610,336 | 9.35 | 8.70 |
| F16 | 17,587,417,696 | 17.59 | 16.38 |
Sizes were read from the Hugging Face API on September 28, 2026 and matched to GGUF-RUNTIME.json. File size is not total RAM or VRAM use: allow additional memory for context, the KV cache, and runtime overhead.
Q4_K_M is the smallest download in this set; Q5_K_M and Q8_0 offer progressively larger representations. F16 is the floating-point GGUF conversion. Choose according to available memory and your own task validation; no quality ranking between these variants has been established here. The BF16 conversion master mentioned in the runtime record is an intermediate artifact, not an additional downloadable variant.
Quick start
Install huggingface_hub and transformers, and authenticate with hf auth login if repository access requires it. Use a working CUDA build of llama.cpp at the tested revision with llama-completion on your PATH and enough GPU memory for full offload.
The example downloads Q4_K_M at a fixed repository revision, checks its SHA-256, renders one of the smoke prompts using the source tokenizer's chat template, and invokes the same completion interface and decoding settings used in the recorded checks. It downloads only the selected GGUF variant; source tokenizer loading does not download the BF16 weight shards.
import hashlib
import subprocess
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
model_path = hf_hub_download(
repo_id="simpledirect/Vinci-MLE-8B-1.0-GGUF",
filename="Vinci-MLE-8B-1.0-Q4_K_M.gguf",
revision="4c8abbe3e2c45cae749d7b9a865ca86c0bdc98b4",
local_dir="./vinci-mle-8b-gguf",
)
with open(model_path, "rb") as model_file:
actual_sha256 = hashlib.file_digest(model_file, "sha256").hexdigest()
if actual_sha256 != "87313e7e349fa9e59a49cf73ec1e41b5764c1e3b5642ac0543229df460efc798":
raise RuntimeError("GGUF checksum mismatch; download the pinned file again.")
tokenizer = AutoTokenizer.from_pretrained(
"simpledirect/Vinci-MLE-8B-1.0",
revision="75256cf293770b6439b5ca52b98d43aa7687294d",
trust_remote_code=False,
)
prompt = tokenizer.apply_chat_template(
[{"role": "user", "content": "Explain in two sentences what a learning-rate warmup does."}],
tokenize=False,
add_generation_prompt=True,
)
subprocess.run(
[
"llama-completion", "-m", model_path, "-p", prompt,
"-n", "48", "--temp", "0", "--top-k", "1", "--seed", "42",
"-no-cnv", "--no-display-prompt", "-ngl", "99", "--no-mmap",
],
check=True,
)
Use Python 3.11 or later for hashlib.file_digest. This is a reference integration of the recorded smoke procedure, not a new run on your hardware. The example requests full GPU offload; CPU-only use, partial offload, other backends, and graphical front ends were not validated by these checks. The source model's Transformers/vLLM weight-loading instructions apply to the BF16 repository, not these GGUF files.
For another variant, change the filename and expected hash together using SHA256SUMS. When verifying a complete snapshot, run sha256sum -c SHA256SUMS (macOS: shasum -a 256 -c SHA256SUMS) inside its directory. The single-file example above verifies only its selected GGUF.
Runtime validation
The uploaded runtime record reports the following for each distributed variant:
| Variant | Nonempty generation outputs | Nonzero generation exit codes | Detected NaN/error log lines | Tool-call format |
|---|---|---|---|---|
| Q4_K_M | 6/6 | 0 | 0 | Parsed read_file call |
| Q5_K_M | 6/6 | 0 | 0 | Parsed read_file call |
| Q8_0 | 6/6 | 0 | 0 | Parsed read_file call |
| F16 | 6/6 | 0 | 0 | Parsed read_file call |
The checks used six fixed prompts with a 48-token generation budget, plus one tool-format prompt with a 96-token budget per variant. Decoding was greedy (temperature 0, top-k 1, seed 42), with CUDA full offload and memory mapping disabled. Prompts were rendered with the staged source tokenizer; embedded chat templates matched the staged template in every distributed file.
These are runtime and format checks, not capability evaluations. The tool check parsed a JSON call and checked the name read_file; it did not validate the arguments, execute the tool, or establish reliable agent behavior. The short generation checks do not measure answer correctness, throughput, minimum memory, or long-context capability.
Additional conversion checks
The first eight re-tokenized output tokens agreed between Q8_0 and F16 on all six generation prompts. This narrow agreement does not establish complete-output equivalence or quality retention against BF16.
An independent 8B conversion produced byte-identical files for the distributed variants and the intermediate BF16 master. This checks conversion reproducibility, not deterministic generation across runtimes.
Evaluation and limitations
Vinci MLE 1.0 has no valid held-out ML-engineering task score. The source card reports broad parent-capability comparisons separately from MLE training scope, including unfavorable results and methodology limitations.
GGUF runtime smoke checks are reported above. Task-level capability retention against the BF16 source has not been measured for these variants. The BF16 cardβs benchmark scores are not GGUF measurements; no benchmark parity or MLE-task improvement is claimed.
The source model's evaluation scope, limitations, and contamination audit remain relevant. Applications must implement and validate their own tools, sandbox, permissions, and workspace; model weights alone are not a complete Vinci MLE agent harness.
Source and conversion provenance
| Record | Value |
|---|---|
| BF16 source artifact reference | 75256cf293770b6439b5ca52b98d43aa7687294d |
| GGUF artifact snapshot | 4c8abbe3e2c45cae749d7b9a865ca86c0bdc98b4 |
| Conversion tool | llama.cpp 6f4f53f2b7da54fcdbbecaaa734337c337ad6176 |
| F16 conversion | Direct conversion from the staged BF16 Safetensors with convert_hf_to_gguf.py --outtype f16 |
| Quantization input | BF16 GGUF master converted from the staged Safetensors |
| Quantization variants | llama-quantize with Q8_0, Q5_K_M, or Q4_K_M; no importance matrix supplied |
| Artifact checksums | SHA256SUMS |
| Runtime checks | GGUF-RUNTIME.json |
| Task-level comparison with BF16 | Not measured |
The conversion workflow checked the staged source files against the release artifact pins before conversion. The BF16 source reference identifies those model artifacts; the GGUF snapshot pins the uploaded conversions and their original runtime record. Later card-only changes do not alter those files. Source-model provenance is available in the BF16 repositoryβs PROVENANCE.json, also referenced by the accompanying source NOTICE.
License and attribution
The source model is licensed under Apache-2.0; its LICENSE and NOTICE are included here. IBM developed Granite; SimpleDirect developed the Vinci MLE fine-tune. No affiliation with or endorsement by IBM is implied.
- Downloads last month
- 110
4-bit
5-bit
8-bit
16-bit
Model tree for simpledirect/Vinci-MLE-8B-1.0-GGUF
Base model
ibm-granite/granite-4.1-8b