Amazing !
#1
by Elsephire - opened
Very nice work! I started to work for compress in GGUF and make this work in my llama.cpp fork!
Thank you for this!
Small update: I've implemented GGUF conversion and model loading, but the model output is only ////.
I'm working on it , I still have a performance issue to resolve.
I'll be publishing the GGUF file and a link to the source code on GitHub of my llama.cpp fork.
Your work is very important for the accessibility of local LLMs. Don't let it drop , it's a great improvement!
Is the repo public? I'd love to take a look and see if I can help