Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I don't have a fancy GPU :(

I do however have a dual xeon with 64GB of ram. Will that work for this?



What is "dual xeon", two xeon CPUs or dual core? On my laptop with an i7 I had some decent performance with it. Don't know token/s , but it took less than 20s to generate an answer.

Should note that I used q5 quantisation with llama.cpp: https://huggingface.co/TheBloke/Mistral-7B-Instruct-v0.1-GGU...


If you're patient, yes.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: