T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — mentioned by compilade

Thu, 17 Apr 2025 14:34:57 GMT

Because lookup tables are not necessarily faster compared to 8-bit SIMD operations, at least when implemented naïvely. Lookup tables can be fast, but it's not simpler, see T-MAC https://arxiv.org/abs/2407.00088 (Note that all comparisons with `llama.cpp` were made before I introduced the types from https://github.com/ggml-org/llama.cpp/pull/8151 where the 1.6-bit type uses the techniques described in the aforementioned blog post). I wanted to try without lookup tables to at least have a baseline, and also because the fixed point packing idea lent itself naturally to using multiplications by…

Paper and discussion history. A mention is not an endorsement.

T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — HN submission

Sat, 17 Aug 2024 11:47:30 GMT

T-Mac: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge · HackerPapers discussions

T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — mentioned by compilade

T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge — HN submission