A new mixed-quantization version of the PLaMo 3 Translate 0.8B model has been released, reducing its weight size to approximately 423MB. This version, converted using mlx-lm 0.32.2, allocates 4-bit quantization to specific layers while using 3-bit for the rest, resulting in an average of 3.804 bits per weight. While the model's translation quality metric shows a slight decrease compared to the uniform 4-bit version, it maintains a decode speed of around 352 tokens/s and peak memory usage of about 565MB on a Mac mini M2 Pro. AI
IMPACT Enables running more capable translation models on resource-constrained devices.
RANK_REASON Release of a smaller, quantized version of an existing LLM with performance metrics. [lever_c_demoted from research: ic=1 ai=1.0]
AI-generated summary · Google Gemini · from 1 sources. How we write summaries →