Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches
Researchers have explored two methods for efficiently fine-tuning large language models for text classification tasks, particularly under resource constraints. The study compared attaching a classification head to a pre-trained causal LLM using its final-token embedding versus instruction-tuning the LLM in a prompt-to-response format. Experiments on patent and public datasets demonstrated that the embedding-based method often matched or surpassed the instruction-tuned approach for single-label classification, requiring significantly fewer trainable parameters. AI
IMPACT Presents efficient fine-tuning techniques for LLMs, potentially lowering the barrier for deploying these models in text classification tasks.