THINKING LLMS: GENERAL INSTRUCTION FOLLOWING WITH THOUGHT GENERATION

This research paper proposes a novel method called Thought Preference Optimization (TPO) to train large language models (LLMs) to "think" before responding to user instructions. TPO utilizes a preference-based training framework where LLMs generate internal thoughts alongside their responses, and these thoughts are then optimized based on the quality of the resulting responses. The authors argue that this approach, unlike previous methods relying on direct supervision, allows LLMs to develop thinking abilities for a broader range of tasks beyond traditional reasoning and problem-solving…