Direct Preference Optimization Beyond Chatbots

Hugging Face published methodology and benchmark results for DharmaOCR, a structured OCR model evaluated on Brazilian Portuguese text extraction tasks. Across tested open-source vision-language model families, vanilla text degeneration rates varied widely, and supervised fine-tuning rarely reduced those rates to acceptable production levels. SFT optimizes for correct outputs token by token without explicitly penalizing completion-level repetition loops, exposing a structural ceiling on task-focused fine-tuning. To address this failure mode, the DharmaOCR pipeline introduced a second training stage using Direct Preference Optimization. While DPO is typically applied to chat alignment using human preference judgments, the OCR pipeline leveraged objective criteria where correct transcriptions were chosen and degeneration loops were rejected. Rather than filtering out degenerate outputs as noise, the training methodology deliberately utilized the SFT model's own degenerate outputs as the negative examples in preference pairs. Across all tested model families, this DPO stage reduced text degeneration by an average of 59.4 percent, with a peak reduction of 87.6 percent. The approach illustrates that DPO can serve as a direct mitigation tool for structural failure modes in non-conversational, objective tasks by reshaping the distribution space and steering the model away from self-reinforcing repetition loops.