Skip to content
GKgkml.dev
← Change mode

The fine-tune that learned the voice and lost the knowledge

RCA drill
LLM internalsMLOpshard

You fine-tune a 7B base model on 4,000 internal support transcripts so it answers in your company's house style. Three epochs, learning rate 1e-4, full fine-tune, batch size 8, no scheduler.

The result writes in exactly the right voice. It also now claims your product supports features it does not, cannot do arithmetic it could do before, and answers general knowledge questions confidently and wrongly. Training loss fell smoothly throughout and validation loss on a held-out slice of the transcripts is at its lowest.

A colleague suggests training for more epochs on more transcripts.

Name the failure mode precisely, explain the mechanism by which this specific configuration produced it, and tell me why the validation loss is not just uninformative but actively misleading here.

Enter sends · Shift+Enter for a new line