A biology lab I worked with needed automated cell counting from fluorescence microscopy images — a task that sounds solved, because public benchmarks for exactly this exist and pretrained models score well on them. We fine-tuned one of those pretrained models on a small set of the lab's own labeled images, expecting a quick win. Performance on held-out images from the same batch was excellent. Performance on images from a different imaging session, same lab, same protocol, same person operating the microscope, was noticeably worse. Nothing about the biology had changed. Something about the images had.

The domain shift was in the hardware, not the cells

It took longer than I'd like to admit to trace it back to illumination variance between imaging sessions — small differences in how the microscope's light source was calibrated session to session, invisible to a human looking at the images side by side, but apparently very visible to a model that had learned to key off absolute intensity values rather than relative structure. Once we added illumination-normalization as a preprocessing step and heavier brightness/contrast augmentation during training, the gap between sessions mostly closed.

This is the part that doesn't show up in a benchmark leaderboard: a model can be extremely good at the task and still fail in deployment for a reason that's entirely about your specific acquisition pipeline, not about the model's capacity or the difficulty of the underlying task. Public dataset performance tells you the model can learn the task. It tells you very little about whether it will generalize across the specific, boring hardware variance your lab will actually produce over time.

The fix wasn't a better model. It was noticing that "same lab, different day" was a distribution shift we hadn't accounted for at all.

What actually helped, in order of impact

  • Illumination normalization as a fixed preprocessing step, applied identically at training and inference time — this mattered more than any architecture choice.
  • Aggressive brightness and contrast augmentation during training, specifically simulating the kind of session-to-session variance we'd measured, not generic augmentation presets.
  • Collecting a small validation set deliberately spread across multiple imaging sessions, instead of a random split from a single session — the single-session split had been quietly hiding the whole problem.

If you're deploying a vision model into any lab or field setting with real hardware in the loop, I'd budget real time for characterizing your actual sources of image variance before trusting a validation split that was collected in one sitting. It's unglamorous work and it matters more than most of the modeling decisions that get all the attention.