Tune parameters without chasing noise
Goal
Understand why repeated search makes the best historical result look better than it may be.
Lesson
Every parameter combination is another chance to fit random variation. The winner of a large search can be a poor estimate of future behavior even when each individual run is implemented correctly. Small samples, many variants, narrow ranges, and repeated inspection all increase this risk.
Keep the search space small and justified. Log how many variants were tried. Compare a tuned result with the original baseline, examine whether nearby settings behave similarly, and evaluate only once on a frozen holdout. Prefer robust behavior across instruments and periods over a single peak score.
Practice
Plot or tabulate performance for neighboring parameter values. If one isolated setting is much better than all nearby settings, list possible explanations besides a true edge.