Distribution-Constrained Optimization for Reliable ML-Guided 5'UTR Sequence Design

The 5' untranslated region (5'UTR) is critical for translation initiation, making its sequence design important for mRNA therapeutics and protein-production cell lines. Deep-learning models predicting translation efficiency (mean ribosome load, MRL) from 5'UTR sequence are often combined with genetic algorithms (GAs) for optimization. However, optimizing against a model trained on offline data risks reward hacking, where the model's estimation error outside the training distribution is exploited, leading to sequences that score high in silico but fail in the wet lab. This paper presents a constrained optimization that keeps candidates within a trust region where the predictor's validated accuracy holds. The authors compare k-nearest-neighbor (KNN) distance in the predictor's embedding space against pseudo-perplexity (PPPL) from the encoder and LM head as out-of-distribution (OOD) scores. For nucleotide sequences, PPPL fails to separate in- vs out-of-distribution due to the small vocabulary, whereas KNN distance is an effective OOD score that can define a trust region even from unlabeled native UTR sequences. Using KNN distance as a hard GA constraint keeps all candidates inside the trust region while maintaining predicted MRL. Under unconstrained optimization, 72-96% of final-generation candidates left the trust region (self-KNN p95), while the hard constraint holds predicted MRL at the unconstrained level and yields about 4.3x more selectable low-risk candidates than post-hoc filtering of the unconstrained output. Comparing an output extrapolation guard, reference-sequence similarity, and structural accessibility (RNAplfold), the guard and similarity constraint suppress OOD as a side effect, while making accessibility a secondary objective broadens the search without suppressing OOD.