A new preprint from Stanford's Human-Centered AI Lab, available on arXiv, challenges the assumption that fine-tuning a pre-trained policy with online reinforcement learning requires carrying over a pre-trained Q-function. The paper argues that in many common settings, this pre-trained Q-function is actively making things worse. The finding impacts the offline-to-online RL pipeline, which is used in robot learning systems and large language model alignment through reinforcement learning from human feedback. The paper, authored by Perry Dong, Ron Polonsky, Dorsa Sadigh, and Chelsea Finn, is titled "Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?" and answers with "often, no." The consensus was built on Cal-QL, published at NeurIPS 2023 by Mitsuhiko Nakamoto, Chelsea Finn, and collaborators, which engineered a Q-function initialization to ensure Q-values were a lower bound on true value but not overly pessimistic. Cal-QL outperformed the state of the art on 9 of 11 fine-tuning benchmark tasks. The Stanford team identifies that offline RL is structurally required to be conservative about unseen state-action pairs, implemented through penalties in the Q-function loss, which can become a hidden bug during online fine-tuning.