I failed to replicate the attack later in the evening in a "new" conversation. It does appear to me the model is learning between conversations, even without human input or RLHF.