Reddit debate highlights safety paradox of training LLMs on harmful data

A viral debate initiated on Reddit by user u/John_Lins argues that large language models (LLMs) are statistically guaranteed to cause harm because they must encode harmful information in their weights to identify and filter it. The post contends that because jailbreaks are always statistically possible due to the non-deterministic nature of LLMs, bad actors will inevitably exploit these systems. The author suggests that to counter this asymmetric threat, LLMs should remain uncensored so that defensive users have access to the same powerful tools as malicious actors. This perspective challenges current safety paradigms focused on strict alignment and censorship, highlighting an ongoing debate within the AI community regarding open-source safety versus centralized guardrails.