Explainable AI (XAI) methods can make the internal logic of machine learning systems interpretable to human reasoning, addressing whether humans can learn from machines that outperform human models. XAI includes feature attribution, counterfactual explanations, data attribution, and mechanistic interpretability, while adjacent tools like symbolic regression, operator learning, and autoencoders form part of a broader explainability workflow for scientific discovery, optimization, and certification. The Perspective distinguishes between model-level explanations and system-level causal claims: XAI methods describe how inputs, learned representations, or internal components influence predictions, but they do not establish causal relationships about the physical system being modeled. System-level causal claims require additional assumptions and evidence, including representative data, validated physical constraints or governing models, robustness under distribution shift, and targeted numerical or physical interventions. XAI is used primarily to generate, organize, and test mechanistic hypotheses; causal interpretations become credible only when the learned model is sufficiently faithful to the system and proposed mechanisms survive independent validation. Explanations have always been central to scientific inquiry, connecting abstract models to causal understanding and experimental validation, but in modern deep learning, learned representations are rarely transparent, so XAI provides an interface between machine predictions and human understanding.