A retrospective evaluation of 228,104 pediatric emergency department (ED) visits from two academic children’s hospitals assessed a large language model’s (LLM) accuracy in determining pairwise triage acuity compared with a clinical reference standard. The study identified 358,040 pediatric ED visits during the study period, with 228,104 meeting inclusion criteria, assembled into 114,052 comparison pairs. The cohort included 53.9% male and 15.7% White children. Immediate and non-urgent assignments comprised 2.3% (n = 5354) and 5.0% (n = 11,439) of visits, respectively, while urgent and less urgent accounted for 38.9% (n = 88,709) and 35.3% (n = 80,556) of visits. The LLM achieved an overall accuracy of 0.73 (95% CI, 0.73–0.74) in identifying the higher-acuity visit in a pair. Accuracy improved as the difference in ESI scores widened, with the LLM performing best on immediate-non-urgent pairs (0.96; 95% CI, 0.93-0.99) and worst on immediate-emergent pairs (0.61; 95% CI, 0.57–0.64). It also performed poorly on urgent-less urgent pairs (0.65; 95% CI,