A significant amount of hazardous compounds has leaked into the environment due to the widespread usage of organic dyes, and it is essential that these dangerous contaminants be removed in a sustainable way. This study used varying amounts of H2O2 (0, 0.5, 1.5, 3, and 5) mM/L to extract the dye from the aqueous solution. Furthermore, concentrations of 0.4, 1, 1.7, and 2.3 mM/L of Fe+2 as FeSO4.7H2O were also utilized. Batch Advanced Oxidation Process (AOP) was carried out under various working conditions, including: contact time (5–60 min), mixing speed (100–300 rpm), and UV light intensity (0–40 Watt). Utilizing experimental data, the AOP efficiency of Dispersed Red 17 Dye was calculated. Genetic Cascade-forward Neural Network (GCNN) was employed as a machine-learning tool to forecast the oxidation efficiency and the amount of dye that would be removed from the aqueous solution, specifically Dispersed Red 17. When compared to experimental data, the best model had an R2 correlation value of 0.955. The findings of the importance analysis showed that the studied parameters affected the discoloration efficiency with order of: H2O2, UV, Fe+2, mixing speed, and contact time. The obtained results demonstrated the effectiveness of GCNN as a novel approach in forecasting the Dispersed Red 17 Dye's AOP efficiency.