This paper presents a multi-input multi-output (MIMO) high-coupled nonlinear model of a twin-rotor aerodynamic system (TRAS). An optimal sliding mode controller (SMC) is proposed to control the TRAS system. Two optimization algorithms, namely Grey Wolf Optimization (GWO) and Whale Optimization Algorithm (WOA), are used to tune the SMC’s parameters. Which are effectively implemented in this controller. The simulation results are given to demonstrate its effectiveness. To evaluate the performance of the proposed controller, a comparison with a previous study—that used Genetic Algorithm (GA), Particle Swarm Optimization (PSO), and Simulated Annealing (SA) algorithms—has been conducted. The results of the proposed controller reveal better performance indices than the previous study. In addition, a novel performance index is presented in this paper as an objective function for designing SMC parameters. Which is identified by the Integral of Quadric Time multiplied by Absolute Error (IQTAE). To verify the effectiveness of the proposed IQTAE, a comparison was conducted with the previous study that used traditional performance indices (ISE, IAE, ITSE, ITAE). To guarantee a fair comparison, apply the same optimization algorithms (GA), (PSO), and (SA) that were utilized in the previous study. By using this method, it was possible to compare the performance indices under the same conditions. The simulation results show the superiority of the novel performance index, especially in reducing overshoot. The percentage enhancements of the overshoot of both azimuth and pitch angles reach, respectively, 58.7% and 99.35% with GA, -0.65% and 70.1% with PSO, and 44.53% and 88.59% with SA.