Aerial Robot Arms (ARAs) enable aerial drones to interact and influence objects in various environments. Traditional ARA controllers need the availability of a high-precision model to avoid high control chattering. Furthermore, in practical applications of aerial object manipulation, the payloads that ARAs can handle vary, depending on the nature of the task. The high uncertainties due to modeling errors and an unknown payload are inversely proportional to the stability of ARAs. To address the issue of stability, a new adaptive robust controller, based on the Radial Basis Function (RBF) neural network, is proposed. A three-tier approach is also followed. Firstly, a detailed new model for the ARA is derived using the Lagrange–d’Alembert principle. Secondly, an adaptive robust controller, based on a sliding mode, is designed to manipulate the problem of uncertainties, including modeling errors. Last, a higher stability controller, based on the RBF neural network, is implemented with the adaptive robust controller to stabilize the ARAs, avoiding modeling errors and unknown payload issues. The novelty of the proposed design is that it takes into account high nonlinearities, coupling control loops, high modeling errors, and disturbances due to payloads and environmental conditions. The model was evaluated by the simulation of a case study that includes the two proposed controllers and ARA trajectory tracking. The simulation results show the validation and notability of the presented control algorithm.