In this paper, a novel hybrid technique is proposed for transient stability analysis on grid connected Wind-Diesel-PV hybrid system. The proposed hybrid methodology is combination of the dwarf mongoose optimization algorithm (DMO) and the recalling enhanced recurrent neural network (RERNN) named DMO-RERNN. The main purpose of this work is to consider various elements on hybrid system for the analysis of transient stability according to different conditions. The voltage profile of hybrid system is enhanced using the proposed unified power flow controller (UPFC), which also has higher performance improving transient performance compared to the conventional ANN, PI and fuzzy-sliding mode controller. Considering the proposed technique, DMO is used to find the optimal global solution for the fault predicted by the RERNN approach. The proposed system is executed on MATLAB work platform; its performance with existing systems is analyzed. The result proves that the proposed hybrid technique based UPFC controller provides better results compared with other existing technique. The efficiency of the PI is 82.136, ANN is 77, Fuzzy Sliding Mode is 65.097% and proposed technique is 97.99038%.