The photoisomerization of the all-trans protonated Schiff base of retinal (SBR+) in solution is highly inefficient. The present theoretical and experimental investigation aims at disclosing the mechanisms of ultrafast, non-reactive relaxation of SBR+ that lead to the drastic decrease in the isomerization yield in non-polar solvents. Our pump-probe measurements demonstrate the sensitivity of the all-trans SBR+ excited-state dynamics on the electrostatic interaction with the surrounding counterions and the crucial importance of the chromophore microenvironment. Our computational study focuses for the first time on the retinal chromophore-counterion pairs that are formed in non-polar solvents. By employing TDDFT-based nonadiabatic dynamics simulations and ADC(2) reaction paths calculations we found that internal conversion from the initially excited state to an inter-molecular charge transfer state with excitation localized on the counterion, leads to dissociation of the chromophore-counterion pair and to the abortion of isomerization. Barriers to conical intersection with the inter-molecular charge transfer state were found in the range 0.42-0.67 eV at the ADC(2) level. The existence of a barrier along the non-reactive relaxation pathways explains the observation that in solution the excitation on the blue edge of the SBR+ absorption leads to decrease in the isomerization yield with respect to the excitation at the red edge.