The visual system can process diverse stimuli and make the decision to execute appropriate behaviors, but it remains unclear where and how this transformation takes place. Innate visually evoked behaviors such as hunting, freezing, and escape are thought to be deeply conserved, and have been described in a range of species from insects to humans. We found that zebrafish larvae would respond to predator-like visual stimuli with immobility and bradycardia, both hallmarks of freezing, in a head-fixed behavioral paradigm. We then imaged the zebrafish visual system while larvae responded to different visual stimuli with hunting, freezing, and escape behaviors and systematically identified visually driven neurons and behaviorally correlated sensorimotor neurons. Our analyses indicate that within the optic tectum, broadly tuned sensory neurons are functionally correlated with sensorimotor neurons which respond specifically during one behavior, indicating that it contains suitable information for sensorimotor transformation. We also identified sensorimotor neurons in four other areas downstream of the tectum, and these neurons are also specific for one behavior, indicating that the segregation of the pathways continues in other areas. While our findings shed light on how sensorimotor neurons may integrate visual inputs, further investigation will be required to determine how sensorimotor neurons in different regions interact and where the decision to behave is made.