The escalating demands of artificial intelligence have underscored the limitations of traditional electronic computing, particularly in terms of power consumption and scalability. Optical computing offers a promising alternative, providing advantages in speed, parallelism, and energy efficiency by utilizing photons instead of electrons. This thesis explores innovative techniques for implementing programmable nonlinear computation in optical neural networks through two main approaches: leveraging nonlinear optics in optical fibers and utilizing linear optics with structural nonlinearity.

The first approach introduced in this work is the Back-end Programmed Scalable Optical Learning Operator (P-SOLO), an extension of the Scalable Optical Learning Operator (SOLO) framework. P-SOLO leverages spatial-spectral optimization using a digital micromirror device coupled with a chromatic dispersion grating to achieve programmable outputs from multimode fibers. This design enables additional control and programming in optical processing. Demonstrated on a COVID-19 X-ray image classification task, P-SOLO achieved accuracy comparable to that of digital neural networks while reducing the number of training parameters by 99%. These results underline its potential as a scalable, power-efficient solution for optical computing.

The second approach features the Vision Transformer with Only Linear Optics (vTOLO), an enhancement to the Nonlinear Processing with Linear Optics (nPOLO) framework, which integrates nPOLO with a vision transformer through knowledge distillation. Extensive experiments on the MNIST, Fashion MNIST, Imagenette, and CIFAR-10 datasets reveal that vTOLO consistently outperforms nPOLO with superior accuracy. Additionally, vTOLO demonstrates power-law scaling of loss relative to the number of parameters, further reinforcing its potential for large-scale optical neural network development. The thesis also provides a detailed examination of the hardware implementation of vTOLO, focusing on miniaturization and enhanced stability. The resulting design employs a compact, solid multi-bounce structure, leading to a robust system capable of reliable operation beyond controlled laboratory environments. This advancement in optical hardware design showcases the feasibility and scalability of optical computing in real-world applications.