The ever-increasing demand for training and inferring with larger machine-learning models requires more efficient hardware solutions due to limitations such as power dissipation and scalability. Optics is a promising contender for providing lower power computation, since light propagation through a nonabsorbing medium is a lossless operation. However, to carry out useful and efficient computations with light, generating and controlling nonlinearity optically is a necessity that is still elusive. Multimode fibers (MMFs) have been shown that they can provide nonlinear effects with microwatts of average power while maintaining parallelism and low loss. We propose an optical neural network architecture that performs nonlinear optical computation by controlling the propagation of ultrashort pulses in MMF by wavefront shaping. With a surrogate model, optimal sets of parameters are found to program this optical computer for different tasks with minimal utilization of an electronic computer. We show a remarkable decrease of 97% in the number of model parameters, which leads to an overall 99% digital operation reduction compared to an equivalently performing digital neural network. We further demonstrate that a fully optical implementation can also be performed with competitive accuracies.