We report on the search for neutrino-induced particle showers, so-called cascades, in the IceCube-40 detector. The data for this search were collected between April 2008 and May 2009 when the first 40 IceCube strings were deployed and operational. Three complementary searches were performed, each optimized for different energy regimes. The analysis with the lowest energy threshold (2 TeV) targeted atmospheric neutrinos. A total of 67 events were found, consistent with the expectation of 41 atmospheric muons and 30 atmospheric neutrino events. The two other analyses targeted a harder, astrophysical neutrino flux. The analysis with an intermediate threshold of 25 TeV leads to the observation of 14 cascadelike events, again consistent with the prediction of 3.0 atmospheric neutrino and 7.7 atmospheric muon events. We hence set an upper limit of E-2 Phi(lim) <= 7.46 x 10(-8) GeV sr(-1) s(-1) cm(-2) (90% C.L.) on the diffuse flux from astrophysical neutrinos of all neutrino flavors, applicable to the energy range 25 TeV to 5 PeV, assuming an E-nu(-2) spectrum and a neutrino flavor ratio of 1: 1: 1 at the Earth. The third analysis utilized a larger and optimized sample of atmospheric muon background simulation, leading to a higher energy threshold of 100 TeV. Three events were found over a background prediction of 0.04 atmospheric muon events and 0.21 events from the flux of conventional and prompt atmospheric neutrinos. Including systematic errors this corresponds to a 2.7 sigma excess with respect to the background-only hypothesis. Our observation of neutrino event candidates above 100 TeV complements IceCube's recently observed evidence for high-energy astrophysical neutrinos.