We have determined limits of detection (LODs) of DNA hybridization using an optical detection by two different method, i.e., surface plasmon fluorescence spectroscopy (SPFS) and optical waveguide fluorescence spectroscopy (OWFS), however, an identical sensor. Both techniques have a similar detection principle using the enhanced electromagnetic field to excite fluorophores and to monitor DNA hybridization in real-time with high sensitivity. Firstly, the detection of DNA hybridization was conducted by SPFS and the LOD value was found to be 2.0 pM (2.0 $\times$ 10$^{-12}$ mol/L). The same study was further conducted by OWFS, and a LOD of 170 fM (1.7 $\times$ 10$^{-13}$ mol/L) and 100 fM (1 $\times$ 10$^{-13}$ mol/L) was achieved for TM$_1$ and TE$_0$ modes, respectively, indicating that an improvement of the LOD of one order of magnitude is possible with OWFS. Furthermore, the photonic mode density at resonance of each mode was calculated, and these values are reflected in the LOD values. The present study illustrates the potential of OWFS as biosensor applications being able to detect in real-time even trace amounts of analyte.