Optical flow calculation algorithms are hard to implement on hardware level in real-time due to their complexity and high computational load. In this work, we present a novel hierarchical block matching based optical flow algorithm. The algorithm estimates the initial optical flow with block matching based methods, and refines the vectors with local smoothness constraints in each hierarchy level. We evaluate the proposed algorithm with novel data sets and provide results compared to ground truth optical flow. Furthermore, we present a hardware architecture of the proposed algorithm for calculating optical flow in real-time. The presented design can process 640x480 resolution at 26 frames per second (fps).