Recent advances in mobile device technology have freed augmented reality (AR) applications from the constraints of desktops, laptops and head-mounted displays. But they are met with a lack of guidelines on the design and user interactions of mobile device-based AR systems. The situation for developers is further exacerbated by closed-license environments and inflexible solutions. We provide an overview of the design of AR applications on handheld devices, the necessary building blocks and problems that future AR systems need to overcome. This experience was gathered during the design and development of an AR framework for the Android™ platform. User experience evaluations showed a great demand for overlay collision avoidance and the value of being able to freeze AR screens. These will be valuable for the design of future mobile device based AR applications.