The high computational costs of deep convolutional neural networks hinder their deployment in real-world applications, including pulmonary nodule detection from CT scans where large 3D image sizes amplify the issue. This paper presents a novel 3D method to detect pulmonary nodules, based on anchor-free U-shaped networks, AFNet. A shifted convolution is further introduced to replace standard 3D convolutions, which reduces both the model sizes and FLOPs (floating-point operations). The shift operator is parameter-free, enabling 3D context fusion between CT slices using 2D convolutions. Extensive experiments on a large-scale lung nodule detection dataset validate the effectiveness of the proposed methods. The AFNet backbone is first proven to be comparable to the previous state of the art (e.g., NoduleNet). We then show that the proposed method with shifted convolutions balances model complexity and performance better than several lightweight methods, and generalizes well with different backbones. As an example, compared to the vanilla model, AFNet with shifted convolutions increases average FROC by 3.08% and reduces FLOPs (floating-point operations) and parameters by 62.40% and 66.62%, respectively.