The effect of wood fiber anisotropy and their geometrical features on wood fiber composite stiffness is analyzed. An analytical model for an N-phase composite with orthotropic properties of constituents is developed and used. This model is a straightforward generalization of Hashin’s concentric cylinder assembly model and Christensen’s generalized self-consistent approach. It was found that most macro-properties are governed by only one property of the cell wall which is very important in attempts to back-calculate the fiber properties. The role of lumen (whether it filled by resin or not) has a very large effect on the composite shear properties. It is shown that several of the unknown anisotropic constants characterizing wood fiber are not affecting the stiffness significantly and rough assumptions regarding their value would suffice. The errors introduced by application of the Hashin’s model and neglecting the orthotropic nature of the material behavior in cylindrical axes are evaluated. The effect of geometrical deviations from circular cross-section, representing, for example, collapsed fibers, is analyzed using the finite element method (FEM) and the observed trends are discussed.