We have constructed a fitness parameter, characterizing the intrinsic attractiveness for patents to be cited, from attributes of the associated inventions known at the time a patent is granted. This exogenously obtained fitness is shown to determine the temporal growth of the citation network in conjunction with mechanisms of preferential attachment and obsolescence-induced aging that operate without reference to characteristics of individual patents. Our study opens a window to understanding quantitatively the interplay of the rich-gets-richer and fit-gets-richer paradigms that have been suggested to govern the growth dynamics of real-world complex networks.