Most people's cognitive abilities decline with age, with significant and partly genetically driven, individual differences in rate of change. Although APOE 4 and genetic scores for late-onset Alzheimer's disease (LOAD) have been related to cognitive decline during preclinical stages of dementia, there is limited knowledge concerning genetic factors implied in normal cognitive aging. In the present study, we examined three potential genetic predictors of age-related cognitive decline as follows: (1) the APOE 4 allele, (2) a polygenic score for general cognitive ability (PGS-cog), and (3) a polygenic risk score for late-onset AD (PRS-LOAD). We examined up to six time points of cognitive measurements in the longitudinal population-based Betula study, covering a 25-year follow-up period. Only participants that remained alive and non-demented until the most recent dementia screening (1-3 years after the last test occasion) were included (n=1087). Individual differences in rate of cognitive change (composite score) were predicted by the PRS-LOAD and APOE 4, but not by PGS-cog. To control for the possibility that the results reflected a preclinical state of Alzheimer's disease in some participants, we re-ran the analyses excluding cognitive data from the last test occasion to model cognitive change up-until a minimum of 6 years before potential onset of clinical Alzheimers. Strikingly, the association of PRS-LOAD, but not APOE 4, with cognitive change remained. The results indicate that PRS-LOAD predicts individual difference in rate of cognitive decline in normal aging, but it remains to be determined to what extent this reflects preclinical Alzheimer's disease brain pathophysiology and subsequent risk to develop the disease.