The hippocampus is affected early in Alzheimer’s disease (AD) and altered hippocampal functioning influences normal cognitive aging. Here, we used task-based functional MRI to assess if the APOE ɛ4 allele or a polygenic risk score (PRS) for AD was linked to longitudinal changes in memory-related hippocampal activation in normal aging (baseline age 50–95, n = 292; n = 182 at 4 years follow-up, subsequently non-demented for at least 2 years). Mixed-models were used to predict level and change in hippocampal activation by APOE ɛ4 status and PRS based on gene variants previously linked to AD at p ≤ 1, p < 0.05, or p < 5e−8 (excluding APOE). APOE ɛ4 and PRSp<5e−8 significantly predicted AD risk in a larger sample from the same study population (n = 1542), while PRSp≤1 predicted memory decline. APOE ɛ4 was linked to decreased hippocampal activation over time, with the most prominent effect in the posterior hippocampi, while PRS was unrelated to hippocampal activation at all p-thresholds. These results suggests a link for APOE ɛ4, but not for AD genetics in general, on functional changes of the hippocampi in normal aging.