The effect of dietary cholesterol on cognitive function is debatable. While eggs contain high levels of dietary cholesterol, they provide nutrients beneficial for cognitive function. This study examined the effects of egg consumption on change in cognitive function among 890 ambulatory adults (N = 357 men; N = 533 women) aged ≥55 years from the Rancho Bernardo Study who attended clinic visits in 1988–1991 and 1992–1996. Egg intake was obtained in 1988–1991 with a food frequency questionnaire. The Mini-Mental Status Exam (MMSE), Trails B, and category fluency were administered at both visits to assess cognitive performance. Sex-specific multiple regression analyses tested associations of egg intake with changes in cognitive function after adjustment for confounders. The mean time between visits was 4.1 ± 0.5 years; average ages were 70.1 ± 8.4 in men and 71.5 ± 8.8 in women (p = 0.0163). More men consumed eggs at higher levels than women; while 14% of men and 16.5% of women reported never eating eggs, 7.0% of men and 3.8% of women reported intakes ≥5/week (p = 0.0013). In women, after adjustment for covariates, egg consumption was associated with less decline in category fluency (beta = −0.10, p = 0.0241). Other associations were nonsignificant in women, and no associations were found in men. Results suggest that egg consumption has a small beneficial effect on semantic memory in women. The lack of decline observed in both sexes suggests that egg consumption does not have detrimental effects and may even have a role in the maintenance of cognitive function.