It is plausible that polymorphisms in the estrogen receptor alpha and beta genes (ESR1 and ESR2) may modulate the association between enterolactone and breast cancer. Seven polymorphisms in ESR1 (rs827422, rs1709184, rs2347867, rs3020328, rs72207, rs2982896, and rs2234693) and 5 polymorphisms in ESR2 (rs915057, rs1269056, rs1256033, rs3020450, and rs3020443) were selected. The risk of breast cancer for these polymorphisms was estimated among 542 cases and 1076 matched controls from the population-based Malmö Diet and Cancer cohort. The joint effect of these polymorphisms and enterolactone was estimated among those individuals about whom we had information on enterolactone blood concentration (365 cases and 728 controls). Breast cancer risk was not significantly associated with any of the selected polymorphisms. We found a tendency for an interaction between a polymorphism in intron 3 of ESR1 (rs2347867) and enterolactone concentration (P = 0.07). Breast cancer and enterolactone concentration were not associated among those homozygous for the major allele (A) (P = 0.93), whereas we found an inverse association among carriers of the minor allele (G) (P = 0.007). None of the other polymorphisms seem to modify the association between enterolactone and breast cancer. This study suggests that the protective association of enterolactone is reasonably robust across the investigated genotypes. The suggested interaction between enterolactone concentration and rs2347867 needs to be confirmed in larger samples.