Analysis of environmental DNA detected in lake sediments shows promise to become a great paleoecological technique that can provide detailed information about organism communities living in past environments. However, when interpreting sedimentary environmental DNA records, it is of crucial importance to separate ecosystem responses to large-scale environmental change from “noise” caused by changes in sediment provenance or potential post-depositional DNA mobility. In this study, we show that plant and mammalian communities reconstructed from sediments are strongly affected by sediment provenance, but unaffected by vertical mobility of DNA after sediment deposition. We observe that DNA from aquatic plants was abundant in background sediment, while embedded detrital event layers (sediment deposited under erosion events) primarily contained terrestrial plants; hence, vertical mobility of aquatic plant DNA across sediment layers was negligible within our studied lakes. About 33% of the identified terrestrial plant genera were only found in detrital sediment, suggesting that sediment origin had a strong impact on the reconstructed plant community. Similarly, DNA of some mammalian taxa (Capra hircus, Ursus arctos, Lepus, and Felis) were only or preferentially found in detrital event layers. Temporal changes across the Holocene were the main drivers of change for reconstructed plant communities, but sediment type was the second most important factor of variance. Our results highlight that erosion and sediment provenance need to be considered when reconstructing past mammalian and plant communities using environmental DNA from lake sediments.