Caveolae are bulb-shaped invaginations of the plasma membrane (PM) that undergo scission and fusion at the cell surface and are enriched in specific lipids. However, the influence of lipid composition on caveolae surface stability is not well described or understood. Accordingly, we inserted specific lipids into the cell PM via membrane fusion and studied their acute effects on caveolae dynamics. We demonstrate that sphingomyelin stabilizes caveolae to the cell surface, whereas cholesterol and glycosphingolipids drive caveolae scission from the PM. Although all three lipids accumulated specifically in caveolae, cholesterol and sphingomyelin were actively sequestered, whereas glycosphingolipids diffused freely. The ATPase EHD2 restricts lipid diffusion and counteracts lipid-induced scission. We propose that specific lipid accumulation in caveolae generates an intrinsically unstable domain prone to scission if not restrained by EHD2 at the caveolae neck. This work provides a mechanistic link between caveolae and their ability to sense the PM lipid composition.