We study ultrafast charge dynamics in plasmonic gold nanoparticles with different sizes. Pump-probe experiments, supported by numerical simulations based on the two-temperature model, reveal cluster-enhanced electron relaxation time with increasing particle size.