DeepSMOTE is an oversampling method that combines autoencoders and the Synthetic Minority Over-sampling Technique (SMOTE) in the autoencoder's latent space to address class imbalances. A key component is a penalty term intended to increase sample diversity, but its formulation and impact are not well understood. We examined two versions: the one described in the original SMOTE paper and the one in the official SMOTE code. We formally analyze both and show that the paper version, contrary to its goal, makes reconstructions within each class more similar, thus reducing diversity. The implemented version instead increases the frequency of class sampling, implicitly rebalancing class contributions to the loss. Building on this analysis, we propose a simple refinement that better matches the intended purpose. Experiments on MNIST, FMNIST, CIFAR-10, and SVHN validate our findings. Code is available at: https://github.com/SG-Azar/DeepSMOTE-penalty.