The current paper studies the possibility of allowing a zero lower bound on the physical density in material distribution based topology optimization. We limit our attention to the standard test problem of minimizing the compliance of a linearly elastic structure subject to a constant forcing. First order tensor product Finite Elements discretize the problem. An elementwise constant material indicator function defines the discretized, elementwise constant, physical density by using filtering and penalization. To alleviate the ill-conditioning of the stiffness matrix, due to the variation of the elementwise constant physical density, we precondition the system. We provide a specific spectral analysis for large matrix sizes for the one-dimensional problem with Dirichlet-Neumann conditions in detail, even if most of the mathematical tools apply also in a d-dimensional setting, d >= 2. It is easy to find an elementwise constant material indicator function so that the resulting preconditioned system matrix is singular when allowing the vanishing physical densities. However, for a large class of material indicator functions, the corresponding preconditioned system matrix has a condition number of the same order as the system matrix for the case when the physical density is one in all elements. Finally, we critically report and illustrate results from numerical experiments: as a conclusion, it is indeed possible to solve large-scale topology optimization problems, allowing a vanishing physical density, without encountering ill-conditioned system matrices during the optimization.