We consider finite difference approximations of the second derivative, exemplified in Poisson's equation, the heat equation, and the wave equation. The finite difference operators satisfy a summation-by-parts (SBP) property, which mimics the integration-by-parts principle. Since the operators approximate the second derivative, they are singular by construction. When imposing boundary conditions weakly, these operators are modified using simultaneous approximation terms. The modification makes the discretization matrix nonsingular for most choices of boundary conditions. Recently, inverses of such matrices were derived. However, for problems with only Neumann boundary conditions, the modified matrices are still singular. For such matrices, we have derived an explicit expression for the Moore-Penrose inverse, which can be used for solving elliptic problems and some time-dependent problems. For this explicit expression to be valid, it is required that the modified matrix does not have more than one zero eigenvalue. This condition holds for the SBP operators with second and fourth order accurate interior stencil. For the sixth order accurate case, we have reconstructed the operator with a free parameter and show that there can be more than one zero eigenvalue. We have performed a detailed analysis on the free parameter to improve the properties of the second derivative SBP operator. We complement the derivations by numerical experiments to demonstrate the improvements.