The COVID-19 pandemic led to reports of increased levels of psychological distress and mental health problems world-wide. In Sweden, contrary to most other countries, the COVID-19 strategy was mainly based on voluntary restrictions. It remains unclear whether this reduced mental health problems. We therefore aimed to investigate the long-term impact of the COVID-19 pandemic on mental health in Sweden in terms of mental health service utilisation, antidepressant and anxiolytic/sedative use, and suicide rates for the two years before, during and after the pandemic in a nationwide retrospective register study, covering the entire Swedish population from ten years of age between 1 January 2018 and 31 December 2023. Publicly available data from three national registers were used. We found that, despite the stress induced by the COVID-19 pandemic, there was neither an overall impact on mental health service utilisation nor a post-pandemic rebound. Nonetheless, there were vulnerable subgroups, which could be overlooked when only examining the population as a whole. Young women and girls fared worse in terms of psychoactive substance use and anxiety. Older men fared worse in terms of suicide rates. Identifying vulnerable populations already now, may be a means to effectively mitigate mental health problems during future pandemics.