The aim of this population-based cross-sectional study was to assess the prevalence of healthcare avoidance during the COVID-19 pandemic and its associated factors among the Sámi population in Sweden. Data from the “Sámi Health on Equal Terms” (SámiHET) survey conducted in 2021 were used. Overall, 3,658 individuals constituted the analytical sample. Analysis was framed using the social determinants of health framework. The association between healthcare avoidance and several sociodemographic, material, and cultural factors was explored through log-binomial regression analyses. Sampling weights were applied in all analyses. Thirty percent of the Sámi in Sweden avoided healthcare during the COVID-19 pandemic. Sámi women (PR: 1.52, 95% CI: 1.36–1.70), young adults (PR: 1.22, 95% CI:1.05–1.47), Sámi living outside Sápmi (PR: 1.17, 95% CI: 1.03–1.34), and those having low income (PR: 1.42, 95% CI:1.19–1.68) and experiencing economic stress (PR: 1.48, 95% CI: 1.31–1.67) had a higher prevalence of healthcare avoidance. The pattern shown in this study can be useful for planning future pandemic responses, which should address healthcare avoidance, particularly among the identified vulnerable groups, including the active participation of the Sámi themselves.