A majority of all infectious diseases manifest some climate-sensitivity. However, many of those sensitivities are not well understood as meteorological drivers of infectious diseases co-occur with other drivers exhibiting complex non-linear influences and feedback. This makes it hard to dissect their individual contributions. Here we apply a novel deep learning Explainable AI (XAI) compartment model with covariate drivers and dynamic feedback to predict and explain the dengue incidence across Sri Lanka. We compare the compartmental Susceptible-Exposed-Infected-Recovered (SEIR) model to a deep learning model without a compartmental structure. We find that the covariate compartmental hybrid model performs better and can describe drivers of the dengue spatiotemporal incidence over time. The strongest drivers in our model in order of importance are precipitation, socio-demographics, and normalized vegetation index. The novel method demonstrated can be used to leverage known infectious disease dynamics while accounting for the influence of other drivers and different population immunity contexts. While allowing for interpretation of the covariate driver influences, the approach bridges the gap between dynamical compartmental and data driven dynamical models.