Refractory liner bricks in the hot zone of rotary lime kilns can sustain wear and corrosion during contact with fuel ashes and quicklime (QL), a product composed mainly of CaO. The effects on a MgO-based refractory after exposure at 1400 °C for 96 h to olive pomace ash (OPA) and coal ash (CA), with and without QL, were investigated. Exposure of the refractory to only OPA caused slag intrusion with no ash deposits remaining on top, while CaMgSiO4 (monticellite) was also identified as a new phase. When exposed to only CA, the refractory exhibited dissolution into the molten slag and 0.5–2 mm cracks were found on the surface interfacing the ash. Mg2SiO4 (forsterite) and CaMgSiO4 were identified as new formed phases. Exposure of the refractory to OPA + QL and CA + QL caused less slag intrusion and substantial amounts of ash/QL deposit remained afterwards. No new phases were identified. The differences in interactions between the exposure materials and refractory were supported by thermochemical equilibrium analysis. Apparent Ca-Si–rich or Ca-rich melts were found in all the exposed samples, but potassium (K) was found to be depleted in all samples, including those involving OPA, which was rich in K. Furthermore, with the exception of exposure to only CA, the other exposures caused the cold crushing strength (CCS) of the refractory to increase compared to its original value. This was attributed to the sintering of the refractory microstructure. The CCS of the refractory decreased after exposure to only CA. The findings of this study enhance understanding of how CA and OPA impact MgO refractories in lime kilns, supporting initiatives aiming at reducing fossil fuel use. The results are encouraging and motivate further investigation.