The central decision-making processes involved in online advertising are often supported by Reinforcement Learning (RL), which serves to optimise long-term accumulative rewards through interactions with evolving environments. While RL’s potential in various real-world applications has been reviewed in extant survey works, the specific ways RL algorithms address online advertising challenges remain unchartered. Therefore, this paper reviews RL applications in this practice area, identifying core challenges and key issues including trust concerns. We categorize reviewed work based on problem domains and propose potential directions for future research. Our goal is to bridge the cross-disciplinary gap in this field, offering perspectives and guidance for researchers and practitioners.