Online advertising opportunities are bought and sold in automated auctions driven by real-time bidding. In the case of contextual advertising, the size of a bid is informed by the media context in which the ad will be displayed. In contrast to personalised advertising, contextual advertising is better aligned with privacy acts such as GDPR and CCPA. We investigate how reinforcement learning with human feedback can help optimise contextual advertising under budget constraints. We propose a dynamic epsilon-greedy algorithm that considers the rate of budget consumption during a finite transaction time. The goal is to maximise long-term rewards in a sustainable manner. Our comparative evaluation of fundamental reinforcement learning algorithms on real data suggests that the approach is feasible and effective.