Web application services and networks encounter a broad range of security and performance anomalies, necessitating sophisticated detection strategies. However, performing anomaly detection in edge cloud environments, often constrained by limited resources, presents significant computational challenges and demands minimized detection time for real-time response. In this paper, we propose a model selection approach for resource efficient anomaly detection in edge clouds by leveraging an adapted Deep Q-Network (DQN) reinforcement learning technique. The primary objective is to minimize the computational resources required for accurate anomaly detection while achieving low latency and high detection accuracy. Through extensive experimental evaluation in our testbed setup over different representative scenarios, we demonstrate that our adapted DQN approach can reduce resource usage by up to 45 % and detection time by up to 85 % while incurring less than an 8 % drop in F1 score. These results highlight the potential of the adapted DQN model selection strategy to enable efficient, low-latency anomaly detection in resource-constrained edge cloud environments.