With the arrival of Large Language Models (LLMs), there is an explosion of agents characterised in terms of LLM-prompts. But LLMs lack consistency with their answers, and they are prone to hallucinations. This means that LLM-based agents are erratic agents. Hence, there are no guarantees that LLM-based agents will be aligned with an expected behaviour. We argue that formal dialogue verification is the way to go for minimising the potential negative side effects of erratic LLM-based agents. Erratic LLM-based agents are far from complying with basic Trustworthy AI principles such as technical robustness and safety. Formal Dialogue Verification methods provide rigorous mathematical frameworks for verifying fundamental behavioral properties of LLM-based agents.