This paper presents an exploratory work on using Large Language Models (LLM) in User Experience (UX) design. Previous research shows that UX designers struggle to envision novel designs and to prototype with AI as a design material. We set out to investigate the question of how designers can be sensitized to LLMs, and their implications for the professional role of UX designers. Using autobiographical design, we develop a prototype of a digital workspace (the “PromptBoard”) for designing and prototyping chatbots utilizing ChatGPT. A design sprint workshop with six participants is performed, in an effort to answer the research questions by working with the PromptBoard. Discussions and participant-designed artifacts are analysed using thematic analysis. Findings include that participants are able to express design ideas and successfully create chatbots using the tool but express a conflicting sense of lacking creativity or ownership of the results. Implications to the field of UX design are discussed.