# Building a prompt engineering playground in 2 days with Retool

May 14, 2024

%20copy.jpg)

At Dashworks, we answer users’ work-related questions with information retrieved from the tools they use daily in their jobs, such as Slack, Google Drive & Notion. We use a mixture of self-hosted / open-source and SaaS / proprietary language models to deliver these answers in natural language. How this disparate information is presented to the language models (i.e., how we construct our prompts) can massively impact the response accuracy. Therefore, we wanted to build a playground where it would be easy to tinker with the prompts we use.

We first checked out tools available on the market today. We looked everywhere, from solutions by leading LLM providers to startups and OSS platforms. We ended up building our tool on the Retool platform and couldn’t be happier with the choice.

### Goals and Requirements

Our main requirement when building this tool was deep integration with our existing internal tooling systems. We extensively dogfood our product and use issues we encounter as the driver for triaging projects. If we’re not happy with aspects of our product, there’s a good chance a portion of our user base won’t be happy either.

We previously built a Retool dashboard that lets us see debug logs for our own personal queries to identify any poorly performing parts of the pipeline quickly. Our goal was to enable the flow:

- A teammate flags a query that is not performing for themselves while dogfooding Dashworks
- The on-call reviews their session and identifies a poor LLM interaction
- The on-call experiments whether changes to the prompt resolve the issue
- The tuned prompt is tested against our standard evaluations and the change is incorporated into production

When we evaluated the current tooling, we found it lacking in a few ways, all involving ease of integration with our existing systems:

- All required static data files for running against multiple parameterized inputs, e.g. we would have to upload a CSV of the session data via some UI
- None could easily integrate with our downstream automated scoring systems, e.g. "Did this answer improve on the last answer?"
- None integrated with our domain-specific parameter tweaking, e.g. most expose "model temperature" but we couldn’t control "max document chunk length."

In the past, I have been highly skeptical of low-code/no-code solutions, but given our team’s prior experience working with Retool we sketched out what our version would look like and set aside a few days to make it happen.

### What we built

So what does a usual interaction with this tool look like?

Let’s start by asking a question that will help me write this blog post:

_`What were the considerations that went into building our prompt engineering toolkit?`_

I can then find this session in a connected Retool dashboard and open it in our prompt engineering toolkit so that we can start tweaking the prompt and re-running the answer generation. This was the original answer:

_The considerations that went into building our prompt engineering toolkit were multifaceted, focusing on deep integration with existing internal systems, ease of iteration on prompts, and the ability to conduct thorough evaluations of prompt changes. Here are the key points:_

**_Deep Integration:_** _The toolkit was designed to integrate deeply with our internal tooling systems, allowing us to use issues we encounter as a primary driver for triaging projects._  
**_Evaluation:_** _The toolkit needed to support evaluations that could compute point-wise metrics for our answer pipeline, such as whether an answer satisfied a query or conformed to a style guide. This was to ensure that any prompt changes did not introduce regressions and improved the user experience._
**_Integration with Automated Scoring Systems:_** _The toolkit had to integrate with automated scoring systems to assess whether an answer improved upon the last one._  
**_Domain-Specific Parameter Tweaking:_** _The ability to tweak domain-specific parameters, such as "max document chunk length," was a requirement, as most available tools did not offer this level of control._  
**_Golden Dataset:_** _A curated golden dataset specific to the answer generation component was considered to test different scenarios and improve the tooling based on common failure modes identified over time._  
**_Existing Tooling:_** _The decision to build in-house tooling was driven by the need for secure integration with our retrieval pipeline and the ability to replicate the most useful features of existing applications without being tightly coupled to their tooling._  
**_Future Evolution:_** _Considerations were also made for future versions of the application, such as supporting agentic interactions and configurable styles, which would require the toolkit to handle a wider range of responses._

_These considerations were aimed at creating a prompt engineering toolkit that was not only functional and integrated but also flexible and capable of evolving with the needs of our product and users._

The answer we generated was already pretty great, so for illustration purposes, I will tweak the prompt by adding “Make sure you answer as sarcastically as possible” to the end of the system prompt.

_Oh, let me put on my detective hat and dive into the sea of information to unearth the treasures of knowledge about our prompt engineering toolkit. Here's what I found:_

**_Integration with Internal Systems:_** _The toolkit was designed for deep integration with existing internal tooling systems, which was a primary driver for its development._  
**_Ease of Use:_** _The goal was to create a playground for easy tinkering with prompts. This included the ability to quickly recreate input conditions, refine prompts, and test changes across a set of known queries._  
**_Automated Evaluations:_** _The toolkit was designed to support automated evaluations, such as pointwise and pairwise comparisons, to ensure that changes in prompt construction did not lead to regressions in response quality._  
**_Customization and Control:_** _The team wanted to avoid the limitations of existing tools, which often required static data files and did not integrate well with automated scoring systems or allow for domain-specific parameter tweaking._  
**_Future-Proofing:_** _Considerations were made for how the application might evolve, such as supporting agentic interactions and configurable styles._  
**_Efficiency:_** _The toolkit aimed to streamline the process of prompt engineering, making it faster and more efficient to iterate on prompt changes and regression test against a variety of queries._  
**_Collaboration:_** _The toolkit was designed to facilitate collaboration among team members, allowing for quick identification and resolution of poorly performing queries._

_These considerations reflect a thoughtful approach to building a tool that is deeply integrated, user-friendly, and adaptable to future needs, all while maintaining a high standard of response quality and efficiency._

### Extra goodies

We got to implement a couple of fun things with Retool with minimal extra effort.

Keyboard shortcuts help to satiate my inner vim user, who always asks for a way to eliminate the mouse entirely. We currently have shortcuts for:

- Navigation in and between each section
- Basic vim motions like gg for “go to top” and G for “go to end.”
- Quick actions like re-running the prompt
- A shortcut (?) for opening a view of the list of shortcuts

JSON explorer components expose a full view of semi-structured data. For example, if the user wants to look at the complete session data, adding a modal that directly exposes the data is really easy.

Consistent brand colors and typography. Everything in Retool conforms to a standard color swatch, so it was easy to plug in our brand colors and give everyone a familiar feeling while working with the tool.

### Challenges

There were a couple of frustrations that would be a big quality of life improvement if fixed:

- The Retool code editor is great at quickly working with your components, but it is still very basic. No syntax highlighting, some weird, quirky state issues & worst of all, no vim bindings (semi-kidding on the last one).
- The new list view components offer a virtual view of components meant to significantly improve render time, but this makes interacting with the list elements challenging. For example, we wanted to add a “reset all text boxes” button that would reset to the standard prompt. Because this virtual list view exposed our prompt elements, it was almost impossible to do this.

### Summary

While not perfect, the experience of building with Retool was quite delightful. I have built many front-ends in the past with plain HTML/CSS/JS, React, Flutter, and a variety of other frameworks. While these still offer far more flexibility for user-facing applications, I wouldn’t hesitate to use Retool again for purpose-built, internal-facing applications where functionality and time to implementation are the biggest motivators.