Streamlining AI Experimentation with Open WebUI, LiteLLM, and Amazon Bedrock

I've been eager to explore the latest language models without the hassle of managing multiple accounts or navigating complex cloud interfaces. My goal was simple: I wanted a streamlined way to access Amazon Bedrock's powerful AI models, particularly Anthropic's Claude 3, without needing to log in to the AWS console.

Why Amazon Bedrock?

Amazon Bedrock's pay-per-use model is particularly appealing, offering flexibility and cost-effectiveness for experimenting with various models. Since I already had an AWS account, it made sense to leverage Bedrock rather than signing up directly with Anthropic or other AI providers.

What's more, Amazon Bedrock's expanding catalog of AI models from various providers meant that as new and improved models become available, I can easily enable and use them without any additional setup.

The Solution

In this post, I'll share how I combined Open WebUI, Ollama, LiteLLM, and Amazon Bedrock to create a user-friendly, flexible AI workspace that meets these needs. This setup not only simplifies access to cutting-edge AI models but also provides a seamless interface for interacting with both local and cloud-based AI solutions.

What is Open WebUI?

Open WebUI is an open-source interface designed for developers to interact with various AI models. It provides a seamless experience for engaging with both local and remote language models, simplifying the process of AI experimentation and development.

Key features of Open WebUI include:

• Unified Interface: A single, intuitive interface for interacting with multiple AI models, both local and cloud-based.
• Model Flexibility: Easy switching between different AI models, facilitating quick comparisons and experimentation.
• Customizable Prompts: Creation and reuse of prompt templates, streamlining development workflows.
• Document Analysis (RAG): Support for Retrieval-Augmented Generation, enabling queries on local documents.
• Integration Capabilities: Seamless integration with tools like Ollama for local models and LiteLLM for cloud services such as Amazon Bedrock.

These features make Open WebUI a powerful tool for developers looking to leverage AI capabilities in their projects or research, offering a balance of simplicity and functionality.

The Best of Both Worlds

Open WebUI integrates smoothly with Ollama, a tool that allows users to run large language models locally. This integration enables users to leverage the power of AI models on their own hardware, providing privacy and reducing latency for certain applications.

In addition to local models, Open WebUI can connect to any remote service providers that adhere to the Open AI API spec. This capability allows users to access state-of-the-art models hosted on remote infrastructure, benefiting from the scale and performance of cloud-based AI solutions.

Enabling Models in Amazon Bedrock

To use Amazon Bedrock models with Open WebUI, you first need to enable the desired models in your AWS account. Follow these steps:

• Log in to the AWS Management Console.
• Navigate to the Amazon Bedrock console.
• In the left sidebar, click on "Model access".
• Browse the available models and select the ones you wish to use.
• For each model, click the "Request access" button.
• Wait for approval (which is usually instantaneous for most models).
• Once approved, the status will change to "Access granted".

After enabling, these models become available for use through the Amazon Bedrock API. In our setup, LiteLLM Proxy will handle the API interactions, abstracting away the complexity from Open WebUI.

Proxying Access to Amazon Bedrock with LiteLLM Proxy

LiteLLM Proxy is a crucial component in connecting Open WebUI to Amazon Bedrock. It exposes an Open AI compatible API which translates requests from Open WebUI into a format compatible with many popular providers.

LiteLLM Proxy requires a configuration file telling it which providers to configure. Here, we configure access to Anthropic's Claude 3's Sonnet and Haiku models hosted on Amazon Bedrock.

In addition to this, AWS access credentials need to be exposed to the LiteLLM process via environment variables. We'll cover that in the next section.

Bringing it all together with Docker Compose

To simplify running both LiteLLM Proxy and Open WebUI we can use Docker Compose to start both services and link them together.

Some things to note:

• Open WebUI will be exposed on port 3000 on your local machine.
• Auth is disabled via WEBUI_AUTH: "false". This prevents you needing to sign-in which is overkill for a single-user setup.
• We set OPENAI_API_BASE_URL on the webui service to point at the litellm service. OPENAI_API_KEY is set to a dummy value of “anything” as it's strictly not needed.
• The extra_hosts section allows Open WebUI to access Ollama running locally on your machine.
• Your local ~/.aws directory is mounted inside the litellm container and the AWS_PROFILE environment variable tells LiteLLM Proxy which profile to use. Likewise, AWS_REGION_NAME is set to us-east-1 as that is the AWS region I have Bedrock configured in.
• The LiteLLM config file mentioned previously is mounted into the litellm container.

To run this simply execute docker compose up in the same directory as the above compose.yaml file.

Once running, open your web browser and navigate to localhost:3000 to access Open WebUI. If everything is configured correctly you should be able to choose the bedrock models from the model drop down.

Conclusion

By combining Open WebUI, Ollama, LiteLLM, and Amazon Bedrock, you can create a powerful and flexible AI interaction platform. This setup allows you to leverage both local and cloud-based models, providing a comprehensive solution for AI experimentation and development.

With the ability to switch between local and cloud-based models, make Open WebUI an versatile tool for a wide range of AI applications. Whether you're a researcher analysing complex datasets, a developer prototyping AI-powered applications, or an AI enthusiast exploring the capabilities of different models, this integrated approach offers the best of both worlds. It provides the privacy and control of local models, the power and scalability of cloud-based AI services, and a user-friendly interface to tie it all together.

By leveraging this setup, you can create a personalised AI workspace that adapts to your specific needs, balancing performance, privacy, and accessibility in a way that suits your individual requirements.

Photo by Daniele Franchi on Unsplash.