Skip to content

Use AnythingLLM with the Gateway

AnythingLLM is a full-stack, self-hosted AI application for document chat, RAG, and AI agents. Connect it to the Gateway to use Claude and other models with your documents, all processed in your chosen region.

Prerequisites

Connect the Gateway as the LLM provider

  1. Open AnythingLLM and go to Settings → AI Providers → LLM
  2. Select Generic OpenAI
  3. Fill in:
Field Value
Base URL https://llm.[your-region].amazee.ai/v1
API Key your LLM API key
Model claude-4-5-sonnet (or any chat model from /v1/models)
Model context window 200000
Max Tokens 4096 (or what your use case needs)
  1. Click Save changes

See Available Models for the full list.

Connect pgvector for embeddings (optional)

AnythingLLM can use pgvector as its vector database. To get pgvector credentials for your workspace, contact ai.support@amazee.io.

  1. Go to Settings → AI Providers → Vector Database
  2. Select PGVector
  3. Fill in:
Field Value
Postgres Connection String postgresql://USERNAME:PASSWORD@vectordb.[your-region].amazee.ai:5432/DATABASE
Vector Table Name leave the default, or pick a name
  1. Click Save changes

Embedding model

Go to Settings → AI Providers → Embedder, select Generic OpenAI, and use:

Field Value
Base URL https://llm.[your-region].amazee.ai/v1
Embedding Model titan-embed-text-v2:0
API Key your LLM API key

Warning

AnythingLLM's Generic OpenAI embedder doesn't work with the Gateway right now. It runs without errors, but document search returns the wrong chunks. We're fixing this on the Gateway side.

Verify

  1. Create a workspace, then upload and embed a document
  2. In the workspace settings, set the chat mode to Chat or Query. New workspaces use Automatic, which can hand your question to the agent instead of answering from your documents.
  3. Ask a question about the document. A working setup answers from the document and lists it as a source.

Troubleshooting

"LLM did not respond" errors
Check that the base URL includes /v1 and the model ID exactly matches what's in Available Models.
Embeddings not working
Make sure the embedding model field is set — AnythingLLM won't use the chat model for embeddings automatically.
Slow responses on large documents
This is normal for large context windows. Claude models support up to 200k tokens. For very large documents, consider chunking them into multiple uploads.