LLM RAG Web Parser

LLM RAG Web Parser is a Retrieval-Augmented Generation API that crawls websites, extracts their content, and turns them into a queryable knowledge base for AI chat. Given a URL, a breadth-first crawler indexes up to five pages by default, stripping scripts, styles, navigation, headers, footers, and iframes with Cheerio before extracting meaningful text. The cleaned content is split into overlapping 500-character chunks, embedded with Google's gemini-embedding-001 model, and stored in Qdrant with a vector dimension detected automatically from the embedding model. Chat requests embed the user's query, run a cosine-similarity search scoped to the indexed site, retrieve the top five relevant chunks, and pass them as context to a Groq-hosted Llama 3.3 70B model for a grounded answer with source URLs attached. The API also exposes endpoints to list and delete indexed sites, and supports replacing an existing index when a site is re-crawled. Built with Node.js, Express, and TypeScript, and backed by PostgreSQL for metadata, the service ships with a Docker Compose setup for local development.