What Is a Vector Database and When Do You Need One?

Your team wants an AI assistant that answers questions from your help center, contracts or product catalog. Someone says you need a vector database. Someone else says your existing database can do it. A vendor says you need their platform.

All 3 might be right, depending on your data and scale. This guide explains what a vector database is in plain English, how vector search works, and, most usefully, how to tell when a regular database is enough and when adding dedicated vector search is worth the cost and complexity.

A vector database in plain English

Traditional databases are great at exact matches. Find the order with ID 10472. List customers in Texas. Show invoices over $5,000. They also handle keyword search reasonably well.

They struggle with questions about meaning. A customer searches “how do I get my money back” and your help article is titled “Refund policy.” No keywords overlap, so a keyword search may miss it.

A vector database solves this by storing embeddings. An embedding is a list of numbers, often hundreds or thousands long, produced by an AI model to represent the meaning of a piece of content. Content with similar meaning gets similar numbers. “Get my money back” and “refund policy” end up close together, even though the words differ.

A vector database stores these embeddings and answers 1 core question very fast: which stored items are most similar to this one?

How vector search works

Most vector search follows the same steps:

1. Create embeddings. Run each document, product description, image or chunk of text through an embedding model, which outputs a vector.
2. Store them. Save each vector alongside the original content and useful metadata, such as title, date, department or access level.
3. Embed the query. When a user searches, convert their query into a vector with the same model.
4. Find the nearest neighbors. Compare the query vector to stored vectors using a distance measure such as cosine distance, and return the closest matches.

Exact vs approximate search

Comparing a query against every stored vector, called exact search, gives perfect results but slows down as your data grows. So most systems use approximate nearest neighbor (ANN) indexes, which trade a little accuracy for big speed gains.

The most widely used ANN method is HNSW, short for Hierarchical Navigable Small World graphs, described by Malkov and Yashunin in a 2016 paper. It organizes vectors in layered graphs so a search can hop quickly toward the closest matches. The pgvector documentation summarizes the trade-off well: without an index you get exact search with perfect recall; with an HNSW or IVFFlat index you get approximate search that is much faster but can return slightly different results.

Vector database vs regular database

The 2 are not rivals. Most real systems use both: the regular database holds the source of truth, and vector search helps find the right records by meaning.

When a regular database is enough

You often do not need a separate vector database. A regular database, or one with built-in vector features, is usually enough when:

• Your data is modest. Small and mid-sized document collections can often be handled by an extension or built-in vector search in a database you already run.
• Exact or keyword search already works. If users search by SKU, name, ID or specific terms, a standard index or full-text search may beat semantic search.
• Your database already supports vectors. PostgreSQL has the pgvector extension, MongoDB Atlas has Vector Search and search engines like Elasticsearch and OpenSearch support vector search. Keeping vectors next to your operational data simplifies permissions, backups and syncing.
• You need strict filtering. When results must respect customer, region or access rules, doing it in the same database as the rest of your data is easier to get right.
• You are prototyping. For a first RAG pilot, even exact search over a small set of embeddings can be fast enough.

pgvector, for example, supports cosine, inner product, L1 and L2 distance, offers HNSW and IVFFlat indexes and can combine vector search with Postgres full-text search for hybrid results. Its documentation notes indexing limits, such as up to 2,000 dimensions for the standard vector type, with half-precision vectors indexable up to 4,000.

When dedicated vector search is worth adding

A dedicated vector database, such as Pinecone, Weaviate, Milvus, Qdrant or Chroma, earns its place when:

• Scale is large. Very large vector collections or very high query volume can strain a general-purpose database.
• Latency is critical. Real-time recommendations or search across a large catalog need consistently fast responses.
• Data changes constantly. Heavy, continuous inserts and updates benefit from engines built to keep indexes fresh.
• You need advanced features. Multi-tenant isolation, built-in hybrid search and reranking, quantization to save memory or managed scaling can save engineering time.
• Search is a product in itself. If semantic search or recommendations are central to what you sell, specialized tooling is often worth it.

Key takeaway: Start with the database you already run if it supports vector search. Move to a dedicated vector database when measured scale, speed or feature needs justify another system to secure, sync and pay for.

Your options compared

Pricing varies widely by option and usage. Managed services typically charge based on storage, compute or query volume, while self-hosted options cost you infrastructure and staff time. Remember that generating embeddings is a separate cost, paid to an embedding model provider or in your own compute. Our guide to FinOps and cloud cost optimization covers how to keep these costs visible.

Common mistakes with vector search

• Treating it as a fix for bad content. If your documents are outdated or contradictory, better search just finds the wrong answer faster.
• Skipping keyword search. Product codes, names and exact phrases often work better with keyword search. Hybrid search combines both.
• Ignoring permissions. Filter results by user access before they reach an AI model, or the assistant may reveal documents a user should not see.
• Changing embedding models casually. Vectors from different models are not comparable. Switching models means re-embedding everything.
• Not measuring. Without a test set of real queries and expected results, you cannot tell whether tuning helped.

Where to start

1. Define the use case. Semantic search, a document Q&A assistant, recommendations or duplicate detection each need different tuning.
2. Check what you already have. If your current database or search engine supports vectors, start there.
3. Prepare content. Clean documents, split them into sensible chunks and attach metadata like source, date and access level.
4. Pick an embedding model and use the same one for content and queries.
5. Build a test set of 30 to 50 real questions with the documents that should come back.
6. Measure, then scale. Track result quality and latency. Move to a dedicated vector database only if the numbers say you need to.

If your goal is an AI assistant that answers from your documents, vector search is one piece of a larger pattern. Our explainer on retrieval-augmented generation covers the full picture, and our guide to what generative AI is explains the models on top. Teams building AI agents often use the same retrieval layer.

The bottom line

A vector database finds content by meaning instead of exact words, using embeddings and fast nearest-neighbor search. It is a key building block for semantic search, recommendations and AI assistants grounded in your data. For most businesses starting out, vector search inside a database you already run is enough. Add a dedicated vector database when your scale, speed or feature requirements prove you need one.

Sources

• pgvector on GitHub — distance metrics, HNSW and IVFFlat indexes, exact vs approximate search, dimension limits and hybrid search
• Malkov and Yashunin, HNSW paper (arXiv:1603.09320) — the Hierarchical Navigable Small World approximate nearest neighbor method
• MongoDB Atlas Vector Search overview — vectors stored with operational data, ANN with HNSW, pre-filtering and hybrid search
• Elastic Docs: kNN search — approximate and exact kNN search with filtering in Elasticsearch