Use RAG when the model needs your facts, and fine-tuning when it must behave differently, like holding a format or doing one narrow task. Availability matters too. OpenAI’s fine-tuning is closed to new users, and from 6 January 2027 active existing customers can’t start jobs. The only Claude model on Bedrock’s fine-tuning list reached end of life in its fine-tuning Region on 10 September 2026.
Many comparisons skip a third option: if your documents fit in a few hundred pages, current models can read them all in one cached prompt.
Key takeaways
- OpenAI’s fine-tuning platform is closed to new users, and from 6 January 2027 active existing customers can’t create new fine-tuning jobs either.
- Claude 3 Haiku, the only Claude model on Bedrock’s fine-tuning list, reached end of life on 10 September 2026 in us-west-2, its only fine-tuning Region.
- For adding facts, RAG consistently beat unsupervised fine-tuning in research, and fine-tuning on new facts made models more likely to hallucinate.
- Below about 200,000 tokens (roughly 500 pages), Anthropic says you can put the whole knowledge base in the prompt instead of building RAG.
- RAG lowers hallucination but doesn’t remove it: in 2024 tests, leading RAG-based legal research tools still hallucinated 17% to 33% of the time.
What is the difference between RAG and fine-tuning?
RAG (retrieval-augmented generation) looks up relevant passages in your documents when a question arrives and adds them to the prompt. Fine-tuning trains a copy of a base model on example inputs and outputs so it responds in a new way. RAG changes what the model can see, and fine-tuning changes how it behaves.
With RAG, documents are split into chunks, turned into vectors (embeddings) and stored in a search index. At question time, the system passes the closest chunks to the model. Embedding is cheap: OpenAI’s text-embedding-3-small costs $0.02 per million tokens. To change what the system knows, you replace or re-index documents, with no retraining.
Fine-tuning changes the model’s weights. OpenAI recommends it when a model “is producing inconsistent results with incorrect formatting,” and says this approach maximizes consistency of behavior. Microsoft bills training on the tokens in your training file times the number of epochs, meaning passes over the data.
Prompting comes before both: OpenAI calls prompt engineering “typically the best place to start”.
Can you still fine-tune GPT or Claude models in 2026?
Only in limited ways. OpenAI’s self-serve fine-tuning is closed to new users, and the only Claude model on Amazon Bedrock’s fine-tuning list is past end of life in its one fine-tuning Region. Gemini on Google Vertex AI and open-weight models remain open routes.
OpenAI says it “is winding down the fine-tuning platform”, and it notified developers on 7 May 2026. Since 2 July 2026, organizations that haven’t run inference on a fine-tuned model in the past 60 days can’t create jobs, and from 6 January 2027 active existing customers can’t either.
Anthropic announced fine-tuning for Claude 3 Haiku in July 2024, for text data with contexts up to 32K tokens. Bedrock’s fine-tuning table still lists Claude 3 Haiku in us-west-2 as its only Anthropic model. But Bedrock’s model lifecycle page shows the model went Legacy in that Region on 10 March 2026, which stopped new fine-tuning jobs on it, and reached end of life there on 10 September 2026.
After end of life, requests to a model fail unless you have a private arrangement with the provider for continued access, and migration doesn’t happen automatically. Bedrock’s own pages disagree on this model, so ask Anthropic or AWS what your account can fine-tune before you plan around a current Claude model.
The same limit applies to every vendor, because a fine-tune is tied to its base model. On Bedrock, a base model in Legacy state takes no new fine-tuning jobs, and OpenAI’s fine-tuned models stay available for inference only until their base models are deprecated. A fine-tune built on a closed model has an expiry date you don’t control.
Vertex AI supports supervised fine-tuning of Gemini models, including Gemini 3.5 Flash and 2.5 Pro, and Google calls it a good option “when you have a well-defined task with available labeled data”. For open-weight models you host, LoRA is a low-cost method. On GPT-3 175B it cut trainable parameters 10,000 times and GPU memory 3 times compared with full fine-tuning.
Do you need either one, or is a long prompt enough?
For a small, stable set of documents, a long prompt is often enough. Anthropic says that if your knowledge base is under 200,000 tokens (about 500 pages), you can include all of it in the prompt. It adds that prompt caching can cut latency by more than 2x and cost by up to 90%.
Caching makes this cheap. On Claude Haiku 5.5, a cache hit costs a tenth of normal input: $0.01 against $0.10 per million tokens for prompts up to 100,000 tokens. Above that line every rate is five times higher. The line counts cached tokens too, so a cached prompt over 100,000 tokens pays the upper tier on every request.
Context windows have grown as well: Claude 4.6 and later models, except Haiku 5.5, include a 1 million token window at standard pricing. Two limits still keep RAG useful. Anthropic’s 200,000-token guidance dates from September 2024, and it adds that a growing knowledge base needs “a more scalable solution”. Models also get worse at using a fact that sits in the middle of a long context.
One team on r/LLMDevs removed its RAG pipeline after a long-context model shipped and put most of it back two weeks later. That is a single story, but it fits the research.
When is RAG the right choice?
RAG fits when the model is missing knowledge, the documents are too large for one prompt, or the content changes often. OpenAI’s guide says to work on context when the model lacks knowledge that wasn’t in its training set.
Typical cases are a support assistant over a help center, search across internal policies and contracts, or product questions over a catalog that changes every week. In each, the facts move faster than a training cycle would.
For adding facts, the research favors RAG: Ovadia et al. found it consistently outperformed unsupervised fine-tuning for knowledge injection. Our RAG development page gives the same order: retrieval first, then “add fine-tuning later if you need the model to reason differently, not just access different data”.
Retrieval quality is where the published numbers are clearest. Standard RAG loses context when it splits and encodes documents. In Anthropic’s tests, contextual embeddings cut top-20 retrieval failures by 35%, adding contextual BM25 keyword search cut them by 49%, and adding reranking cut them by 67%.
When you get a RAG quote, check that hybrid search and reranking are in scope, since they produced the larger gains. For the database side of retrieval, see our Laravel vector search postmortem.
When is fine-tuning the right choice?
Fine-tuning fits when the model has the knowledge it needs but responds inconsistently: wrong format, uneven behavior, or a narrow task it gets wrong too often.
It pays off on tasks like Anthropic’s moderation test, where a fine-tuned Claude 3 Haiku raised classification accuracy from 81.5% to 99.6% and cut tokens per query by 85%. Shorter prompts are a stated benefit, because a fine-tuned model needs fewer examples and less context in each request.
Plan for the data work. OpenAI suggests starting with 50 or more examples, evaluating, then growing the set, and says quality matters more than quantity. Microsoft warns that repeat training runs, sampling, grading and evaluation all add to the cost of the final model.
Combining the two can stack the gains. In a Microsoft study on agriculture questions, fine-tuning added over 6 percentage points of accuracy, and RAG added another 5 points on top. A sensible split is RAG for the facts and fine-tuning for the format, as long as your chosen model can still be fine-tuned.
Which approach hallucinates less?
Neither one removes hallucination. Grounding the model in retrieved text reduces it, while fine-tuning on new facts can make it worse.
Anthropic says its hallucination-reduction techniques, grounding included, “significantly reduce hallucinations” but “don’t eliminate them entirely”. In 2024 tests, a Stanford study found that leading RAG-based legal research tools each hallucinated between 17% and 33% of the time.
Fine-tuning carries its own risk. Gekhman et al. found that as a model learns examples containing new knowledge, its tendency to hallucinate rises linearly. Ovadia et al. found that LLMs struggle to learn new facts through unsupervised fine-tuning in the first place. Long context fails more quietly: the fact is in the prompt, but the model misses it because it sits in the middle.
What does each option cost to run? A worked example
Monthly token costs for all three options are small next to the build. For how tokens are counted and billed, see AI tokens explained. Prices are published rates. The inputs are assumptions, not measurements:
- 1,000 questions a day for 30 days, so 30,000 questions a month
- A knowledge base of 150,000 tokens, under Anthropic’s 200,000-token line but over Haiku 5.5’s 100,000-token price line
- Each question: 300 tokens of user text plus 500 tokens of instructions, so 800 input tokens
- Each answer: 400 tokens
- RAG retrieves 5 chunks of 400 tokens, adding 2,000 tokens per question
- Fine-tuning set: 1,000 examples of 800 tokens, trained for 2 epochs, the count used in Microsoft’s example
Option A: the whole knowledge base in a cached prompt (Claude Haiku 5.5). Each request carries 150,800 tokens, so it pays the over-100,000 rates. Reading the cached 150,000 tokens 30,000 times is 4,500 million tokens at $0.05 per million, or $225. Uncached input adds $12 and output adds $30, for about $267 a month. Each time the 5-minute cache goes cold, rewriting it costs about $0.09.
Option B: RAG on the same model. Input grows to 2,800 tokens per question, well under 100,000, so the lower rates apply: 84 million input tokens at $0.10 per million is $8.40, and output is $6. Embedding the questions costs $0.18, and embedding the corpus once costs under a cent. That comes to about $14.58 a month in tokens. Add $100 to $400 a month for a vector database and hosting, the range our cost guide gives for an advanced RAG chatbot, for about $115 to $415 all in.
Option C: fine-tuned gpt-4.1-mini, no retrieval. Training on 1.6 million tokens (800,000 tokens × 2 epochs) at $5 per million is $8 per run. Inference at the fine-tuned rates of $0.80 input and $3.20 output per million comes to $19.20 plus $38.40, or about $58 a month. The same traffic on base gpt-4.1-mini would cost $28.80, so fine-tuned inference is twice the base rate. Only active existing OpenAI fine-tuning customers can do this, and only until 6 January 2027.
RAG has the smallest token bill, but the vector database and hosting make up most of its running cost. Option C is the weakest fit for document questions, because the model still won’t reliably know your facts. On Azure’s Standard and Global Standard deployments, a fine-tuned model also carries an hourly hosting fee on top of tokens, and Microsoft’s monthly example bills 720 hosting hours. GPU-hosted deployments bill while running, even with no traffic.
On Haiku 5.5, the architecture moves the token bill most: Option A’s tokens cost about 18 times Option B’s. Model choice matters too. At Claude Haiku 4.5 prices, Option A would cost about $534 a month, roughly twice as much, and the same RAG setup would cost about $144 in tokens, about ten times as much.
All in, running costs here land between about $58 (Option C, on gpt-4.1-mini) and $415 a month. A basic document Q&A RAG build is $25,000 to $40,000 in our AI development cost guide. Option A sits inside RAG’s all-in range and skips the vector database and the retrieval build. Fit, freshness and upkeep should drive the choice, not token price.
When is each option the wrong choice?
Fine-tuning is the wrong tool for adding or updating facts. RAG beat it in research, models learn new facts poorly this way and hallucinate more as they do, and every content change means another training run, billed by tokens times epochs. Use RAG, where an update means swapping the index.
Fine-tuning a closed model is a risky bet for a product you plan to run for years. OpenAI’s platform is closed to new users and inactive customers and ends new jobs for everyone on 6 January 2027, and a closed-model fine-tune lasts only as long as its base model. If you need fine-tuning, look at Gemini on Vertex AI or an open-weight model trained with LoRA.
RAG is more than you need for a small, stable document set: under about 200,000 tokens, a cached prompt is simpler and needs no index. Long context is the wrong fit for a large or changing corpus, where mid-prompt facts get missed and Anthropic says a growing knowledge base needs a more scalable solution.
RAG won’t fix a format or behavior problem. OpenAI places inconsistent formatting on the model side, not the context side. Try stricter prompts first, then fine-tune on a model that still supports it.
RAG is also the wrong promise if the use case needs zero errors: leading RAG-based legal tools hallucinated 17% to 33% of the time in 2024 tests. Plan human review where a wrong answer is costly, and keep validating after launch.
RAG vs fine-tuning vs long context: side by side
| Attribute | Long prompt | RAG | Fine-tuning |
|---|---|---|---|
| Fixes | Missing knowledge, small corpus | Missing knowledge, large corpus | Inconsistent format or behavior |
| Updating knowledge | Edit the prompt | Re-index documents | New training run, tokens × epochs |
| Teaching facts | Yes, in the prompt | Beat fine-tuning in research | Poor, raises hallucination |
| Hallucination risk | Misses mid-context facts | Reduced, not zero | Rises with new facts |
| Running cost driver | Cached input tokens; higher tier past 100K on Haiku 5.5 | Retrieved tokens plus vector DB | Higher per-token rate (OpenAI) or hourly hosting fee (Azure) |
| Closed-model access in 2026 | 1M tokens at standard pricing on Claude 4.6+ (not Haiku 5.5) | Works with any model | OpenAI closing; Claude 3 Haiku past end of life in us-west-2; Gemini open |
| Lifecycle risk | Switch models freely | Switch models, keep the index | Ends with its base model |
| Data needed | Your documents | Your documents | 50+ curated examples to start |
Frequently asked questions
Can fine-tuning teach a model new knowledge from my documents?
Not reliably. Ovadia et al. found that LLMs struggle to learn new facts through unsupervised fine-tuning, and Gekhman et al. found that as a model learns new facts through fine-tuning, its tendency to hallucinate rises. If the goal is answering questions from your documents, use RAG or a long prompt instead. A RAG index updates without retraining.
RAG vs fine-tuning vs prompt engineering: which should come first?
Prompt engineering. OpenAI calls it “typically the best place to start”. If the model is missing knowledge and your material is under about 200,000 tokens, put that material in the prompt. Move to RAG when the material grows past that point or changes often, and consider fine-tuning only for format or behavior problems that prompts can’t fix.
Is LoRA an alternative to RAG or to fine-tuning?
LoRA is a way to fine-tune, so it competes with full fine-tuning, not with RAG. It trains a small set of added parameters instead of the whole model. On GPT-3 175B it cut trainable parameters 10,000 times and GPU memory 3 times. That suits open-weight models you host yourself, and the result can still sit behind a RAG pipeline.
With 1M-token context windows, is RAG dead?
No. Claude 4.6 and later models, except Haiku 5.5, offer a 1 million token window at standard pricing, but models still miss facts placed in the middle of long contexts, and Anthropic says a growing knowledge base needs a more scalable solution. In a single report, one team on r/LLMDevs removed its RAG pipeline after a long-context model shipped, then put most of it back two weeks later.
How do you evaluate a RAG or fine-tuned system?
Test before launch and keep testing after. Barnett et al. concluded that validating a RAG system is only feasible in operation, so log real questions and check answers against sources. For retrieval, track how often the right passage is missing from the top results, as Anthropic did with its top-20 failure rate. For fine-tuning, OpenAI suggests starting with 50+ examples, evaluating, then growing the set.
If your project needs accurate answers from documents that grow or change, that is the problem our RAG development team works on.
