Democratizing Custom LLMs inside SaaS

At low volume, calling the OpenAI API is the rational choice. A few thousand requests per day at a few cents per call is a rounding error on most SaaS operating budgets. The model is good, the integration is simple, and you can ship features in days instead of months.

The economics change at scale. A B2B SaaS product processing 500,000 inference requests per day at an average of 800 input tokens and 400 output tokens — a typical workload for document summarization or in-app search — is spending roughly $15,000 to $25,000 per month on API calls, depending on the model tier. At a million requests, you're approaching $40,000. That's not a feature cost — that's a line item that shows up in gross margin conversations with your board.

But cost isn't even the harder problem. Data exposure is.

The data liability SaaS companies can't sign away

When a user's data hits a third-party inference API, that data is processed on infrastructure you don't control. Even with a data processing agreement in place, the practical exposure is real:

• Inference logs exist. API providers retain request and response data for varying periods — for abuse detection, quality monitoring, and model improvement (unless explicitly opted out, and even then, retention for safety logging often persists).
• Your customers' customers are exposed. If your SaaS product serves enterprise clients, those clients have their own data governance policies. Many large enterprise procurement teams now explicitly ask whether user data is sent to third-party LLM providers during security reviews. A "yes" answer increasingly triggers extended review cycles or outright disqualification.
• You can't audit what you don't host. SOC 2 Type II and ISO 27001 audits require demonstrating control over data processing environments. When inference happens on someone else's GPUs, your audit evidence is a contract, not a control.

Self-hosting eliminates these problems structurally. Your data never leaves your VPC. There's nothing to negotiate, nothing to opt out of, and nothing to explain in a security questionnaire.

The VPC deployment architecture

The production stack we deploy for SaaS companies running their own models consists of four components behind a VPC boundary with no public ingress:

• Inference engine: vLLM running on GPU-attached instances (typically AWS g5.xlarge with A10G GPUs or GCP a2-highgpu-1g with A100s, depending on model size and latency requirements). vLLM handles continuous batching, PagedAttention for memory-efficient KV-cache management, and OpenAI-compatible API endpoints — so your application code doesn't change when you switch from a hosted API to self-hosted.
• Autoscaling layer: A Ray Serve deployment manages replica scaling based on request queue depth. Scaling from 2 to 8 replicas during a traffic spike takes roughly 90 seconds on pre-warmed instances. Cold starts are slower (4-6 minutes to load model weights), so we maintain a warm pool during business hours.
• Load balancer: An internal application load balancer distributes requests across replicas. Health checks verify that each replica can complete a test inference within 2 seconds — not just that the process is alive, but that the model is loaded and responsive.
• VPC boundary: On AWS, this runs inside a VPC with no internet gateway. On GCP, VPC Service Controls (VPC-SC) create a security perimeter that prevents data from leaving the project, even if an IAM policy is misconfigured. Model weights are pulled from a private artifact registry at boot time. No runtime traffic exits the boundary.

The application talks to this stack exactly as it would talk to an external API — POST a prompt, receive a completion. The only difference is that the endpoint is a private DNS name instead of api.openai.com.

Model selection: the 7B-13B sweet spot

The question every SaaS team asks first is: "Will an open-source model be as good as GPT-4?" The honest answer is no, not at general-purpose reasoning. But that's the wrong question.

The right question is: "Will an open-source model be good enough for our specific use case, given that we can fine-tune it on our own data?"

For most production SaaS workloads — document classification, entity extraction, in-app search, summarization, structured data generation — a 7B to 13B parameter model fine-tuned on domain-specific data outperforms a much larger general-purpose model that hasn't seen your vocabulary.

The practical options today:

• Llama 3 8B — Strong general-purpose baseline. Handles summarization and classification well. Fits comfortably on a single A10G GPU with 4-bit quantization, delivering 40-60 tokens per second at batch size 1.
• Mistral 7B — Slightly better at structured output generation and instruction following in our benchmarks. Similar hardware requirements to Llama 3 8B.
• Llama 3 70B (quantized to 4-bit) — For use cases that genuinely need stronger reasoning. Requires 2x A100 80GB or equivalent. We reach for this when the task involves multi-step logic over long documents, not for high-throughput classification.

The performance gap between a fine-tuned 8B model and GPT-4 on domain-specific tasks is typically 5-15% on internal benchmarks — and that gap closes further with good prompt engineering and task-specific evaluation sets. On generic benchmarks, the gap is larger. On your actual production data, it's usually small enough to be irrelevant.

LoRA fine-tuning for domain specificity

The mechanism that makes small models competitive on domain tasks is LoRA (Low-Rank Adaptation) fine-tuning, part of the PEFT (Parameter-Efficient Fine-Tuning) family. Instead of retraining the entire model — which would require hundreds of GPU-hours and a dedicated ML team — LoRA trains a small set of adapter weights that modify the model's behavior for your specific task.

The practical workflow:

1. Collect examples. 500 to 2,000 input-output pairs from your production data — real user queries and the correct responses. This is the most labor-intensive step, and it's mostly curation, not generation.
2. Format for training. Convert examples to the model's instruction format (typically a system prompt, user message, and expected completion).
3. Train. A LoRA fine-tune on 1,000 examples with a 7B model takes 2-4 hours on a single A100 GPU. The adapter weights are roughly 50-100 MB — trivial to store and version.
4. Evaluate. Run the fine-tuned model against a held-out evaluation set and compare to the base model and your current API provider. Measure the metrics that matter to your product — not BLEU scores, but task-specific accuracy, latency, and user-facing quality.
5. Deploy. vLLM supports loading LoRA adapters at runtime, so you can serve multiple fine-tuned variants from the same base model without duplicating GPU memory.

The entire cycle — from collecting examples to serving the fine-tuned model in production — typically takes one to two weeks of engineering time. Not months.

Cost arithmetic: when self-hosting breaks even

Here's a real comparison from a B2B document processing SaaS. The workload: summarizing uploaded contracts, extracting key terms, and generating clause-level risk assessments. Average request: 1,200 input tokens, 500 output tokens.

| Volume (requests/day) | GPT-4o API cost/month | Self-hosted (2x A10G) cost/month | Savings | |---|---|---|---| | 10,000 | ~$2,400 | ~$3,200 | -$800 (API wins) | | 50,000 | ~$12,000 | ~$3,200 | +$8,800 | | 200,000 | ~$48,000 | ~$6,400 (4x A10G) | +$41,600 | | 500,000 | ~$120,000 | ~$9,600 (6x A10G) | +$110,400 |

The break-even point sits around 25,000-30,000 requests per day. Below that, the operational overhead of managing your own infrastructure isn't worth the savings. Above that, the economics become lopsided — and they get more lopsided as you scale, because GPU costs scale sub-linearly (batching efficiency improves) while API costs scale linearly with volume.

The cost table tells half the story. The other half is what you get beyond cost savings: complete data isolation, no rate limits during traffic spikes, the ability to fine-tune on your own data, and a security posture that doesn't depend on a third party's data handling practices.

For SaaS companies processing more than 30,000 LLM requests per day and handling customer data with any sensitivity, self-hosting isn't a cost optimization. It's an architectural decision that removes an entire category of risk from the business. The economics just happen to make it easy to justify.