How Should A Startup Switch Between API Models And Open Weights

Mistral Large 4 weights, due in late October 2026, need about 1.05 terabytes of GPU memory at 8 bit precision, so switching from the API is an engineering decision. Mistral released the model on October 6, 2026 with 1.05 trillion total parameters, 49 billion active per token, a one million token context window, and prices of $1.36 per million input tokens and $4.18 per million output tokens. A startup that treats the weights release as a price cut will be disappointed. A startup that builds a provider neutral model layer now will treat it as a configuration change.

The arithmetic surprises people. Only 4.7 percent of the parameters are active for any token, which makes compute cheap. The router, though, can send any token to any expert, so every expert must sit in memory. At 8 bit that is 1.05 terabytes of weights, more than the 640 gigabytes in an eight GPU H100 node and just under the roughly 1,128 gigabytes in an eight GPU H200 node, with about 78 gigabytes left for the cache that long prompts consume. Quantized to 4 bit, the weights shrink to about 525 gigabytes and fit on the H100 node, with quality loss that nobody has yet measured on this model.

What does self hosting actually cost compared with the API?

Self hosting beats the API price only when a cluster stays busy around the clock. A vendor of rented GPUs lists an eight GPU H200 node at $44.08 per hour on demand. At the 3 to 1 input to output mix many applications run, the blended Mistral price is $2.065 per million tokens, so that node must process about 5,900 tokens per second, every second, to match it. The same vendor’s guidance for the smaller Mistral Large 3 reports about 2,500 tokens per second on that hardware, and its own cost table disagrees with its throughput math by a factor of three. I trust the division over the table.

A redundant production pair at spot prices runs roughly $464,000 per year before any engineer is paid. That buys about 225 billion tokens a year at API rates. Below that volume the API wins on price. Above it, or when a customer’s security addendum forbids third party subprocessors, or when the product depends on a fine tuned variant, the private path earns its cost.

What should the model layer contain?

The layer needs six parts. A normalizer converts every application call into one internal schema, so the product never touches a vendor SDK. A policy router picks a target per task class, such as contract extraction or code review, from configuration that carries a fallback order, a cost cap, a latency budget, and a data class rule. Adapters absorb vendor quirks in tool calls, streaming, and usage fields. An evaluation harness holds frozen test sets drawn from de identified production traffic and runs offline, shadow, and canary comparisons. An observability store records tokens measured by an independent tokenizer, because different tokenizers count the same text differently and a price per million tokens means little until measured on your own traffic. A weights supply chain downloads once, verifies checksums, mirrors to private storage, pins revisions, and loads only safe tensor files.

Prompts must be versioned per model. I have reviewed enough vendor transitions to say that the prompt is the part that breaks first, and it breaks quietly: tool call arguments drift, JSON closes late, refusals change. A regression suite that runs on every target change catches the drift before a customer does.

Which benchmark numbers deserve trust?

Few of them yet. Mistral reports 61.7 percent on DeepSWE v1.1, 93 percent on Cybench, and a human evaluation score of 3.74 out of 5.0 that ranks second of five models. The first independent broad index puts the model at 48.05 percent, 32nd of 44. A legal agent benchmark ranks it sixth of 75 with a 15.83 percent pass rate. Independent testing found effective context nearer 512,000 tokens than one million. Capability is uneven by task, which is the best argument for routing by task class instead of choosing one default model.

What does the license decide?

The license decides whether the self hosted path exists for a commercial product. Mistral Large 2, released July 24, 2024, shipped under a research license that barred commercial use and required a separate commercial license for self deployment. Mistral Large 3, released December 2, 2025, shipped under Apache 2.0. Large 4 has no published license. In thirty years of advising technology companies I have read the license before the benchmark table every time, because the table describes what the model can do and the license describes what you may do with it.

The European Union Artificial Intelligence Act adds a second variable. Regulation (EU) 2024/1689 gives open source model providers exemptions from some documentation duties, but the exemptions do not cover models presumed to present systemic risk above 10 to the 25th floating point operations of training compute, and Mistral has not published the training compute for Large 4. A startup that fine tunes the model stays below the Commission’s one third of original compute threshold for becoming a provider, so ordinary fine tuning does not shift the provider obligations to it.

What should happen before and after release day?

Before the weights arrive, wire the gateway to the API and to Mistral Large 3, which is Apache 2.0 and already served by vLLM and by cloud marketplaces. That gives a rehearsal of the full switch on a model of the same family. Build the frozen test sets. Draft the license checklist: field of use, user or revenue thresholds, fine tuning and redistribution rights, output ownership, acceptable use terms, termination, and indemnity. On release day, read the license first, mirror and verify the weights, test 8 bit and 4 bit variants against the test sets, shadow five percent of traffic, then canary. Keep a second closed API in every critical fallback list, because a cold s