The Nairobi Neocloud: A Builder's Guide to Inference

The photograph is easy: a dark room, a row of expensive machines, a blue light reflected in a polished floor. The difficult part arrives at 10:17 on a Tuesday, when one customer submits a long document, another opens a voice session, and a third sends a thousand small requests at once.

A neocloud earns its name in that queue. Its product is reliable useful computation delivered at an economic price. Owning accelerators is only one input.

This is a practical design argument for a Nairobi inference business, grounded in serving research and current systems. It is not a procurement quotation; exact hardware prices, model licenses, and hosting terms must be checked when buying.

Begin with the customer promise

Choose an initial workload before choosing an engine. Interactive chat, batch document processing, voice, image understanding, and long-running coding agents have different latency and memory needs. A cloud that promises every model to every customer on day one will struggle to optimize anything.

Define time to first token, inter-token latency, completed-request success, and an end-to-end task metric. Specify percentile targets rather than averages. An impressive average can hide a terrible experience for the customer who waits longest.

I would begin with two service classes: a latency-sensitive endpoint for a carefully selected model and a lower-cost batch endpoint with explicit completion windows. This is a proposed starting architecture, not a claim about a current operator.

Understand the two phases

Prefill processes the input; decode generates output incrementally. Long prompts can interfere with short interactive requests if scheduling ignores their different demands. DistServe studies separating these phases to improve goodput: useful throughput that meets service objectives. Disaggregation is therefore a candidate architecture, not a requirement for every small deployment.

Start simple and measure. On a small cluster, transfers between prefill and decode workers may cost more than the separation saves. At larger scale, network fabric, cache movement, and workload isolation become central design questions.

Track input length and output length separately. A single “tokens per second” number does not describe a mixed workload. Chargeback and admission control should reflect the resources requests actually consume.

Memory is a product constraint

Model weights must fit; then request state must fit. The PagedAttention paper introduced a memory-management approach that reduces waste in KV-cache allocation and supports more efficient sharing. It underpins the vLLM lineage and remains an important conceptual foundation.

SGLang's research explores efficient structured language-model execution, including reuse of shared prefixes. Prefix caching can be valuable when many requests share a substantial instruction or document prefix. It is less valuable when prompts are unrelated or privacy constraints require isolation.

For a conventional transformer, an approximate uncompressed KV-memory estimate is: two times layer count times KV-head count times head dimension times cached tokens times bytes per element. Multiply by concurrent sequences, while accounting for shared prefixes. Architecture changes, sliding attention, quantization, and cache policies change the actual footprint. Measure the deployed model rather than trusting a generic spreadsheet.

Pick an engine through experiments

Before benchmarking, do a worked capacity estimate. Consider an explicitly hypothetical dense model with 32 layers, eight KV heads, head dimension 128, and a two-byte KV representation. Each cached token needs approximately 2 × 32 × 8 × 128 × 2 = 131,072 bytes, or 128 KiB. An 8,192-token sequence therefore needs about 1 GiB of uncompressed KV state; 32 such sequences need about 32 GiB before other overheads. This is arithmetic for the stated imaginary architecture, not a measurement of a named model.

Now add weights and runtime memory. An illustrative eight-billion-parameter model represented at two bytes per parameter occupies roughly 16 GB for raw weights, before allocator, kernels, activations, and other overhead. Decimal GB and binary GiB are different units. Long context and concurrency can consume the remaining memory much faster than a sales slide suggests. Quantized weights do not automatically mean the cache uses the same precision.

The capacity question is consequently joint: which model, precision, context length, concurrency, and service objective fit together? A configuration that loads successfully can still be unusable under real traffic.

Evaluate maintained versions of vLLM, SGLang, and TensorRT-LLM against your exact model and hardware. These are relevant enterprise and open-source options; no single project is permanently best across every workload.

Use the real tokenizer, representative prompt lengths, realistic concurrency, and a fixed quality test set. Compare precision settings and quantization against the uncompressed baseline. A cheaper response that corrupts financial extraction is expensive.

Speculative decoding is another candidate: a draft process proposes tokens and a verifier checks them. Its value depends on acceptance rate and overhead. Do not assume a published speedup transfers to every model pair or traffic pattern.

Pin versions. Record kernels, driver, model revision, tokenizer, quantization recipe, and configuration. An unexplained performance improvement is not a reproducible business advantage.

Choose the shape of parallelism

Data parallelism replicates a model across workers and routes different requests to each replica. It is operationally attractive when a full model fits on one worker and demand needs more independent capacity. Replicas also make rolling updates and failure isolation easier to reason about.

Tensor parallelism splits operations across accelerators. It helps fit larger models and can improve performance, but introduces frequent communication. The interconnect becomes part of the model-serving design. Two configurations with the same total accelerator memory can behave very differently if one has a much slower connection between devices.

Pipeline parallelism assigns different layers or stages to different workers. It can help distribute a model that does not fit conveniently in one place, but scheduling and idle pipeline bubbles matter. Combining pipeline and tensor parallelism adds flexibility and operational complexity.

Mixture-of-experts models introduce expert parallelism: different experts can be placed on different devices, with tokens routed to the appropriate experts. A small active-parameter count does not mean the total weights, memory footprint, or communication cost is small. Test expert imbalance and all-to-all traffic before promising a particular price.

These descriptions are general architecture guidance. Use the chosen engine's current support matrix before selecting a topology, and benchmark the real rack-level network. Avoid turning a feature list into a design before measuring whether it helps your service.

Start with a bounded configuration

For an initial single-worker pilot, my proposed configuration is one model revision, one compatible precision, an explicit 8,192-token maximum context if the model and engine support it, bounded output, a conservative concurrency limit, and no automatic download of unreviewed remote model code. Treat these numbers as starting assumptions to test, not universally optimal settings. Increase concurrency only while the measured latency objective and memory margin remain healthy.

Keep the public endpoint behind a gateway; the inference worker belongs on a private network. Separate interactive and batch queues even if they initially share hardware. Give cancellation a real path to the worker so that a customer who closes a request does not keep paying for abandoned generation indefinitely.

Document the settings using the exact flags and configuration fields for the pinned engine release. Copying a command from an old article can silently change behavior as defaults evolve. The article's durable recommendation is the experiment and its constraints, not an eternal launch command.

Route requests by supported model, tenant quota, queue delay, and relevant cache affinity. A cache-aware router can improve reuse, but it should not trap requests behind an overloaded worker. Keep a fallback policy and test the trade-off under bursty demand.

The control plane must survive a bad afternoon

Put authentication, tenant quotas, request limits, and billing outside the model server. Use admission control before memory exhaustion. Bound input size, output size, and concurrent long-context sessions. Return an explicit overload response rather than letting every customer time out together.

Keep a request ledger with IDs that survive retries. Monitor queue time, execution time, cache usage, accelerator utilization, errors, and cancellations. Separate sensitive prompt content from operational telemetry; logs should not become an accidental customer-data warehouse.

Roll out new models through canaries with automatic rollback on quality or latency regression. Maintain tested recovery paths for a dead worker, failed disk, broken network link, and unavailable upstream dependency. The customer's task does not care that the accelerator was technically healthy.

Autoscaling needs a realistic cold-start budget. Capacity is not ready when a machine boots; weights must be present, the worker must initialize, kernels may need preparation, and readiness checks must pass. Keep warm capacity for the service objective you sell. A promise of instant elasticity can become very expensive when model loading is slow.

Cache isolation deserves an explicit decision. Shared public instruction prefixes and private customer documents have different confidentiality requirements. Determine which namespaces may share entries, how eviction works, and whether cache timing or logs could expose tenant activity. If the isolation guarantee is uncertain, prefer separation over an attractive hit-rate graph.

Model supply-chain controls matter too. Record the source and license, verify downloaded artifacts, review execution requirements, and restrict the server's network permissions. An inference service should not acquire broad infrastructure authority merely because a model repository requested a convenience feature.

Calculate the business honestly

Estimate cost per successful task as total operating cost divided by successfully completed tasks that meet the service objective. Include power, cooling, networking, storage, staff, financing, depreciation, spares, and idle capacity. Peak accelerator throughput does not pay a salary when demand is absent.

For Nairobi, compare local latency, customer-data requirements, renewable electricity contracts, connectivity, and realistic utilization against renting capacity elsewhere. Local ownership may provide strategic value, but that value does not excuse a broken unit economy.

An illustrative monthly model is: required revenue equals fixed monthly costs plus metered operating costs plus a reserve for failures and replacement. Break-even price per billable unit equals that total divided by realistic billable units. Use a conservative demand scenario as well as a full-utilization scenario. If revenue only works when every accelerator is busy all the time, the business has not priced its idle risk.

Define the billable unit unambiguously. Input tokens, output tokens, cached input, images, audio, and reserved capacity may require different accounting. Record model and tokenizer version; reconcile the gateway's request ledger with worker usage. State how retries, cancellations, and failed requests are billed. A customer's trust should survive their first disputed invoice.

A rollout sequence that earns the next investment

In the first stage, replay representative traffic offline. Establish quality and latency baselines with one pinned configuration. Publish the assumptions internally, including what the service cannot yet handle. The exit condition is reproducible performance, not an attractive dashboard.

In the second stage, run a limited customer pilot with bounded traffic and staffed incident response. Measure the work customers actually finish, inspect billing, and exercise restore procedures. Keep the pilot small enough that a failure produces useful learning rather than a company-threatening outage.

In the third stage, add a second independent worker or failure domain and prove failover. Test overloaded queues, broken dependencies, and an interrupted model rollout. Introduce automated scaling only after the warm-up and routing behavior are understood.

In the fourth stage, expand the model catalog or infrastructure only when customer demand justifies the added complexity. Consider disaggregated prefill and decode, larger parallel groups, or specialized hardware when a measured bottleneck makes the choice rational. A research technique becomes an operational advantage after it improves your actual workload.

This sequence is deliberately adaptable. A regulated customer, a batch-only service, and a consumer voice product may need different milestones. The common requirement is that each stage leave behind evidence strong enough to support the next commitment.

The strongest first customer may need a dependable local workflow rather than the largest model in the world. Serve that workflow well. Build evidence. Expand when the queue, the balance sheet, and the customers agree.

A neocloud is a promise about completed work. The machines are how you keep it.

Editorial note: This essay expresses the founder’s ideas, developed with AI-assisted research and writing and reviewed against the linked sources on 29 October 2026. Convictions, proposals, and forecasts are identified in the text; scenes framed as imagined are illustrative.

Cover photograph / illustration: International Gemini Observatory/NOIRLab/NSF/AURA/Manuel Paredes. CC BY 4.0. Original source. Illustrative image: Data Center Fish Eye View (noirlab-racks-155).tiff. Credit: International Gemini Observatory/NOIRLab/NSF/AURA/Manuel Paredes; CC BY 4.0. This image does not document the research or proposed facility described.