Cloud GPUs vs. dedicated servers: Compare costs at the same SLO
Cloud GPUs and dedicated servers should be compared by monthly total cost for useful throughput that meets the same SLO, not by a price list per hour. Put fixed and variable cost, standby capacity, and failure response into the same formula to make the decision conditions visible.

Design scenario: cost of running two replicas for customer-support RAG
Each replica handles 12 active streams, for 24 in normal operation and 12 after one node fails. Amounts are inputs for a comparison formula; the operating target is an always-on response service averaging 730 hours per month.
- Service time
- 24-hour operation / budgeted at a monthly average of 730 hours
- Target capacity
- 24 streams normally, 12 after node failure / TTFT p95 at or below 2.5 seconds and ITL p95 at or below 80ms
- Billing unit
- H is pool-wide ready time, distinct from the summed hours of individual GPUs
- Comparison scope
- Compute, storage, transfer, operations, and failure readiness / VAT excluded
Match useful throughput and standby capacity first
If one environment abandons the SLO at peak while another includes spare equipment, this is not a price comparison. Both candidates must satisfy the same acceptance criteria: TTFT and ITL targets for 24 active streams, and 12 streams retained during a node failure. Confirm this criterion by validated useful throughput, not GPU count.
In this article, H is not the time requests keep a GPU busy; it is the time both replicas are ready together (pool-hours). The 300h and 500h rows are possible only for an operating plan that serves requests during those windows and holds the rest until the next window. An always-on 24×7 response SLA requires H=730h, and the 300h and 500h rows must not be read as always-on plans. Capacity kept warm for peaks may look idle, but it is an SLO cost.
Tools such as the AWS Pricing Calculator help review monthly and annual estimates by configuration; they are not performance-test results.
Split cloud by pool ready time and dedicated by two-node monthly fixed cost
The formula prevents omitted items. Cloud = H×Rc(pool) + S + E + O. H is the time two replicas are warm concurrently; Rc(pool) is the hourly configuration coefficient for the entire pool; S is storage and backup; E is egress and inter-region transfer; and O is operations. Standby or failure-ready replicas are already in Rc(pool), so do not add R separately. Dedicated = F(2 nodes) + P + C + N + W + S + O. F is the monthly cost of two nodes; P is electricity; C is rack space and cooling; N is networking; W is warranty, spares, and replacement.
| Item | Cloud | Dedicated | Actual quote or cost |
|---|---|---|---|
| Two-replica compute | H×Rc(pool) | F(2 nodes) | Input required |
| Data and network | S+E | N+S | Input required |
| Facilities and support | Check whether included in configuration | P+C+W | Input required |
| Operations | O | O | Hours by role required |
Do not include lease payments and depreciation in F at the same time. For owned equipment, calculate monthly cost from acquisition cost, residual value, and depreciation period; for leased equipment, use monthly payment and contract incidentals.
If each replica uses one GPU, 730 pool-hours equal 1,460 summed GPU-hours. Because the formula’s rate is for the whole pool, do not multiply H by two again. Seven hundred thirty hours is a monthly budget average of 365 days × 24 ÷ 12. Actual monthly billing uses 28-, 30-, or 31-day runtime plus start and stop times. Model downloads and warmup billed during ready time also belong in H.
Separate pool ready time from fixed items in the worked calculation
The following are comparison inputs, not provider prices. Set Rc(pool) for the two-replica pool to KRW 3,000 per hour. Set cloud fixed items to S KRW 120,000 + E KRW 60,000 + O KRW 240,000 = KRW 420,000 per month. Set dedicated cost at KRW 1,620,000: two-node F KRW 900,000 + P KRW 180,000 + C KRW 180,000 + N KRW 90,000 + W KRW 120,000 + S KRW 60,000 + O KRW 90,000. Fill the actual quote or internal cost for each item in the final column. Cloud cost is then 3,000×H+420,000.
| H: warm pool time | Cloud calculation | Dedicated calculation | Operating feasibility |
|---|---|---|---|
| 300h | KRW 1,320,000 | KRW 1,620,000 | Requests outside the service window wait for the next window |
| 500h | KRW 1,920,000 | KRW 1,620,000 | Requests outside the service window wait for the next window |
| 730h | KRW 2,610,000 | KRW 1,620,000 | 24×7 warm; eligible for always-on response-SLA review |
The break-even time is (1,620,000−420,000)÷3,000=400h. It holds only when both replica pools provide the same useful throughput and 12-stream failure condition, and the same policy allowing requests to wait outside service windows.
| Item | Cloud input | Dedicated input | Quote or actual record |
|---|---|---|---|
| Compute / equipment F | Two-replica pool KRW 3,000/hour × H | Two-node monthly depreciation KRW 900,000 | Enter SKU, configuration, and contract |
| Electricity P | Included in compute rate | KRW 180,000 | Enter consumption and rate |
| Rack and cooling C | Included in compute rate | KRW 180,000 | Enter colocation contract |
| Egress E / networking N | KRW 60,000 | KRW 90,000 | Enter transfer and overage |
| Warranty and spares W | Included in compute support scope | KRW 120,000 | Enter replacement and support scope |
| Storage and backup S | KRW 120,000 | KRW 60,000 | Enter retained volume and recovery cost |
| Operations O | 8 hours × KRW 30,000 = KRW 240,000 | 3 hours × KRW 30,000 = KRW 90,000 | Enter monthly work hours |
| Total used in calculation | H×3,000 + 420,000 | KRW 1,620,000 | Recalculate after quote confirmation |
Three dedicated-operation hours are a stable-state input where patching and incident-response automation are in place; do not apply it unchanged to a new environment. If dedicated operations also require eight hours per month, total cost becomes KRW 1,770,000 and the break-even under the same formula becomes 450 hours. One-time adoption and migration cost and tax remain in a separate budget outside this recurring-month table.
The sensitivity table separates service windows from an always-on SLA
Change the most uncertain input one at a time. Actual quotes, discounts, electricity rates, transfer volume, and operating labor vary by organization and time, so fill the “input required” fields. Every value below is calculated as H×Rc(pool)+420,000.
| H: warm pool time | Rc=2,400 | Rc=3,000 | Rc=3,600 | SLA decision |
|---|---|---|---|---|
| 300h | 1,140,000 | 1,320,000 | 1,500,000 | Not feasible for an always-on 24×7 SLA |
| 500h | 1,620,000 | 1,920,000 | 2,220,000 | Not feasible for an always-on 24×7 SLA |
| 730h | 2,172,000 | 2,610,000 | 3,048,000 | Always-warm condition |
The 300h and 500h rows are not an operating recommendation that lower utilization is cheaper. They are sensitivities when service windows and a request-wait policy are acceptable. If always-on response is contractual, use the 730h row or a more stringent standby-capacity assumption.
Do not erase failure readiness with a discount
A cheaper single dedicated server does not meet this scenario if a failure leaves zero streams. Put the second server, parts and replacement lead time, or a burst path in another environment into both cost and operating procedures for retaining 12 streams. Cloud needs the same scrutiny for availability zones, capacity reservations, allocation failures, and model reload time. Because the calculation already includes two replicas, do not add a separate R for standby cost.
| Event | Prebuilt design | Validation evidence | Actual cost or quote |
|---|---|---|---|
| Node failure | Remaining replica retains 12 streams | SLO from failure exercise | Input required |
| Peak surge | Queue and burst rules | Load test | Input required |
| Equipment replacement | Spares and support agreement | Replacement lead time | Input required |
When a node stops, its streams end as failures; do not automatically send unfinished answers. Assign at most 12 concurrent streams to the remaining replica and route the excess to the queue or rejection policy. Use a five-second health-check interval, stop assigning new requests after two consecutive failures, and set confirmation of the reduced-capacity path within 30 seconds as the failure-test target. A single-node failure differs in scope from a region- or facility-wide failure; if the latter is required, add the cost of a replica elsewhere.
Do not mix financing method and asset cost
Calculate dedicated equipment cost F through either ownership or leasing. For ownership, divide acquisition cost less residual value by the internal depreciation period, then review cost of capital and tax treatment under finance policy. For leasing, separate monthly payments, upfront payment, renewal and termination terms, and included maintenance from the contract. Adding both monthly depreciation and monthly lease payment for the same equipment double-counts cost.
The same principle applies to cloud commitments. Differences among on-demand, reserved, and savings commitments belong in Rc and the ability to stop. Committed capacity that cannot be used is effectively fixed cost even when it appears variable. Record contract period, prepayment, capacity guarantee, and constraints on early changes in the comparison table.
| Choice | Monthly cost input | Record separately |
|---|---|---|
| Ownership | Depreciation-based F | Residual value and replacement plan |
| Lease | Contract-payment-based F | Expiry and penalty terms |
| Cloud commitment | Effective Rc and minimum spend | Region and capacity guarantee |
Read monthly cost in useful units of work too
A lower hourly cost can look favorable, but the comparison must also show how much valid work completes under the same SLO. If one environment keeps a long queue and causes requests to be cancelled, lower billed time does not mean less customer-support work. Record completed requests, processed input and output tokens, failures and retries, and p95 TTFT and ITL for the same period as the cost table.
Choose a denominator that fits the product, such as “monthly total cost ÷ valid completed requests” or “monthly total cost ÷ work that passes the quality bar.” The denominator must be the same in both environments. Do not inflate throughput with shorter outputs or a weaker quality criterion. If batch work uses spare time on dedicated equipment, state that work’s useful throughput and allocation rule as well.
Do not hide operating labor O as an estimate. Record time by role for patching, incident response, model deployment, security review, and provider-ticket handling, then include it in monthly cost. Automation may reduce O, but the cost of building automation and its owner also require review.
Refresh the monthly comparison table as an operating record
Each month, record actual H, useful throughput, p95 latency, storage and transfer, and on-call time beside the inputs. Reducing billed time is not a cost improvement if queue rejections rise. Conversely, if spare time on fixed equipment serves batch work, make explicit which service receives that value.
Write the conclusion conditionally. The 400h formula only has meaning when comparing the same two-replica pools, the same 12-stream failure capacity, and the same service window. For a required 24×7 response, use the 730h always-warm row. Use the 300h and 500h sensitivity rows only when requests outside service windows can wait for the next window. In every case, a current table completed with actual quotes and contract terms replaces the calculation inputs.
- Fix the SLO
Define useful throughput and minimum capacity during failure
- Validate configuration
Both candidates pass the same load matrix
- Break down cost
Separate variable, fixed, and operating items
- Test sensitivity
Calculate changes in service windows and pool rate
- Refresh monthly
Review invoices and SLO together