Technical guides

Cloud GPUs vs. dedicated servers: Compare costs at the same SLO

Cloud GPUs and dedicated servers should be compared by monthly total cost for useful throughput that meets the same SLO, not by a price list per hour. Put fixed and variable cost, standby capacity, and failure response into the same formula to make the decision conditions visible.

Precision computing modules layered like clouds

Design scenario: cost of running two replicas for customer-support RAG

Each replica handles 12 active streams, for 24 in normal operation and 12 after one node fails. Amounts are inputs for a comparison formula; the operating target is an always-on response service averaging 730 hours per month.

Service time
24-hour operation / budgeted at a monthly average of 730 hours
Target capacity
24 streams normally, 12 after node failure / TTFT p95 at or below 2.5 seconds and ITL p95 at or below 80ms
Billing unit
H is pool-wide ready time, distinct from the summed hours of individual GPUs
Comparison scope
Compute, storage, transfer, operations, and failure readiness / VAT excluded

Match useful throughput and standby capacity first

If one environment abandons the SLO at peak while another includes spare equipment, this is not a price comparison. Both candidates must satisfy the same acceptance criteria: TTFT and ITL targets for 24 active streams, and 12 streams retained during a node failure. Confirm this criterion by validated useful throughput, not GPU count.

In this article, H is not the time requests keep a GPU busy; it is the time both replicas are ready together (pool-hours). The 300h and 500h rows are possible only for an operating plan that serves requests during those windows and holds the rest until the next window. An always-on 24×7 response SLA requires H=730h, and the 300h and 500h rows must not be read as always-on plans. Capacity kept warm for peaks may look idle, but it is an SLO cost.

Tools such as the AWS Pricing Calculator help review monthly and annual estimates by configuration; they are not performance-test results.

Split cloud by pool ready time and dedicated by two-node monthly fixed cost

The formula prevents omitted items. Cloud = H×Rc(pool) + S + E + O. H is the time two replicas are warm concurrently; Rc(pool) is the hourly configuration coefficient for the entire pool; S is storage and backup; E is egress and inter-region transfer; and O is operations. Standby or failure-ready replicas are already in Rc(pool), so do not add R separately. Dedicated = F(2 nodes) + P + C + N + W + S + O. F is the monthly cost of two nodes; P is electricity; C is rack space and cooling; N is networking; W is warranty, spares, and replacement.

Cost categories without double counting
ItemCloudDedicatedActual quote or cost
Two-replica computeH×Rc(pool)F(2 nodes)Input required
Data and networkS+EN+SInput required
Facilities and supportCheck whether included in configurationP+C+WInput required
OperationsOOHours by role required

Do not include lease payments and depreciation in F at the same time. For owned equipment, calculate monthly cost from acquisition cost, residual value, and depreciation period; for leased equipment, use monthly payment and contract incidentals.

If each replica uses one GPU, 730 pool-hours equal 1,460 summed GPU-hours. Because the formula’s rate is for the whole pool, do not multiply H by two again. Seven hundred thirty hours is a monthly budget average of 365 days × 24 ÷ 12. Actual monthly billing uses 28-, 30-, or 31-day runtime plus start and stop times. Model downloads and warmup billed during ready time also belong in H.

Separate pool ready time from fixed items in the worked calculation

The following are comparison inputs, not provider prices. Set Rc(pool) for the two-replica pool to KRW 3,000 per hour. Set cloud fixed items to S KRW 120,000 + E KRW 60,000 + O KRW 240,000 = KRW 420,000 per month. Set dedicated cost at KRW 1,620,000: two-node F KRW 900,000 + P KRW 180,000 + C KRW 180,000 + N KRW 90,000 + W KRW 120,000 + S KRW 60,000 + O KRW 90,000. Fill the actual quote or internal cost for each item in the final column. Cloud cost is then 3,000×H+420,000.

Calculation by service window
H: warm pool timeCloud calculationDedicated calculationOperating feasibility
300hKRW 1,320,000KRW 1,620,000Requests outside the service window wait for the next window
500hKRW 1,920,000KRW 1,620,000Requests outside the service window wait for the next window
730hKRW 2,610,000KRW 1,620,00024×7 warm; eligible for always-on response-SLA review

The break-even time is (1,620,000−420,000)÷3,000=400h. It holds only when both replica pools provide the same useful throughput and 12-stream failure condition, and the same policy allowing requests to wait outside service windows.

Monthly cost-input specification · VAT excluded, KRW
ItemCloud inputDedicated inputQuote or actual record
Compute / equipment FTwo-replica pool KRW 3,000/hour × HTwo-node monthly depreciation KRW 900,000Enter SKU, configuration, and contract
Electricity PIncluded in compute rateKRW 180,000Enter consumption and rate
Rack and cooling CIncluded in compute rateKRW 180,000Enter colocation contract
Egress E / networking NKRW 60,000KRW 90,000Enter transfer and overage
Warranty and spares WIncluded in compute support scopeKRW 120,000Enter replacement and support scope
Storage and backup SKRW 120,000KRW 60,000Enter retained volume and recovery cost
Operations O8 hours × KRW 30,000 = KRW 240,0003 hours × KRW 30,000 = KRW 90,000Enter monthly work hours
Total used in calculationH×3,000 + 420,000KRW 1,620,000Recalculate after quote confirmation

Three dedicated-operation hours are a stable-state input where patching and incident-response automation are in place; do not apply it unchanged to a new environment. If dedicated operations also require eight hours per month, total cost becomes KRW 1,770,000 and the break-even under the same formula becomes 450 hours. One-time adoption and migration cost and tax remain in a separate budget outside this recurring-month table.

The sensitivity table separates service windows from an always-on SLA

Change the most uncertain input one at a time. Actual quotes, discounts, electricity rates, transfer volume, and operating labor vary by organization and time, so fill the “input required” fields. Every value below is calculated as H×Rc(pool)+420,000.

Cloud pool-rate sensitivity by service window
H: warm pool timeRc=2,400Rc=3,000Rc=3,600SLA decision
300h1,140,0001,320,0001,500,000Not feasible for an always-on 24×7 SLA
500h1,620,0001,920,0002,220,000Not feasible for an always-on 24×7 SLA
730h2,172,0002,610,0003,048,000Always-warm condition

The 300h and 500h rows are not an operating recommendation that lower utilization is cheaper. They are sensitivities when service windows and a request-wait policy are acceptable. If always-on response is contractual, use the 730h row or a more stringent standby-capacity assumption.

Do not erase failure readiness with a discount

A cheaper single dedicated server does not meet this scenario if a failure leaves zero streams. Put the second server, parts and replacement lead time, or a burst path in another environment into both cost and operating procedures for retaining 12 streams. Cloud needs the same scrutiny for availability zones, capacity reservations, allocation failures, and model reload time. Because the calculation already includes two replicas, do not add a separate R for standby cost.

Availability-review questions
EventPrebuilt designValidation evidenceActual cost or quote
Node failureRemaining replica retains 12 streamsSLO from failure exerciseInput required
Peak surgeQueue and burst rulesLoad testInput required
Equipment replacementSpares and support agreementReplacement lead timeInput required

When a node stops, its streams end as failures; do not automatically send unfinished answers. Assign at most 12 concurrent streams to the remaining replica and route the excess to the queue or rejection policy. Use a five-second health-check interval, stop assigning new requests after two consecutive failures, and set confirmation of the reduced-capacity path within 30 seconds as the failure-test target. A single-node failure differs in scope from a region- or facility-wide failure; if the latter is required, add the cost of a replica elsewhere.

Do not mix financing method and asset cost

Calculate dedicated equipment cost F through either ownership or leasing. For ownership, divide acquisition cost less residual value by the internal depreciation period, then review cost of capital and tax treatment under finance policy. For leasing, separate monthly payments, upfront payment, renewal and termination terms, and included maintenance from the contract. Adding both monthly depreciation and monthly lease payment for the same equipment double-counts cost.

The same principle applies to cloud commitments. Differences among on-demand, reserved, and savings commitments belong in Rc and the ability to stop. Committed capacity that cannot be used is effectively fixed cost even when it appears variable. Record contract period, prepayment, capacity guarantee, and constraints on early changes in the comparison table.

How to record contract items
ChoiceMonthly cost inputRecord separately
OwnershipDepreciation-based FResidual value and replacement plan
LeaseContract-payment-based FExpiry and penalty terms
Cloud commitmentEffective Rc and minimum spendRegion and capacity guarantee

Read monthly cost in useful units of work too

A lower hourly cost can look favorable, but the comparison must also show how much valid work completes under the same SLO. If one environment keeps a long queue and causes requests to be cancelled, lower billed time does not mean less customer-support work. Record completed requests, processed input and output tokens, failures and retries, and p95 TTFT and ITL for the same period as the cost table.

Choose a denominator that fits the product, such as “monthly total cost ÷ valid completed requests” or “monthly total cost ÷ work that passes the quality bar.” The denominator must be the same in both environments. Do not inflate throughput with shorter outputs or a weaker quality criterion. If batch work uses spare time on dedicated equipment, state that work’s useful throughput and allocation rule as well.

Do not hide operating labor O as an estimate. Record time by role for patching, incident response, model deployment, security review, and provider-ticket handling, then include it in monthly cost. Automation may reduce O, but the cost of building automation and its owner also require review.

Refresh the monthly comparison table as an operating record

Each month, record actual H, useful throughput, p95 latency, storage and transfer, and on-call time beside the inputs. Reducing billed time is not a cost improvement if queue rejections rise. Conversely, if spare time on fixed equipment serves batch work, make explicit which service receives that value.

Write the conclusion conditionally. The 400h formula only has meaning when comparing the same two-replica pools, the same 12-stream failure capacity, and the same service window. For a required 24×7 response, use the 730h always-warm row. Use the 300h and 500h sensitivity rows only when requests outside service windows can wait for the next window. In every case, a current table completed with actual quotes and contract terms replaces the calculation inputs.

Sequence for reviewing cost at the same SLO
  1. Fix the SLO

    Define useful throughput and minimum capacity during failure

  2. Validate configuration

    Both candidates pass the same load matrix

  3. Break down cost

    Separate variable, fixed, and operating items

  4. Test sensitivity

    Calculate changes in service windows and pool rate

  5. Refresh monthly

    Review invoices and SLO together

References