Skip to main content
OpenAI logo
OpenAILanguage · Open weights

gpt-oss-120b

OpenAI’s open-weight mixture-of-experts reasoning model: 117B total parameters with 5.1B active per token, MXFP4 expert weights and the Harmony conversation format.

On 1 × NVIDIA RTX PRO 6000 Blackwell Server Edition (96 GB), gpt-oss-120b measured 171.04 tokens/s at concurrency 1 and 1,985.18 tokens/s aggregate at concurrency 64 in a 10-minute sustained test (1,024 input / 256 output tokens, measured 2026-10-14).

Context131,072Active5.1BParams117BWeightsMXFP4Apache 2.0Measured on RTX PRO 6000
View benchmarksModel card

From the model card

Specifications

Published by the model author. Reviewed 2026-10-11.

Publisher model card
Architecture
Mixture of experts (MoE)
Total parameters
117B
Active parameters
5.1B per token
Weight format
MXFP4 (expert weights)
Published context
131,072 tokens
Input → output
Text → text
Conversation format
Harmony
License
Apache 2.0

Measured by EcoHash

How it scales on one RTX PRO 6000

Closed-loop sustained load with 256 output tokens fixed. Concurrency is the level tested, not a capacity limit.

1 × NVIDIA RTX PRO 6000 Blackwell Server EditionvLLM 0.29.0MXFP4 expert weightsAtlanta, Georgia, US
Aggregate output throughputtokens/s vs concurrent requests
Aggregate output throughput. 1,024 in / 256 out: 1 → 171 tokens/s, 2 → 253 tokens/s, 4 → 409 tokens/s, 8 → 648 tokens/s, 16 → 984 tokens/s, 32 → 1,419 tokens/s, 64 → 1,993 tokens/s. 8,192 in / 256 out: 1 → 140 tokens/s, 2 → 190 tokens/s, 4 → 259 tokens/s, 8 → 335 tokens/s, 16 → 406 tokens/s, 32 → 471 tokens/s, 64 → 486 tokens/s05001,0001,5002,00012481632641,024 in / 256 out8,192 in / 256 out
1,024 in / 256 out8,192 in / 256 outx axis: concurrent requests (log₂)
Time per output token, P50ms vs concurrent requests
Time per output token, P50. 1,024 in / 256 out: 1 → 5.62 ms, 2 → 7.54 ms, 4 → 9.22 ms, 8 → 11.35 ms, 16 → 14.54 ms, 32 → 19.99 ms, 64 → 28.95 ms. 8,192 in / 256 out: 1 → 5.81 ms, 2 → 8.54 ms, 4 → 11.56 ms, 8 → 19.15 ms, 16 → 34.21 ms, 32 → 62.43 ms, 64 → 78.92 ms02040608012481632641,024 in / 256 out8,192 in / 256 out
1,024 in / 256 out8,192 in / 256 outsustained load · 2026-10-14
1,024-token prompts: 11.7× the single-request throughput at 64 concurrent. Aggregate output rose from 171 to 1,993 tokens/s between concurrency 1 and 64, while median time per output token grew 5.2× and P95 time to first token reached 1.24 s.
8,192-token prompts: 3.5× the single-request throughput at 64 concurrent. Aggregate output rose from 140 to 486 tokens/s between concurrency 1 and 64, while median time per output token grew 13.6× and P95 time to first token reached 24.88 s.
Stable for 10 minutes at 64 concurrent. A 602-second soak (4,672 requests) measured 1,985 tokens/s, within 0.4% of the 123-second run at the same level, with 0 failed requests.

Test results at a glance

Pick the column closest to your workload. Each column pairs throughput and latency from one run; higher total throughput trades off latency.

Metric
Single request
1,024 in / 256 out
concurrency 1
Sustained concurrent load
1,024 in / 256 out
concurrency 64 · 10-min soak
Long input · single request
8,192 in / 256 out
concurrency 1
Long input · concurrent load
8,192 in / 256 out
concurrency 16
met recorded internal targets
Output throughput(tokens/s)171.041,985.18139.61406.17
TTFT P50 / P95(ms)61.68 / 64.81838.8 / 895.7349.89 / 351.611,354.31 / 2,558.41
TPOT P50 / P95(ms per token)5.62 / 5.6329.17 / 30.965.81 / 5.8234.21 / 37.66
End-to-end P50 / P95(s)1.5 / 1.58.25 / 8.381.83 / 1.8310.07 / 11.27
Request success rate100%100%100%100%
Measured requests814,67266192
Test duration(min)21022

TTFT is the wait for the first output token. TPOT is the average time per subsequent token. P50 / P95 are request percentiles. Recorded concurrency is the level tested, not a service SLA or a physical capacity limit. Rounds are reported separately, never averaged. “Met recorded internal targets” repeats the pass recorded with that run; it is not a service commitment.

All 19 measured rows1,024 and 8,192-token inputs · fixed rounds and sustained load · 8,684 requests

Measured workload profiles

Each row pairs throughput and latency from one run. Aggregate throughput counts output tokens from all requests, including time spent processing the input. P50 / P95 are request percentiles; TPOT is the average time per output token after the first.

Tokens in / outConcurrentRunOutput tokens/sReq/sTTFT P50 / P95 (ms)TPOT P50 / P95 (ms/token)End-to-end P50 / P95 (s)Requests
1,024 / 25612026-10-14 · Sustained load
171.04
0.6761.68 / 64.815.62 / 5.631.5 / 1.581
1,024 / 25622026-10-14 · Sustained load
253.07
0.99114.37 / 139.597.54 / 7.662.03 / 2.06120
1,024 / 25642026-10-14 · Sustained load
409.26
1.6191.26 / 216.449.22 / 9.792.5 / 2.61192
1,024 / 25682026-10-14 · Sustained load
647.56
2.53253.6 / 358.4311.35 / 12.23.18 / 3.35304
1,024 / 256162026-10-14 · Sustained load
983.68
3.84490.13 / 612.7914.54 / 15.684.15 / 4.32464
1,024 / 256322026-10-14 · Sustained load
1,419.23
5.54675.12 / 892.4619.99 / 21.575.78 / 5.97672
1,024 / 256642026-10-14 · Sustained load
1,993.07
7.79841.93 / 1,236.0428.95 / 30.778.22 / 8.5960
1,024 / 256642026-10-14 · Sustained load · 602 s soak
1,985.18
7.75838.8 / 895.729.17 / 30.968.25 / 8.384,672
1,024 / 25612026-10-11 · Fixed rounds
170.76
0.6761.73 / 62.375.63 / 5.631.5 / 1.524
1,024 / 25642026-10-11 · Fixed rounds
394.44
1.54195.52 / 236.159.45 / 9.842.6 / 2.6424
Reliability · lowest success rate 100% · failures 0 · 8,684 measured requests across 19 rowsFixed-length outputs: P50 and P95 can be nearly equal. Not representative of variable-length requests.

Measured by EcoHash

Memory & storage

Resource values use the saved test configuration (2026-10-11). Workload rows have their own dates and sequence limits. These are recorded values, not minimum requirements.

Fits on one 96 GB GPU, no CPU offloading87.9 GB peak of 96 GB

87.9 GB of GPU memory at peak on a 96 GB GPU, including the KV cache reserved by the engine. 8.1 GB was not used at peak.

GPU memory · peak workload
87.9 GB
Including KV cache reserved by the engine
Host RAM · peak
6.8 GB
Process memory, excludes page cache
Weight storage
65.2 GB
On-disk storage for the served checkpoint
Cache footprint (downloads + runtime)
65.3 GB
Separate from the served weights

Host RAM is the summed process memory and excludes the filesystem page cache. Cache footprint is separate from the served weights and can include downloaded checkpoints that were not served.

Source & use

License & links

Where this model comes from and what you may do with it.

Apache 2.0Commercial use permitted

Commercial use is permitted under Apache 2.0, subject to its notice and other requirements. Preserve the Harmony format required by this model.

Publisher
OpenAI
Model ID
gpt-oss-120b
License
Apache 2.0
Model card / source
OpenAI on Hugging Face
Weight revision tested
b5c939de8f754692c1647ca79fbf85e8c1e70f8a

Specifications reviewed 2026-10-11. Measurements are EcoHash’s own, recorded with their full methodology below.

Measured by EcoHash

Methodology & provenance

How each number on this page was produced, with the raw archives that back it.

Run provenance2 campaigns · 3 raw archives
Sustained load2026-10-14

15 rows · 8,588 measured requests · server sequence limit 64. Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL.

SHA256 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628 · 88e9e5ae5287c75ba12753f277a1107bce318714e99da4595509ef3b865578e3

Fixed rounds2026-10-11

4 rows · 96 measured requests · server sequence limit 4. Local streaming HTTP, native token lengths; 3 rounds × 8 requests and 4 warmups per profile. Throughput uses summed measured round durations. Fixed 256-token outputs with ignore_eos=true; prefix caching off.

SHA256 d793a3de97528ed53afb662577d11f0fd57988e8d7f782cc4262e4484f4a818c

Every row’s recorded conditions
  • 1,024 in / 256 out · concurrency 1

    2026-10-11 · Fixed rounds · 36.0 measured seconds · server sequence limit 4

    Local streaming HTTP, native token lengths; 3 rounds × 8 requests and 4 warmups per profile. Throughput uses summed measured round durations. Fixed 256-token outputs with ignore_eos=true; prefix caching off. This run did not establish a maximum serving capacity. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: d793a3de97528ed53afb662577d11f0fd57988e8d7f782cc4262e4484f4a818c

  • 1,024 in / 256 out · concurrency 4

    2026-10-11 · Fixed rounds · 15.6 measured seconds · server sequence limit 4

    Local streaming HTTP, native token lengths; 3 rounds × 8 requests and 4 warmups per profile. Throughput uses summed measured round durations. Fixed 256-token outputs with ignore_eos=true; prefix caching off. This run did not establish a maximum serving capacity. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: d793a3de97528ed53afb662577d11f0fd57988e8d7f782cc4262e4484f4a818c

  • 8,192 in / 256 out · concurrency 1

    2026-10-11 · Fixed rounds · 44.0 measured seconds · server sequence limit 4

    Local streaming HTTP, native token lengths; 3 rounds × 8 requests and 4 warmups per profile. Throughput uses summed measured round durations. Fixed 256-token outputs with ignore_eos=true; prefix caching off. This run did not establish a maximum serving capacity. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: d793a3de97528ed53afb662577d11f0fd57988e8d7f782cc4262e4484f4a818c

  • 8,192 in / 256 out · concurrency 4

    2026-10-11 · Fixed rounds · 23.9 measured seconds · server sequence limit 4

    Local streaming HTTP, native token lengths; 3 rounds × 8 requests and 4 warmups per profile. Throughput uses summed measured round durations. Fixed 256-token outputs with ignore_eos=true; prefix caching off. This run did not establish a maximum serving capacity. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: d793a3de97528ed53afb662577d11f0fd57988e8d7f782cc4262e4484f4a818c

  • 1,024 in / 256 out · concurrency 1

    2026-10-14 · Sustained load · 121.2 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 2 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 25 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 1,024 in / 256 out · concurrency 2

    2026-10-14 · Sustained load · 121.4 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 2 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 25 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 1,024 in / 256 out · concurrency 4

    2026-10-14 · Sustained load · 120.1 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 2 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 25 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 1,024 in / 256 out · concurrency 8

    2026-10-14 · Sustained load · 120.2 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 2 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 25 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 1,024 in / 256 out · concurrency 16

    2026-10-14 · Sustained load · 120.8 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 2 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 25 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 1,024 in / 256 out · concurrency 32

    2026-10-14 · Sustained load · 121.2 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 2 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 25 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 1,024 in / 256 out · concurrency 64

    2026-10-14 · Sustained load · 123.3 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 2 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 25 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 8,192 in / 256 out · concurrency 1

    2026-10-14 · Sustained load · 121.0 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 5 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 40 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 8,192 in / 256 out · concurrency 64

    2026-10-14 · Sustained load · 140.8 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 5 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 40 s; success ≥ 99%. This profile did not meet those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 1,024 in / 256 out · concurrency 64

    2026-10-14 · Sustained load · 602.5 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 2 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 25 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 266c58a3131e45fe20eeb1fb3b7ebe7923b76a4b0a08dfd8c024f04b602b8628

  • 8,192 in / 256 out · concurrency 2

    2026-10-14 · Sustained load · 121.2 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 5 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 40 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 88e9e5ae5287c75ba12753f277a1107bce318714e99da4595509ef3b865578e3

  • 8,192 in / 256 out · concurrency 4

    2026-10-14 · Sustained load · 122.6 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 5 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 40 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 88e9e5ae5287c75ba12753f277a1107bce318714e99da4595509ef3b865578e3

  • 8,192 in / 256 out · concurrency 8

    2026-10-14 · Sustained load · 122.3 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 5 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 40 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 88e9e5ae5287c75ba12753f277a1107bce318714e99da4595509ef3b865578e3

  • 8,192 in / 256 out · concurrency 16

    2026-10-14 · Sustained load · 121.0 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 5 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 40 s; success ≥ 99%. This profile met those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 88e9e5ae5287c75ba12753f277a1107bce318714e99da4595509ef3b865578e3

  • 8,192 in / 256 out · concurrency 32

    2026-10-14 · Sustained load · 121.7 measured seconds · server sequence limit 64

    Synthetic native-token inputs; closed-loop client. Internal provisional latency targets. Inter-chunk gaps are not token-level ITL. OOM requires server-log review. Internal comparison targets: TTFT P95 ≤ 5 s; TPOT P95 ≤ 80 ms; end-to-end P95 ≤ 40 s; success ≥ 99%. This profile did not meet those targets. These are not a service SLA. Measurement: local HTTP, excluding public internet and gateway latency. TTFT measures request send to first non-empty output; TPOT = (last output time - first output time) / (actual output tokens - 1). P50/P95 are computed across individual successful requests in this profile; warmups excluded.

    Raw archive SHA256: 88e9e5ae5287c75ba12753f277a1107bce318714e99da4595509ef3b865578e3

Recorded test configuration & full methodologyrevision 5 · updated 2026-10-14
Hardware SKU
1 × NVIDIA RTX PRO 6000 Blackwell Server Edition; 96 GB VRAM (97,887 MiB reported)
Deployment region
Atlanta, Georgia, US
GPU power limit
600 W
NVIDIA driver
580.105.08
CUDA version
13.0
Engine & version
vLLM 0.29.0; PyTorch 2.13.0+cu130; Transformers 5.16.1
Container image digest
vllm/vllm-openai@sha256:082ca6f035279109041ffd3fe0695cb568b29bc580b35c4f297a66a08b216c1b
Weight revision
b5c939de8f754692c1647ca79fbf85e8c1e70f8a
Precision / quantization
MXFP4 expert weights; BF16 compute; Marlin MXFP4 kernel
CPU offloading
None
Test date
2026-10-11
Warmup runs
4 per profile (16 total)
Measured repeats
3 rounds × 8 requests per profile
Sample count
24 measured requests per profile; 96 across four profiles; plus 16 warmups and 1 smoke check
Input tokens
1,024 baseline; 8,192 secondary profile
Output tokens
256 fixed
Reasoning setting
Synthetic completions; functional chat smoke: reasoning_effort=low
Test context limit (tokens)
9,216
KV cache
gpu_memory_utilization=0.85; prefix caching off
Tested concurrency levels
1 and 4
Seed
20260920
Test notes
Workload: synthetic native-token inputs of 1,024 or 8,192 tokens; 256 fixed output tokens; temperature 0; ignore_eos=true. The workload table contains all four 2026-10-11 profiles (96 measured requests) and all 2026-10-14 capacity profiles (8588 measured requests). Local HTTP; public internet and gateway latency excluded. Each row records its own concurrency, sample count, duration and server sequence limit. The original flat performance and resource measurements below refer to 2026-10-11; the newer latency and throughput results are in the workload table. Latency definitions: TTFT is request send to first non-empty output. TPOT is (last output time - first output time) / (actual output tokens - 1), excluding first-token wait. P50/P95 are percentiles across individual successful requests within each profile; warmups excluded. Chunk intervals are not exact per-token ITL. Capacity: 2026-10-14 used server max_num_seqs=64, with closed-loop concurrency scans and a 10-minute short-input soak. Internal latency targets and passing status are recorded per profile; these are not a service SLA or a physical maximum. Request failures were zero; startup allocation OOM warnings recovered before service readiness and are not counted as failed inference requests. Quality: not evaluated. One functional chat smoke check passed (expected answer 323); its reasoning setting applies to that check. GPU memory: workload peak 83,864 MiB = 87.9378 GB (121 samples inside the workload window, startup and uncertain boundary samples excluded), includes KV-cache reservation. Full-lifecycle peak including initialization and profiling: 97,236 MiB = 101.9593 GB, within the 97,887 MiB reported capacity. Host RAM: peak summed process PSS 6.7750 GB (benchmark client + server), excludes filesystem page cache; full-lifecycle PSS peak 9.2647 GB. Container cgroup peak including page cache: 103.0775 GB. 277 telemetry samples, max interval 1.471 s. Storage: served shard set 65.2489 GB (model.safetensors.index.json); total HF cache 130.5257 GB; runtime cache 0.0273 GB. Additional HF download bytes outside the served shard set: 65.2768 GB; with runtime cache: 65.3041 GB. This extra cache footprint is not part of the minimum served-weight storage. 1 GB = 1e9 bytes. Not measured separately: idle VRAM, cold and warm start, output storage. Startup to readiness 170.04 s includes download and compile; it is not an isolated cold-start measurement. Raw archive SHA256: d793a3de97528ed53afb662577d11f0fd57988e8d7f782cc4262e4484f4a818c

Updated 2026-10-14 · revision 5

More model profiles

Specifications & test evidence

All models
Qwen · Vision-languageQwen3.8-27BSpecifications, measurements & methodologyAutoTrust AI · LanguageJEV-27BSpecifications, measurements & methodologyLightricks · Video + audioLTX-2.5Specifications, real outputs & license