DeepSeek official channel DeepSeek V4.1 Flash review: comparison baseline, 15 invalid requests at single concurrency

· AIHubMix · 12 min read · Provider Benchmark

DeepSeek official channel DeepSeek V4.1 Flash review: comparison baseline, 15 invalid requests at single concurrency

In the 2026-11-03/04 APB Full run, the DeepSeek official channel is the answer-quality baseline. Its four scores are Q 42.45, F 94.00, P 86.14 and R 100, with an overall of 73.76, only 0.37 from Alibaba Cloud: not distinguishable. Fifteen single-concurrency speed requests in window 1 were invalid.

On 2026-11-03/04, DeepSeek V4.1 Flash 1017 was tested on six providers through AIHubMix with the APB Full suite, using the same questions, parameters and request scripts for each. This APB Profile covers the DeepSeek official channel only: what each section measured and how it came out.

The numbers below are scores, not a ranking or a provider recommendation. The four scores, Q, F, P and R, come first; the overall is a summary index, not a certification, and it has no confidence interval. The scoring rules are defined in AIHubMix Provider Benchmark Standard v2.0.

Scores across six providers

Overall score by provider

Provider Q quality F function P speed K cache R stability Overall
ByteDance 42.05 74.98 84.37 93.83 100 66.37
Baidu 38.00 88.39 37.77 68.85 100 56.48
Alibaba Cloud 41.75 89.78 88.58 99.37 100 73.38
DeepSeek official channel (this post) 42.45 94.00 86.14 98.38 100 73.76
Fireworks 41.42 71.78 79.51 60.68 100 64.10
Pareto 44.58 93.42 57.69 27.58 100 68.72

Providers are listed in a fixed order, not a ranking. K is a sub-score of P, accounting for 20% of it, and is listed separately for reference. The overall score is computed with the composition rules of AIHubMix Provider Benchmark Standard, which also take in the comparison with the official channel and evidence completeness; it is not a direct weighting of the four scores in the table. It is a summary index with no confidence interval. The DeepSeek official channel and Alibaba Cloud differ by only 0.37 overall, which is not a distinguishable difference.

At a glance

  • Answer quality: Q 42.45, with BFCL 86.13%, HLE 15.06% and AA-LCR 53.20%. The other five providers' Q is compared against this one. HLE output hit the max_tokens ceiling 7 times, counted as wrong.
  • Function support: F 94.00; 137 of 147 counted checks passed. Another 27 checks that depend on exact token counts were judged invalid and left out of the denominator.
  • Response speed: P 86.14. In window 1, 15 of 30 single-concurrency requests were invalid (12 × 503, 2 at the deadline, 1 read timeout). With invalid requests counted as infinite, the median first body token is 1.87 s (1.72 s valid-only); 10-way 1.72 s, TPOT 2.84 ms.
  • Cache: K 98.38. All 18 hot requests at 8k and 32k hit, and no cold control read the cache.
  • Service stability: R 100. All 300 dedicated requests succeeded on the first attempt.

Test setup

Item Detail
Model DeepSeek V4.1 Flash 1017
Requested model ID deepseek-flash
Returned model name deepseek-flash
Access route Through AIHubMix's DeepSeek official provider route, not a direct connection to the official API
Parameters Fixed APB v2 request profile, temperature 1.0, max_tokens 16,384, 300 s timeout, no automatic retry
Test profile The five providers other than Pareto shared one fixed APB v2 profile for Q/P/K/R; F also checked its parameter contract
Time 2026-11-03 18:07 – 11-04 16:33; cache requests 11-04 22:10–22:27; all UTC+8. All six providers shared one Mac mini, one network and one account.
Samples Q 4,250 question-instances; F 180 checks; P 180 requests; R 300; K 30

Requests used the AIHubMix quick start route with a provider-specific model ID; the other providers' IDs are listed on the models page. The model's own parameters are described in DeepSeek's API documentation.

Q — answer quality: 42.45

Answer quality draws on three suites: 500 BFCL questions answered 3 times each, 250 HLE questions answered 10 times each, and 25 AA-LCR questions answered 10 times each. BFCL is scored by automatic rules; HLE and AA-LCR are judged right or wrong by the judge model deepseek-flash. All six providers received the same questions, and the three suites are pooled into one accuracy. Ten answers were removed because the question could not be scored, leaving 4,240 scored question-instances.

Suite Questions × rounds Correct Accuracy Service failures
BFCL tool calls, auto-scored 500 × 3 1,292/1,500 86.13% 0
HLE hard QA, judge-scored 250 × 10 375/2,490 15.06% 7
AA-LCR long-doc reasoning, judge-scored 25 × 10 133/250 53.20% 0
Total · 95% interval 40.08%–44.73% — 1,800/4,240 42.45% 7

Service failures (failed requests or truncated output) stay in the denominator as wrong answers: HLE output hit the max_tokens ceiling 7 times.

BFCL category Passed Pass rate
simple_python: one function, right call and arguments 334/360 92.78%
multiple: pick the right one of several functions 268/300 89.33%
parallel: one function called several times at once 271/300 90.33%
parallel_multiple: several functions at once 207/240 86.25%
irrelevance: no relevant function, so no call 113/150 75.33%
ahm_multi_turn: three-stage tool chain 99/150 66.00%

Each BFCL question was answered 3 times; the table sums the 3 rounds.

Compared with the other five

Q answer quality by task

Q difference from the official channel

Provider BFCL HLE AA-LCR Combined Vs official
ByteDance 85.20% 15.34% 49.20% 42.05% −0.40 (−1.46 to +0.66), not distinguishable
Baidu 86.40% 9.36% 32.80% 38.00% −4.46 (−5.78 to −3.16), below official
Alibaba Cloud 87.00% 13.29% 53.60% 41.75% −0.71 (−1.72 to +0.26), not distinguishable
DeepSeek official channel (this post) 86.13% 15.06% 53.20% 42.45% Baseline
Fireworks 85.53% 14.38% 46.00% 41.42% −1.04 (−2.14 to +0.05), not distinguishable
Pareto 85.27% 19.32% 52.00% 44.58% +2.12 (−0.02 to +4.18), not distinguishable

Vs official: the accuracy gap in percentage points, with the 95% interval in brackets; an interval that crosses 0 is recorded as not distinguishable. The HLE and AA-LCR judge, deepseek-flash, belongs to the same DeepSeek family as the tested model and was not calibrated, so its judgments may be lenient.

F — function support: 94.00

The function suite looks only at whether interface behaviour matches the protocol, not at whether the model answers well. Sixty cases in six groups ran once in each of three windows: 180 checks, judged against the format each provider documents. Checks that were not applicable, invalid or unobserved leave the denominator. DeepSeek official channel passed 137 of 147 counted checks and scored 94.00 on the group weights.

Case group Weight Passed Excluded Group score
Streaming and message protocol 20% 27/27 3 100.00
Tool interface 20% 30/30 — 100.00
Output constraints and parameters 20% 30/30 — 100.00
Context and state 15% 14/15 15 93.33
Errors and recovery 15% 24/30 — 80.00
Metering and cache protocol 10% 12/15 15 80.00
Total · weighted score — 137/147 33 94.00

The weighted score applies the group weights; it is not simply passed checks divided by 147.

Failed case Windows Reason
Malformed messages · errors and recovery 3/3 400 with an empty body, no structured error
Malformed stream_options · errors and recovery 3/3 400 with an empty body, no structured error
Reasoning-token metering · metering and cache protocol 3/3 No reasoning-token count returned
Isolation between requests · context and state 1/3 Step 1 did not reply only STORED, so isolation could not be checked

“Windows” is how many of the 3 windows failed the check.

Status Case Count Reason
Unobserved Streaming stop sequence 3 The request for this step was not sent after the previous step, so it could not be judged
Invalid 9 cases (listed below the table) 27 The pre-fixed native-count template did not match this channel's usage, so exact-count checks were invalid
Not applicable Explicit cache_control 3 Registered in advance as unsupported by this interface

These checks leave the denominator; they count as neither a pass nor a failure.

The 9 invalid cases: 10k long input, within context limit, 32k long input, 100k long input, over context, native input count, tool definitions counted in input, input count stable with cache, streaming vs non-streaming input.

The malformed-messages and malformed-stream-options error cases failed on several providers: they returned 400 with an empty body, so the client gets no structured error explanation. Reasoning-token metering behaved differently from provider to provider.

Compared with the other five

F function support

Provider Passed Failed Excluded Weighted score
ByteDance 124 50 6 74.98
Baidu 148 26 6 88.39
Alibaba Cloud 155 22 3 89.78
DeepSeek official channel (this post) 137 10 33 94.00
Fireworks 122 55 3 71.78
Pareto 29 4 147 93.42

P — response speed: 86.14

Each request carried about 10,000 input tokens and asked for about 1,100 output tokens of body text. In each of three windows, 30 requests ran at single concurrency and 30 at 10-way concurrency: 180 requests in all.

Item Detail
First body token (TTFA) Median 1.87 s single (valid-only 1.72 s), 1.72 s at 10-way
Generation speed (TPOT) 2.84 ms/token single, 2.67 ms/token at 10-way
Effective throughput 1,546.0 token/s at 10-way
Valid requests 75/90 single (window 1: 15/30), 90/90 at 10-way; invalid: 12 returned 503 service_unavailable_error, 2 hit the deadline unfinished, 1 read timeout
Sub-scores TTFA 51.65, TPOT 100.00, throughput 100.00; TTFA and TPOT take the lower tier, weighted 35% / 35% / 30%: response score 83.08
P score Response score 83.08 × 80% + cache K 98.38 × 20% = 86.14

One valid request waited 106.45 s for its first body token. Invalid requests are scored as infinite, so the medians in the table can be higher than valid-only medians. TPOT is estimated from streamed chunks, not measured per token on the hardware. Throughput is valid output tokens at 10-way concurrency divided by batch wall-clock time.

Compared with the other five

P response speed

Provider Valid requests TTFA single TTFA 10-way TPOT (slower) Throughput tok/s Response
ByteDance 180/180 2.12 s 2.10 s 3.43 ms 1,250 82.00
Baidu 118/180 2.26 s ∞ 50.00 ms 1,329 30.00
Alibaba Cloud 180/180 1.34 s 1.35 s 4.69 ms 1,314 85.88
DeepSeek official channel (this post) 165/180 1.87 s 1.72 s 2.84 ms 1,546 83.08
Fireworks 179/180 1.63 s 1.64 s 3.29 ms 1,505 84.22
Pareto 110/180 5.85 s 6.13 s 4.63 ms 519 65.22

All six shared one machine, network and account. The three windows were scheduled provider by provider, so collection periods do not fully overlap; small differences should not be over-read. ∞ means more than half the requests were invalid, so the median counts as infinite.

K — cache: 98.38

Each group sends five requests. The first carries a fixed prefix of about 8k or 32k tokens and seeds the cache. Requests 2–4 keep the prefix and change only the final question, at gaps of 5 s, 30 s and 120 s. The fifth uses a new prefix as a cold control. Three windows × two lengths make 30 requests. K is the cache tokens read by hot requests divided by the tokens that should be reusable under the rules, with the two lengths weighted equally; K is 20% of P.

Window 1 · seed 2 · +5 s 3 · +30 s 4 · +120 s 5 · cold
8k prefix · hot hits 9/9 · read 70,272 of 72,261 reusable · 97.25 — — — — —
Window 1 0 7,808 7,808 7,808 0
Window 2 0 7,808 7,808 7,808 0
Window 3 0 7,808 7,808 7,808 0
32k prefix · hot hits 9/9 · read 286,848 of 288,270 reusable · 99.51 — — — — —
Window 1 0 31,872 31,872 31,872 0
Window 2 0 31,872 31,872 31,872 0
Window 3 0 31,872 31,872 31,872 0
K (8k and 32k weighted equally) — — — — 98.38

Values are cache-read tokens reported by the server; “Not returned” means the field was absent from the response.

All nine 8k hot requests hit, each reading 7,808 tokens; all nine 32k hot requests hit, each reading 31,872 tokens. Cold requests read no cache, so hot and cold separate cleanly.

Compared with the other five

K cache hit rate

Provider 8k hits 8k 32k hits 32k K
ByteDance 9/9 98.84 8/9 88.81 93.83
Baidu 5/9 49.60 8/9 88.10 68.85
Alibaba Cloud 9/9 98.83 9/9 99.90 99.37
DeepSeek official channel (this post) 9/9 97.25 9/9 99.51 98.38
Fireworks 3/9 32.81 8/9 88.54 60.68
Pareto 2/9 21.89 3/9 33.27 27.58

R — service stability: 100

The 300 requests fall into four types: short text, long input, tool calls and multi-turn history. They ran at single concurrency across three windows. Only whether the first attempt returned a valid, complete response is checked; answers are not graded and nothing is retried. Failures in the answer-quality and speed sections are counted in Q and P, not again in R.

Request type First-attempt success
Short text 120/120
10k long input 90/90
Tool calls 60/60
Multi-turn history 30/30
Total · 95% interval 98.78%–100% 300/300

All 300 requests returned a valid, complete response on the first attempt. R = 100 describes these 300 dedicated requests only; it does not mean the whole chain had no failures. The lower bound of the 95% interval is 98.78%: 300 successes out of 300 can only say the failure rate is likely below 1.22%.

Compared with the other five

Provider First-attempt success 95% interval
ByteDance 300/300 98.78%–100%
Baidu 300/300 98.78%–100%
Alibaba Cloud 300/300 98.78%–100%
DeepSeek official channel (this post) 300/300 98.78%–100%
Fireworks 300/300 98.78%–100%
Pareto 300/300 98.78%–100%

R covers only these 300 dedicated requests; failures in the answer-quality and speed sections are recorded in Q and P and are not counted again in R.

FAQ

Is this a direct connection to DeepSeek's API?
No. The requests go through AIHubMix's DeepSeek official provider route; they are not sent directly to the official API.

Why is the DeepSeek official channel the Q baseline?
The other five providers' answer-quality results are expressed as gaps from this channel, with 95% intervals. Being the baseline does not make it a ranking.

What happened in the first single-concurrency speed window?
Fifteen of 30 requests were invalid: 12 returned 503 service_unavailable_error, 2 reached the deadline unfinished and 1 hit a read timeout. Invalid requests count as infinite, so the median is 1.87 s; with valid requests only it is 1.72 s.

What are the 27 invalid function checks?
Nine cases that depend on exact token counts, such as long inputs, the context limit, native input counts and input consistency with cache, each in three windows. The test side's pre-fixed native-count template did not match the usage this channel reported, so these checks were left out of the denominator rather than scored as failures.

Does R 100 guarantee the route never fails?
No. R covers only the 300 dedicated stability requests, and failures in the answer-quality and speed runs are recorded in Q and P. With 300 of 300 successful, the lower bound of the 95% interval is 98.78%, so the failure rate is likely below 1.22%, not proven to be zero.

How does the cache test separate hot reads from cold?
Each group seeds an 8k or 32k prefix, repeats it at 5, 30 and 120 seconds with a new final question, then sends a new prefix as a cold control. Both lengths hit 9 of 9 hot requests, and every cold control read zero cached tokens.

Keep reading: the DeepSeek V4.1 Flash series

Sources

  1. DeepSeek V4.1 Flash 1017 model page (AIHubMix)
  2. AIHubMix Provider Benchmark Standard v2.0 (AIHubMix docs)
  3. Quick start (AIHubMix docs)
  4. Models (AIHubMix)
  5. DeepSeek API documentation (DeepSeek)

More from the blog