公开信号

I observed an inconsistency in the multimodal behavior of google/gemma-4-31B-it when served through the OpenAI-compatible API. The model successfully understands and summarizes spoken content when the audio is embedded inside an MP4 video. However, when I provide the exact same audio as a standalone MP3 file, the request fails with an error indicating that the model does not support audio input. Since both inputs contain identical speech, the only difference is the container format (MP4 vs. MP3), making the behavior appear inconsistent. Environment Model: google/gemma-4-31B-it Serving Framework: vLLM 0.24.0 API: OpenAI-compatible Chat Completions API Case 1: Video Input (Works) Request curl http://10.10.5.54:8701/v1/chat/completions \ -H…

DeepSeek model output sometimes contains DSML tool-call markup, but the API/client does not parse it into structured toolcalls. Instead, the whole assistant message is returned as plain text in message.content. Example returned content: Expected behavior: The DSML block should be parsed into a structured tool call, for example: Actual behavior: The DSML block is returned inside normal assistant text content, so downstream code receives it as a plain string and must manually parse it. Possible issue: The emitted DSML token appears degraded from the documented full-width format: to an ASCII double-pipe-like form: Also, the markup may be emitted without the newlines expected by the reference parser. This likely prevents the tool-call parser from…

(APIServer pid=143) Traceback (most recent call last): (APIServer pid=143) File "/root/vllm/.venv/bin/vllm", line 10, in <module (APIServer pid=143) sys.exit(main()) (APIServer pid=143) ^^^^^^ (APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 95, in main (APIServer pid=143) args.dispatchfunction(args) (APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 148, in cmd (APIServer pid=143) uvloop.run(runserver(args)) (APIServer pid=143) File "/root/vllm/.venv/lib/python3.12/site-packages/uvloop/init.py", line 96, in run (APIServer pid=143) return asyncio.run( (APIServer pid=143) ^^^^^^^^^^^^^^ (APIServer pid=143) File…

When i use tool calling, reasoning content was parsed uncorrectly, remain main content with <think token Plz Update like ke this PR (https://github.com/vllm-project/vllm/pull/45413)

Hi! Has anyone successfully served Mistral-Small-4-119B-2603 on 2×H200 with vLLM? I'm stuck at startup. What happens: The weights load fine to GPU (59 GiB per GPU), then it freezes right after this line: [fp8.py] Using FLASHINFERCUTLASS Fp8 MoE backend ... After that: no more logs, 0% GPU util, /health refused, no "Uvicorn running". rank-0 sits at 70% CPU (looks like it's JIT-compiling) while the other worker stays idle. Setup: - 2×H200 (NVLink), tensor-parallel-size 2 - Official vLLM recipe command (FLASHATTNMLA, reasoning-parser mistral, gpu-memory-utilization 0.8) - Tried vLLM 0.21 / 0.22 / 0.23, same behavior Already tried (no luck): TP=1 and TP=2, FlashInfer-CUTLASS and Triton MoE backends, --enforce-eager, --disable-custom-all-reduce,…

I'd really like to use this model as my primary model, but the looping issue prevents me from using it. I found a prompt which will make the model loop 99.9% of the time, regardless of quantization level. "Tell me the name of the painting where the clown is sitting on a chair with a look of disillusionment." Please fix this issue to make this model actually useful. The wasted potential of this model makes me sad.

!Screenshot2026-06-16at11-00-02 Hello, I'm a pro subscribe user and I found the rate limit quota too little. Anybody can solve this?

Hello, I‘m experiencing a persistent, unbreakable token truncation precisely at 1024 generated tokens (excluding prompt tokens). Even when I explicitly override both maxnewtokens and maxlength at all possible entry points, the model output is abruptly cut off mid-sentence once the generation sequence length reaches exactly 1024 tokens. Is there an undocumented private parameter (e.g., maxthinkingtokens, maxthinkinglength) or a specific sub-module configuration that I need to modify to unlock longer reasoning trajectories for this Omni-Thinking model? Any guidance on how to properly override this private 1024 constraint would be highly appreciated. Best regards, ww

Summary The chattemplate shipped with this model may be vulnerable to a multi-turn reasoning re-injection issue that can cause verbatim repetition loops during agentic / tool-calling use (e.g. OpenCode, llama.cpp --jinja, any harness that sends back reasoningcontent on prior assistant tool-call steps). It is harmless for single-turn chat — it only triggers in a multi-step tool-calling sequence — which is why it does not show up in standard single-turn benchmarks. Mechanism The template renders each prior assistant turn's thinking back into the prompt: The guard loop.index0 lastuseridx and message.toolcalls restricts this to the assistant tool-call steps that occur after the final user message — i.e. exactly the in-request agentic loop. Each new…

Affected models: google/gemma-4-26B-A4B-it and google/gemma-4-31B-it When continuing a partial assistant message with continuefinalmessage=true / addgenerationprompt=false, the majority of completions degenerate into a single repeated token or short phrase. Reproduces on two independent inference stacks: - self-hosted vLLM - 3rd party Gemma4 hosting provider Example request Observed Response (repeats until maxtokens) This makes continuefinalmessage unusable for production assistant-prefill / "continue the turn" flows.

Was running modelingstep3p5.py via huggingface, and I kept running into the error: In SafeTensors there seem to be 48 layers. I fixed this by updating the config.json; numhiddenlayers = 48 Is this a typo in the config.json?

My teammate is facing issue when he bumped the transformers to the version 5.0.0 or higher, the model's performance decreased significantly. Below is the comparison table for the humanitarian task, image only (canonical is the result I got previously. OLD env is the new experiment result on old transformers. NEW env is the result on new transformers). Both new results came from the exact same Notebook file (using the exact same code). metric, canonical, OLD env (transformers 4.47.1), NEW env (transformers 5.4.0), drift accuracy, 0.7654, 0.7613, 0.6691, -9.2 pts weightedf1, 0.7571, 0.7530,0.6815, -7.2 pts macrof1, 0.6275, (≈0.62), 0.5642, -6.3 pts

Hi Xiaomi MiMo team, I found a compatibility issue when serving XiaomiMiMo/MiMo-V2.5 with the recommended SGLang image and using OpenAI-compatible clients such as Roo Code. I do believe I have the latest version of the templates as I only downloaded the model after that warning message. The model works correctly when the system message content is a plain string, but crashes when system.content is provided as an OpenAI-style content-part array. Environment: Model: XiaomiMiMo/MiMo-V2.5 Serving image: lmsysorg/sglang:dev-mimo-v2.5 Backend: SGLang OpenAI-compatible /v1/chat/completions Command includes: --reasoning-parser mimo --tool-call-parser mimo --trust-remote-code --attention-backend fa3 Minimal repro: curl -s…

Today there was a new template added: additionalchattemplates/reranker.jinja Before, this model needed a custom template. Now one has been added, but it breaks existing templates. You get an error like: TypeError: expected str, bytes or os.PathLike object, not NoneType This is true with transformers 4.57.1. A workaround is to run sglang with this option: --revision a718125bb61a08e0248962cb02a58deb53ca2d27

I need to load GLM-5 on 2 x H200 nodes with 8 GPUs each. While I set tensor-parallel = 8 but when I set 'pipelineparallelsize' 2, I am receiving following error 'NotImplementedError: Pipeline parallelism is not supported for this model. Supported models implement the 'SupportsPP' interface. What is the other method to run GLM-5 on two nodes.

sglang serve --model-path /models Traceback (most recent call last): File "/usr/local/bin/sglang", line 6, in <module sys.exit(main()) ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/cli/main.py", line 40, in main serve(args, extraargv) File "/usr/local/lib/python3.12/dist-packages/sglang/cli/serve.py", line 116, in serve executeservecmd(parsedargs, remainingargv) File "/usr/local/lib/python3.12/dist-packages/sglang/multimodalgen/runtime/entrypoints/cli/serve.py", line 33, in executeservecmd serverargs = ServerArgs.fromcliargs(args, unknownargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/sglang/multimodalgen/runtime/serverargs.py", line 958, in fromcliargs return cls.fromdict(providedargs)…

Upon inferencing the model, the model produces random chinese characters from the document's image, even though the whole document is english. I have also specifically mentioned in the prompt as well to return the output in english character only. Does anyone has faced this issue, how to avoid this hallucination. please help on this issue !image

Audio handling seems to break for pretty much all use-cases that don't exclusively involve transcription, summarization or translation. For example, "Transcribe this audio" prompts seem to work well but the model points the audio inputs out as unusual with other prompts. Example code adapted from the official audio handling guide: Output (user prompt omitted): This behavior isn't a hallucination that only manifests when the model is asked about finding issues with the audio. It also occurs when trying to "voice-chat" with the model (sending user messages as audio), where the model has issues understanding the input and points out repeated phrases or letters (although it also seems to often be able to correctly respond to such voice-chat…

1. Qwen-Image pipeline concurrent requests causing an index out of bounds error Issue When two image generation requests run concurrently using the same preloaded pipeline instance for inference (numinferencesteps=50), both requests fail around step 26 with the following error: index 51 is out of bounds for dimension 0 with size 51 --- Suspected root cause Both requests are sharing the scheduler’s private timestep index. When each request runs 26 steps concurrently (26 × 2 = 52), the combined progression exceeds the scheduler’s timesteps size (50), resulting in an out-of-bounds access. --- Code Known workarounds Add a lock to serialize requests → Safe, but eliminates concurrency Batching requests → Requires request aggregation, increases latency…

!image use LLama-factory ,lora ,deepseek-r1-0528

I have tried 2 times on web demo. The output looks like "thoughts", but <Think is missing. According to the tech report, <Think or <think should be generated.