Discussions

• 1 You must be logged in to vote

  🙏

  请问一下在prefill计算时，ProcessKV和FlattenKV的作用分别是什么？

  sitabulaixizawaluduo asked Jul 17, 2024 in Q&A · Unanswered

  

  

  2
• 1 You must be logged in to vote

  🙏

  lmdeploy serve api_server 可以把两个模型部署到同一个端口吗？

  thomZ1 asked Jul 17, 2024 in Q&A · Answered

  

  

  2
• 1 You must be logged in to vote

  🙏

  How multimodal batch inference works?

  bonre asked Jul 16, 2024 in Q&A · Answered

  

  

  4
• 1 You must be logged in to vote

  🙏

  Q: Continuous Batching without Turbomind?

  awaiting response

  SinanAkkoyun asked Jul 15, 2024 in Q&A · Answered

  

  

  2
• 1 You must be logged in to vote

  💬

  Does lmdeploy support InternVL2?

  nickmitchko started Jul 11, 2024 in General

  

  

  2
• 1 You must be logged in to vote

  💬

  LMDeploy有相关的论文或者技术报告吗

  IcyFeather233 started Jul 9, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  🙏

  MPI related undefined reference errors while lmdeploy build from source

  anisingh1 asked Sep 28, 2023 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  KV Cache int8 cooperate with Flash-attention2

  yhyu13 asked Nov 15, 2023 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  What's the difference between Persistent Batch and continuous batching?

  luweigen asked Nov 20, 2023 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  离线推理 pipeline 模式支持连续批处理吗？

  shell-nlp asked Apr 4, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  lmdeploy中TP设置成2时，当模型不能完全加载到一张显卡中，会对模型进行分层并加载到两张GPU中吗？

  hello-gary-2022 asked Apr 23, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💬

  chatGLM支持

  Artist2001 started May 23, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  🙏

  我们显卡cuda是11.4，lmdeploy能支持么？

  colinlzh asked Jul 4, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  Different behavior when compared to HF

  danieltudosiu asked Jun 27, 2024 in Q&A · Closed · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  How to run LMDeploy gradio but for vision models

  bks5881 asked Jun 5, 2024 in Q&A · Answered

  

  

  

  

  5
• 1 You must be logged in to vote

  🙏

  lmdeploy有generator api可以只返回output ids而不是output string吗？

  haoxufd asked May 9, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  💬

  How to load a model in bf16 format

  georgel2004 started May 1, 2024 in General · Closed

  

  

  1
• 1 You must be logged in to vote

  💬

  支持的transformers版本可以更新到4.39以上吗w

  IcyFeather233 started Apr 25, 2024 in General

  

  

  1
• 1 You must be logged in to vote

  🙏

  Does lmdeploy support llama3 deployment?

  Lisaiplus asked Apr 23, 2024 in Q&A · Unanswered

  

  

  1
• 1 You must be logged in to vote

  🙏

  stream_infer input_embeddings

  JidongZhang-THU asked Dec 26, 2023 in Q&A · Answered

  

  

  

  

  4
• 1 You must be logged in to vote

  🙏

  How to custom <|SYS|> contents when using API? (after deployed)

  maxchiron asked Dec 19, 2023 in Q&A · Answered

  

  

  

  

  8
• 1 You must be logged in to vote

  🙏

  CUDA out of memory while Deploy

  anisingh1 asked Oct 5, 2023 in Q&A · Unanswered

  

  

  

  

  5
• 1 You must be logged in to vote

  🙏

  How do I remove the sharing of gradio in public url?

  supersuper1234 asked Sep 29, 2023 in Q&A · Unanswered

  

  

  

  

  2
• 2 You must be logged in to vote

  🙌

  如何使用 LMDeploy 把 InternLM-20B-4bit 部署为服务

  vansin started Sep 22, 2023 in Show and tell

  

  

  

  

  11
• 1 You must be logged in to vote

  💡

  Does LMdeploy support latest version of flash attention, page attention?

  ywangwxd started Sep 26, 2023 in Ideas

  

  

  2