update mscale calculation to keep back compatible with previous phi models #20

zelinms · 2024-08-22T11:04:56Z

update mscale calculation to keep back compatible with previous phi models

…ect#6920) Co-authored-by: youkaichao <[email protected]>

)

Co-authored-by: Varun Sundar Rabindranath <[email protected]>

)

…#7012) Co-authored-by: Robert Shaw <[email protected]>

…t#5036)

…ings (vllm-project#6758) Signed-off-by: Travis Johnson <[email protected]>

…m-project#6695) Co-authored-by: Aurick Qiao <[email protected]>

…vided by user (vllm-project#6954)

Co-authored-by: Michael Goin <[email protected]>

…m-project#6992)

…ject#7002)

Signed-off-by: Muralidhar Andoorveedu <[email protected]>

…t#7279) Co-authored-by: Nick Hill <[email protected]>

Signed-off-by: Alex-Brooks <[email protected]>

…tion (vllm-project#7109)

Co-authored-by: Michael Goin <[email protected]>

…ject#5428)

Co-authored-by: afeldman-nm <[email protected]>

…#7520)

…-during-spec-decoding (vllm-project#7665) Signed-off-by: Travis Johnson <[email protected]>

…on (vllm-project#7508)

Signed-off-by: kevin <[email protected]>

)

…ct#7670)

…#7673) [Core] Refactor executor classes to make it easier to inherit GPUExecutor (vllm-project#7673)

Those commits include: * New model named PhiMoE. * New A100 inference kernels for PhiMoE FP8 quantization.

…odels

comaniac and others added 30 commits July 31, 2024 12:02

[MISC] Introduce pipeline parallelism partition strategies (vllm-proj…

bd70013

…ect#6920) Co-authored-by: youkaichao <[email protected]>

[Bugfix] Support cpu offloading with fp8 quantization (vllm-project#6960

460c188

)

[Kernel] Enable FP8 Cutlass for Ada Lovelace (vllm-project#6950)

93548eb

Co-authored-by: Varun Sundar Rabindranath <[email protected]>

[Kernel] Tuned int8 Cutlass Kernels for SM75 (T4) (vllm-project#6996)

35e9c12

Co-authored-by: Varun Sundar Rabindranath <[email protected]>

[Misc] Add compressed-tensors to optimized quant list (vllm-project#7006

a0dce93

)

Revert "[Frontend] Factor out code for running uvicorn" (vllm-project…

7eb0cb4

…#7012) Co-authored-by: Robert Shaw <[email protected]>

[Kernel][RFC] Refactor the punica kernel based on Triton (vllm-projec…

7ecee34

…t#5036)

[Model] Pipeline parallel support for Qwen2 (vllm-project#6924)

1d2e7fb

[Bugfix][TPU] Do not use torch.Generator for TPUs (vllm-project#6981)

23993a7

[Bugfix][Model] Skip loading lm_head weights if using tie_word_embedd…

630dd9e

…ings (vllm-project#6758) Signed-off-by: Travis Johnson <[email protected]>

PP comm optimization: replace send with partial send + allgather (vll…

0437492

…m-project#6695) Co-authored-by: Aurick Qiao <[email protected]>

[Bugfix] Set SamplingParams.max_tokens for OpenAI requests if not pro…

3c10591

…vided by user (vllm-project#6954)

[core][scheduler] simplify and improve scheduler (vllm-project#6867)

c8a7e93

[Build/CI] Fixing Docker Hub quota issue. (vllm-project#7043)

a72a424

[CI/Build] Update PyTorch to 2.4.0 (vllm-project#6951)

7e0861b

Co-authored-by: Michael Goin <[email protected]>

[Bugfix] Fix RMSNorm forward in InternViT attention qk_layernorm (vll…

2dd3437

…m-project#6992)

[CI/Build] Remove sparseml requirement from testing (vllm-project#7037)

fb3db61

[Bugfix] Lower gemma's unloaded_params exception to warning (vllm-pro…

f4fd390

…ject#7002)

[Models] Support Qwen model with PP (vllm-project#6974)

fc912e0

Signed-off-by: Muralidhar Andoorveedu <[email protected]>

Update run-amd-test.sh (vllm-project#7044)

562e580

[Misc] Support attention logits soft-capping with flash-attn (vllm-pr…

805a8a7

…oject#7022)

[CI/Build][Bugfix] Fix CUTLASS header-only line (vllm-project#7034)

6a11fdf

[Performance] Optimize get_seqs (vllm-project#7051)

6ce01f3

[Kernel] Fix input for flashinfer prefill wrapper. (vllm-project#7008)

954f730

[mypy] Speed up mypy checking (vllm-project#7056)

3bb4b1e

[ci][distributed] try to fix pp test (vllm-project#7054)

2523577

Fix tracing.py (vllm-project#7065)

cf2a1a4

[cuda][misc] remove error_on_invalid_device_count_status (vllm-projec…

660dea1

…t#7069)

[Core] Comment out unused code in sampler (vllm-project#7023)

db35186

[Hardware][Intel CPU] Update torch 2.4.0 for CPU backend (vllm-projec…

c16eaac

…t#6931)

ywang96 and others added 29 commits August 17, 2024 13:30

[VLM] Refactor MultiModalConfig initialization and profiling (vllm-…

bbf55c4

…project#7530)

[TPU] Skip creating empty tensor (vllm-project#7630)

ce14335

[TPU] Use mark_dynamic only for dummy run (vllm-project#7634)

0c2fa50

[TPU] Optimize RoPE forward_native2 (vllm-project#7636)

ab7165f

[ Bugfix ] Fix Prometheus Metrics With zeromq Frontend (vllm-projec…

e3b3182

…t#7279) Co-authored-by: Nick Hill <[email protected]>

[CI/Build] Add text-only test for Qwen models (vllm-project#7475)

40e1360

Signed-off-by: Alex-Brooks <[email protected]>

[Misc] Refactor Llama3 RoPE initialization (vllm-project#7637)

200a2ff

[Core] Optimize SPMD architecture with delta + serialization optimiza…

ff7ec82

…tion (vllm-project#7109)

[Core] Use flashinfer sampling kernel when available (vllm-project#7137)

f710fb5

Co-authored-by: Michael Goin <[email protected]>

[Bugfix] Fix xpu build (vllm-project#7644)

1a36287

[Misc] Remove Gemma RoPE (vllm-project#7638)

df845b2

[MISC] Add prefix cache hit rate to metrics (vllm-project#7606)

3ac50b4

[Bugfix] fix lora_dtype value type in arg_utils.py - part 2 (vllm-pro…

dad961e

…ject#5428)

[core] Multi Step Scheduling (vllm-project#7000)

47b65a5

Co-authored-by: afeldman-nm <[email protected]>

[Core] Support tensor parallelism for GGUF quantization (vllm-project…

7601cb0

…#7520)

[Bugfix] Don't disable existing loggers (vllm-project#7664)

da11523

[TPU] Remove redundant input tensor cloning (vllm-project#7660)

43735bf

[Bugfix] use StoreBoolean instead of type=bool for --disable-logprobs…

67e02fa

…-during-spec-decoding (vllm-project#7665) Signed-off-by: Travis Johnson <[email protected]>

[doc] fix doc build error caused by msgspec (vllm-project#7659)

e54ebc2

[Speculative Decoding] Fixing hidden states handling in batch expansi…

312f761

…on (vllm-project#7508)

[ci] Install Buildkite test suite analysis (vllm-project#7667)

0df7ec0

Signed-off-by: kevin <[email protected]>

[Bugfix] support tie_word_embeddings for all models (vllm-project#5724

f4fc733

)

[CI] Organizing performance benchmark files (vllm-project#7616)

3d8a5f0

[misc] add nvidia related library in collect env (vllm-project#7674)

c4be16e

[XPU] fallback to native implementation for xpu custom op (vllm-proje…

e6d811d

…ct#7670)

[misc][cuda] add warning for pynvml user (vllm-project#7675)

ad28a74

[Core] Refactor executor classes for easier inheritance (vllm-project…

b6f99a6

…#7673) [Core] Refactor executor classes to make it easier to inherit GPUExecutor (vllm-project#7673)

PhiMoE for vllm

86f1682

Those commits include: * New model named PhiMoE. * New A100 inference kernels for PhiMoE FP8 quantization.

update mscale calculation to keep back compatible with previous phi m…

5de796b

…odels

zelinms closed this Aug 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update mscale calculation to keep back compatible with previous phi models #20

update mscale calculation to keep back compatible with previous phi models #20

zelinms commented Aug 22, 2024

update mscale calculation to keep back compatible with previous phi models #20

update mscale calculation to keep back compatible with previous phi models #20

Conversation

zelinms commented Aug 22, 2024