Skip to content
This repository was archived by the owner on May 11, 2025. It is now read-only.

Exllama kernels support #313

Merged
merged 12 commits into from
Jan 21, 2024
Merged

Exllama kernels support #313

Changes from 1 commit
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
45ae7df
added exllama wqlinear layer
IlyasMoutawwakil Jan 19, 2024
d76075d
added exllama interface in auto model class
IlyasMoutawwakil Jan 19, 2024
2c3d6c3
added example
IlyasMoutawwakil Jan 19, 2024
b3a48d6
added exllamav2
IlyasMoutawwakil Jan 19, 2024
c13bf28
added exllama ppl comparison
IlyasMoutawwakil Jan 19, 2024
8c5192f
optimized packing/unpacking
IlyasMoutawwakil Jan 20, 2024
c7a281f
pseudo-native integration of exllama layers
IlyasMoutawwakil Jan 21, 2024
ac374de
more optimized packing/unpacking
IlyasMoutawwakil Jan 21, 2024
ae01d9e
post inits readability
IlyasMoutawwakil Jan 21, 2024
77ef2a3
Update import of exl kernels
casper-hansen Jan 21, 2024
5af3c59
Fused + ExLlama. Cleanup MLP.
casper-hansen Jan 21, 2024
2b35947
Fix exllamav2 loading
casper-hansen Jan 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Fix exllamav2 loading
  • Loading branch information
@casper-hansen
casper-hansen committed Jan 21, 2024
commit 2b3594701bc969bde518613661d454f707038e6a
6 changes: 5 additions & 1 deletion awq/models/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -312,7 +312,11 @@ def from_quantized(
elif use_exllama_v2:
# creates q4 handle and allocates scratch spaces wrt max_input_len and
# max_batch_size, which are hardcoded for now but might be worth interfacing
model = exllamav2_post_init(model, max_input_len=2048, max_batch_size=1)
model = exllamav2_post_init(
model,
max_input_len=max_new_tokens,
max_batch_size=int(os.getenv("AWQ_BATCH_SIZE", 1))
)

return self(
model,
Expand Down