this post was submitted on 18 Oct 2023
54 points (93.5% liked)

Technology

58760 readers
4996 users here now

This is a most excellent place for technology news and articles.

Our Rules

  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago
MODERATORS
L3s@lemmy.world
enu@lemm.ee
fry@fry.gs
L3s@fry.gs
enu@lemmy.world
L4s@fry.gs
L4s@lemmy.world
54
Large Language Models up to 4x Faster on RTX With TensorRT-LLM for Windows (blogs.nvidia.com)
submitted 1 year ago by ijeff@lemdro.id to c/technology@lemmy.world
1 comments fedilink hide all child comments
 

cross-posted from: https://lemdro.id/post/2377716 (!aistuff@lemdro.id)

top 1 comments
sorted by: hot top controversial new old
[–] korewa@reddthat.com 3 points 1 year ago

Dang I need to try these for now only the stable diffusion extension for automatic 1111 is available.

I wonder if it will accelerate 30b models that doesn’t fit all in the gpu vram.

If it only accelerates 13b then it was already fast enough