43
submitted 2 years ago by BrikoX@lemmy.zip to c/technology@lemmy.zip

Running AI models without matrix math means far less power consumption—and fewer GPUs?

you are viewing a single comment's thread
view the rest of the comments
[-] blindsight@beehaw.org 3 points 2 years ago

This could be huge, but we'll need to wait and see. The economic and ecological footprint of LLMs is problematic.

That said, will this actually help, or will they just use 3T parameter models to outcompete competitors 1T parameter models using GPUs? Really, this is more about small-scale models competing with midsize models. Like, this could bring a model as big as GPT 3.5 down to be something you could run on affordable hardware, right?

That would be really compelling for my sector (education) where there's a lot of concern about student data privacy. I could definitely pitch building a local $5K-cost LLM server that could handle a dozen or so simultaneous users. That would be enough for a small school district.

this post was submitted on 25 Jun 2024
43 points (97.8% liked)

Technology

7221 readers
199 users here now

News community around technology, social media platforms, information technology and governmental policy surrounding it.

What doesn't fit here?

The core of the story has to be technology focused.


Post guidelines

Title formatPost title should mirror the news source title. If you don't like the title of article, look for an alternative source instead of editorializing it.
URL formatPost URL should be the original link to the article (even if paywalled) and archived copies left in the body. It allows avoiding duplicate posts when cross-posting.
[Opinion] prefixOpinion (op-ed) articles must use [Opinion] prefix before the title. Opinion articles refer to articles that their publisher doesn't explictly endorse.
Country prefixCountry prefix can be added to the title with a separator (|, :, etc.) if the news is from a local publisher who doesn't clearly mention the country.


Rules

1. English onlyTitle and associated content has to be in English.
2. Use original linkPost URL should be the original link to the article (even if paywalled) and archived copies left in the body. It allows avoiding duplicate posts when cross-posting.
3. Respectful communicationAll communication has to be respectful of differing opinions, viewpoints, and experiences.
4. InclusivityEveryone is welcome here regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, education, socio-economic status, nationality, personal appearance, race, caste, color, religion, or sexual identity and orientation.
5. Ad hominem attacksAny kind of personal attacks are expressly forbidden. If you can't argue your position without attacking a person's character, you already lost the argument.
6. Off-topic tangentsStay on topic. Keep it relevant.
7. Instance rules may applyIf something is not covered by community rules, but are against lemmy.zip instance rules, they will be enforced.


Companion communities

!globalnews@lemmy.zip
!interestingshare@lemmy.zip


Icon attribution | Banner attribution


If someone is interested in moderating this community, message @brikox@lemmy.zip.

founded 2 years ago
MODERATORS