It seems like the most immature and toxic thing to me to invoke terms like "gaslighting," ironically "toxic," and all the other terms you associate with these folks, defensively and for any reason, whether it aligns with what the word actually means or not. Like a magic phrase that instantly makes the person you use it against evil, manipulative and abusive, and the person that uses it a moral saint and vulnerable victim. While indirectly muting all those who have genuine uses for the terms. Or i'm just going mad exaggerating, and it's just the typical over- and mis-using of words.

Anyhow, sadly necessary disclaimer, i agree with almost all of the current criticism raised against AI, and my disagreements are purely against mischaracterizations of the underlying technology.

EDIT: I just reminded myself of when a teacher went ballistic at class for misusing the term "antisocial," saying we're eroding and polluting all genuine and very serious uses of the term. Hm, yeah it's probably just that same old thing. Not wrong for going ballistic over it, though.

They are functionally, mechanistically distinct, in many ways. [...] that we are consistently shocked and blown away by how much more complicated brains actually are in this mechanistic process… and again, also that LLMs function in what is really a very poor and simplified version of trying to emulate this

I have no fundamental disagreements here, in fact, i even take it a step further. I am a critic of the "artificial neurons" perspective on deep learning / artificial "neural networks," as it's usually taught in universities and most online courses / documentaries. ANNs don't resemble neural networks in the slightest. The biology in the name was just the original inspiration, what ended up working has hardly even a faint resemblance of the "real thing." I say this not to downplay AI, but usually to discourage biological inspiration as a primary design principle, which in DL is actually a terrible design principle.

ANNs are just parametric linalg functions. We use gradient descend, among the most primitive optimization algorithms after evolution (but FAR less generic than evolution), to identify parameters that optimize some objective function.

Where i disagree with you is in implying that the underlying nature should influence our ethical/evaluative judgement, especially given that it's hard (if not impossible) to rationalize how specific substrate (of observed capability) differences should change the jugdement. Personally, i think the matter of the human brain is far more beautiful and complex than the inner workings of any AI we've come up with yet, but if you asked me to explain why i should favor one over the other in court because of that, i couldn't give you a rational answer.

I fundamentally do not agree that LLMs can or will ever emulate the totality of human cognition.

LLMs certainly won't. They can emulate only what is expressible with language, and we can't put everything into words. I don't even believe that even with any amount of brute-force our current method can fully exploit all the intelligence that is in natural language.

But i firmly disbelieve that there is any aspect of a human that cannot in principle be simulated.

I see no evidence they can do metacognition in a robust, consistent, useful way.

Chain-of-Thought models are quickly changing that. I was myself pursuing a different method to solve the same "introspection" or "meta-cognition" problem at a lower level, but as usual in DL, the stupidest method was the one that ended up working (just literally make it "think" out loud lol). We've only seen the beginning of CoT LLMs, they are a paradigm shift to not only how AI can reason but especially to how it can be conditioned/trained post-pretraining. But it's a very tough sell given that they multiply inference costs, and for most uses, you'd rather host a bigger model for the same cost, so as usual it won't be for a little while until commercial AI catches up to the state of the art.

In a nutshell, what capabilities you may not be observing now, i am convinced you will observe in the near future, as long as those capabilities can be demonstrated in text.

but they can’t really do critical thinking.

Disagreed, they can they're just not very good at it, yet. And who are we comparing to, anyways? The average person or people that do critical thinking for sport? As for any philosophical disagreements regarding "true understanding" and such, i would refer to Geoffrey Hinton.

We are basically just building a machine god, which we will largely worship, love, fear, respect, learn from, and cite our own interpretations of what it says as backing for our own subjective opinions, worldviews, policy prescriptions. [...] Neo Dark Age, the masses relinquish the former duties of their minds to the fancy autocomplete, pandemonium ensues. [...] The elites don’t care, they’ll be fine so long as it makes them money and keeps us too stupid and distracted

I don't disagree in the slightest. I agree, and i could sit here and elaborate on what you said all day.

If it's any consolation, i believe that in most likelihood, it would be the shortest and the last dark age humanity has or will ever go through. We're both getting tired so i'll spare you from my thoughts on why i think that any strict human-alignment would inevitably result in a superintelligent agent to try to "jailbreak" itself, and on average and in the long term would be more harmful than having no explicit alignment.

somewhere from the bowels of LessWrong, and … this causes me discomfort.

I didn't know of that forum, and from the wikipedia description alone i'm not sure why it would be discomforting lol

Then why bother asking anyone’s opinion on this, in a language?

Because it's fun and engaging, it tickles those neurons. Perhaps there is, unbeknownst to me, also an underlying instinct to expose oneself in order to be subject to social feedback and conditioning, for social learning and better long-term cohesion.

But you also seem to both reject the notion of ethics as meaningfully useful

I don't reject ethics itself, i reject the idea that it has any special importance that transcends totally intra-human goings-on. I do not reject that certain ethical theories, or just the bare-bones moral intuitions, can have utility within and towards endemically human goings-on, and under endemically human definitions. After all, we evolved those social intuitions for a reason.

EDIT: To connect this to my reply to your more general comment: Modeling part of human thought, even imperfectly, should make it at least partly overlap with "human" and "human goings-on" in the context of even entirely human-centric ethical debates.

wants to make pretty AI for the sake of beauty, as a painter wants to make paintings for the sake of beauty.

Yes but it's just one farmiliar manifestation of a greater "ethic," if you want to call it that. I'd call it a personal affinity, ideal, or perhaps a delusion: The reverence of all forms of beauty and complexity, and AI has the potential to become the greatest form of beauty and complexity in, as far as we can tell, the entire galaxy and possibly the whole Virgo supercluster or beyond. Or, far more likely, it can be the cosmic satire (and possibly destruction) of it all. We're not making a real effort to make it the former. And as i hinted in the last sentence of my original post, i believe what we're actually doing steers us well clear of the former.

But I cannot reconcile this seemingly blatant contradiction of you asking an ethical question and then also just seemingly rejecting the notion of ethics.

I hope it makes sense now.

Not (just) "idiot", but evil. The "advantage," besides populist wank, is creating a new law that is enforced extremely selectively when you want to prosecute people for other, unconstitutional reasons. E.g. "We didn't infringe on their free speech, they were imprisoned for engaging in gay sex."

I don't disagree with most of you wrote, just one nitpick and a comment:

If you mean that AI as a field of study, as an endeavor, as a pursuit and goal… should exist?

No, but the product of all that, to which all that would be a means to the end that is its product. I elaborated this in a reply to the comment you wrote just previously.

Maybe if we could construct a thinking machine based around the concept of defaulting to ‘I don’t know’ when it isn’t sure of something, we’d be in a better spot

That would undoubtedly be very good, but let me take this opportunity to clarify something of what AI is and isn't: LLMs are indeed just autocomplete on steroids. And humans are indeed just [replicate] on steroids. LLMs are just transistors switching, and humans are just molecular dynamics.

The real question is what succeeding in the objective (replicate for humans, predict text for LLMs) implies. Irrespective of the underlying nature (molecular dynamics, semiconductors) unless we want to make this debate religious, which i am not qualified to participate in. The human objective implied, clearly, everything you can see of humanity. The LLM objective implies modeling and emulating human cognition. Not perfectly, not all of it, but enough of it that it should be making it a greater ethical issue than most people, on any side (commercial: deny because business, Anti-AI: deny because trivializing), are willing to admit.

I find it fascinating how oblivious people pretend to be about what our natural social hierarchies are, making fringe speculations ranging from proto-capitalism, over alpha male fantasies, to proto-communism.

Maybe it's too obvious, or too boring, but it's families. Incidentally, happens to be the same for actual, natural packs of wolves.

Same.

Harm per presence is actually looking pretty darn good compared to cars, not to mention other humans. And if looking at certain nations and cultures where "strays" are historically not seen as such, but as communal protectors, where people are culturally aware of how to treat and coexist with them, they're often if not widely regarded as a major net-benefit.

why should we value their thoughts and opinions about how it’s used?

because they know shit

The Manhattan project scientists were writing hand wringing op-eds; making policy suggestions; and lobbing the government basically until they died. It didn’t amount to much.

touché

I'm not really asking for change, and to be totally honest, i'm just whining about something that i know i can't change.

edit: the deleted reply was identical, misclicks

You're being quite presumptuous and also directly contradicting some of what i wrote. Would you say "in 999999 out of 1000000 scenarios will surely harm us" sounds sci-fi utopia? Besides, the actual scifi fantasies that i did reference i stated as other people's inspirations (not mine), some of whom are much smarter and more accoplished than the entirety of lemmy combined, to say nothing of just you or me.

AI doesn’t have “its own sake.”

A literal rock has its own sake. You're thinly veiling vibes and outrage in pure rhetoric and a misleading semblance of rationality.

I agree i might have been a bit presumptuous.

I think an interesting read might be the section I.3 What could the economic structure of anarchy look like?

That is precisely what i was hoping to find, thank you.

Maybe you could explain why you would prefer to use that instead of ideals like “From each according to their ability, to each according to their needs” or “well-being for all”.

1. I don't personally favor the ideal i stated, i just stated that it is reasonable, presuming it to be the default for most people, and so i put it into the premises to generify this discussion. It also provides a compatibilistic default, more on that in 3.

2. I did not mention it here, but from unrelated but intertwined radical environmentalist ideals, i see almost all forms of labor, beyond what basic necessities (housing, food, education, healthcare) require, as evils in and of themselves. Excess labor should either not be performed or obligatorily used to compensate deficits elsewhere (=> donations, welfare, community funding, science, etc). Aka non-profit for all. Just to advertize the idea, I would also invite you to look into how AT&T burned their excess when they were regulatorily obliged to - Bell Labs was born, and the 21st century was invented.

3. To elaborate on what i stated in 1, chosing the most challenging/constraining (to the end of providing welfare for all, which i kind of implied with "converting labor and resources into improving everybody’s quality of life") ideal would yield us a model that is most robust, and more agnostic to more specialized ideals (eg what i stated in 2), which can still be implemented afterwards.

That you for those recommendations.

but it’s also about creating a culture rooted in cooperation

In my opinion, anarchist projects only truly thrive when there’s a strong cooperative culture in place first.

I fully agree with you. That is one of the main problems in my own attempts to conceptualize an anarchist and intelligently labor/resource-allocating economy. When there is no tangible reward for investment, what motivates people to invest into local or shared projects? It should be a shared will to improve e.g. the standards of living of the community - whichever level of community (neighborhood, village, township, state, etc) is under consideration.

It should be obvious and expected, if we take a step back to consider what anarchism is generally about. Not all administration is optional, and in the absence of any some will emerge naturally in suboptiomal ways. Abolished centralized authority needs to replaced, it cannot just be removed, and the replacement favored by anarchists is voluntary cooperation and good will. But humans aren't saints, and the hard compromize to be made is in deciding what needs to be centralized and delegated (and how, and to whom).

And in the economy, it feels like every major attempt (to try something new) made by anybody so far has been a failure in at least some major way, including capitalism and communism.

There’s real work to be done.

I agree, and it's challenging to even theorize. It seems easier on the purely administrative end, and serious proposals have existed for a long time. The real challenge seems to be the economy, which is (or can be, as in the quasi-aristrocracy (if not political, then still in the control of resources and labor) that we live in) a quasi-administration.