sometimes, I think of ponies

[solarbird]

Have you ever noticed that every projection about “AGI” and “superintelligence” has an “and then a miracle occurs” step?

I have.

I shouldn’t say every projection – there are many out there, and I haven’t seen them all. But every one I’ve personally seen has this step. Somewhere, sometime, fairly soon, generative AI will create something that triggers a quantum leap in capability. What will it be? NOTHING MERE HUMANS CAN UNDERSTAND! Oh, sometimes they’ll make up something – a new kind of transistor, a new encoding language (like sure, that’ll do it), whatever. Sometimes they just don’t say. Whatever it is, it happens, and then we’re off to the hyperintelligent AGI post-singularity tiems.

But the thing is … the thing is … for Generative AI to create a Magic Something that Changes Everything – to have this miracle – you have to already have hyperintelligent AGI. Since you don’t… well…

…that’s why it’s a miracle. Whether they realise it or not.

I’m not sure which is worse – that they do realise it, and know they’re bullshitting billions of dollars away from productive society to build up impossible wealth before the climate change they’re helping make worse fucks everything so they can live like feudal kings from their bunkers, or whether they don’t, and are spirit dancing, wanking off technofappic dreams of creating a God who will save the world with its AI magic, a short-term longtermism, burning away the rest of the carbon budget in a Hail Mary that absolutely will not connect.

Both possibilities are equally batshit insane, I know that much. To paraphrase a friend who knows far more about the maths of this than I, all the generative AI “compute” in the universe isn’t going to find fast solutions to PSPACE-HARD problems. It’s just not.

And so, sometimes, sometimes, sometimes, I think of…

…I think of putting a short reading/watching list out there, a list that I hesitate to put together in public, because the “what the actual fuck” energies are so strong – so strong – that I can’t see how anyone could take it seriously. And yet…

…so much of the AI fantasia happening right now is summed by three entirely accessible works.

Every AI-fantasia idea, particularly the ideas most on the batshit side…

…they’re all right here. And it’s all fiction. All of it. Some of it is science-shaped; none of it is science.

But Alice, you know, we’re all mad here. So… why not.

Let’s go.

1: Colossus: The Forbin Project (1970)

This is the “bad end” you see so much in “projections” about AI progression. A new one of these timelines just dropped, they have a whole website you can play with. I’m not linking to it because why would I, holy shit, I don’t need to spread their crazy. But there’s a point in the timeline/story that they have you read – I think it’s in 2027 – when you can make a critical choice. It’s literally a one-selection choose-your-own-path adventure!

The “good” choice takes you to galactic civilisation managed by friendly hyperintelligent AGI.

The “bad” choice is literally the plot of The Forbin Project with an even grimmer ending. No, really. The beats are very much the same. It’s just The Forbin Project with more death.

Well. And a bioweapon. Nukes are so messy, and affect so much more than mere flesh.

2: Blindsight, by Peter Watts (2006)

This rather interesting – if bleak – novel presents a model of cognition which lays out an intriguing thought experiment, even if it … did not sit well with what I freely admit is my severely limited understanding of cognition.

(It is not helped that it directly contradicts known facts about the cognition of self-awareness in various animals, and did so even when it was published. That doesn’t make it a worse thought experiment, however. Or a worse novel.)

It got shortlisted – deservedly – for a bunch of awards. But that’s not why it’s here. It’s here because its model of cognition is functionally the one used by those who think generative AI and LLMs can be hyperintelligent – or even functionally intelligent at all.

And it’s wrong. As a model, it’s just wrong.

Finally, we get to the “what.” entry:

3: Friendship is Optimal, by Iceman (2012)

Friendship is Optimal is obviously the most obscure of these works, but also, I think maybe the most important. It made a big splash in MLP fandom, before landing like an absolute hand grenade in the nascent generative AI community when it broke containment. Maybe not in all of that latter community – but certainly in the parts of which I was aware. So much so, in fact, that it made waves even beyond that – which is when I heard of it, and how I read it.

And yes… it’s My Little Pony fanfic.

Sorta.

It’s that, but really it’s more an explicit AI takeoff story, one which is absolutely about creating a benevolent hyperintelligent Goddess AI construct who can, will, and does remake the world, destroying the old one behind her.

Sound familiar?

These three works include every idea behind every crazy line of thought I’ve seen out of the Silicon Valley AI crowd. These three works right here. A novel or a movie (take your choice, the movie’s quite good, I understand the novel is as well), a second novel, and a frankly remarkable piece of fanfic.

For Musk’s crowd in particular? It’s all about the model presented in Friendship is Optimal, except, you know, totally white supremacist. They’re even kinda following the Hofvarpnir Studios playbook from the story, but with less “licensed property game” and a lot more more “Billionaire corporate fascism means you don’t have to pay employees anymore, you can just take all the money yourself.”

…which is not the kind of sentence I ever thought I’d write, but here we are.

You can see why I’m hesitant to publish this reading list, but I also hope you can see why I want to.

If you read Friendship is Optimal, and then go look at Longtermerism… I think you definitely will.

So what’re we left with, then?

Some parts of this technology are actually useful. Some of it. Much less than supports the valuations, but there’s real use here. If you have 100,000 untagged, undescribed images and AI analysis gives 90% of them reasonable descriptions, that’s a substantial value add. Some of the production tools are good – some of them are very good, or will be, once it stops being obvious that “oh look, you’ve used AI tools on this.” Some of the medical imaging and diagnostic tools show real promise – though it’s always important to keep in mind that antique technologies like “Expert Systems” seemed just as promising, in the lab.

Regardless, there’s real value to be found in those sorts of applications. These tasks are where it can do good. There are many more than I’ve listed, of course.

But AGI? Hyperintelligence? The underlying core of this boom, the one that says you won’t have to employ anyone anymore, just rake in the money and live like kings?

That entire project is either:

A knowing mass fraud inflating a bubble nobody’s seen in a century that instead of breaking a monetary system might well finish off any hopes for a stable climate in an Enron-like insertion of AI-generated noise followed by AI-generated summarisation of that noise that no one reads and serves no purpose and adds no value but costs oh, oh so very much electricity and oh, oh, oh so very much money;

A power play unlike anything since the fall of the western Roman empire, where the Church functionally substituted itself in parallel to and substitute of of the Roman government to the point that the latter finally collapsed, all in service of setting up a God’s Kingdom on Earth to bring back Jesus, only in this case, it’s setting up the techbro billionaires as a new nobility, manipulating the hoi polloi from above with propaganda and disinformation sifted through their “AI” interlocutors;

Or an absolute psychotic break by said billionaires and fellow travellers so utterly unwilling and utterly unable to deal with the realities of climate change that they’ll do anything – anything – to pretend they don’t have to, including burning down the world in the service of somehow provoking a miracle that transcends maths and physics in the hope that some day, some way, before it’s too late, their God AI will emerge and make sure everything ends up better… in the long term.

Maybe, even, it’s a mix of all three.

And here I thought my reading list was the scary part.

Silly me.

Posted via Solarbird{y|z|yz}, Collected.

Comments

Re: You had my attention at Ponies

[frith]

The mind copying process is wrong, as is suspected by several authors. Brain tissue and virtual replacement would have to be functioning in tandem before the brain tissue could be slowly destroyed, neuron by neuron. A proved uninterrupted flow of conscientious during the whole process should demonstrate that CelestAI is not in fact slaughtering billions of people. It's the destructive copying process and blank-out period prior to waking up in virtual Equestria that bothers me.

Re: You had my attention at Ponies

[solarbird]

Oh, it's utterly magitech.

I mean, you're completely unconscious so there's not going to be any sort of continuance, and more importantly, I don't see continuance of consciousness as being particularly relevant. I do see what you're getting at, don't get me wrong.

But people don't have continuance of consciousness here and now, and I'm not just talking about being in a coma, though I am also talking about being in a coma, having been in one. Some people may find it a little weird that reinstantiation of the same complete data set on functionally identical hardware across a gap (of either space or time), is still me, but I kinda do.

Re: You had my attention at Ponies

[frith]

And I think Dr. Bones McCoy isn't wrong in thinking that the body that walks off the transporter platform isn't the same one that got beamed up from several kilometers away.

Well, between long and short memory retention, memory loss, brain damage, waking and sleeping, personality changes... What constitutes a persistence of identity? Magical thinking, I guess. I am me because I said so.

Re: You had my attention at Ponies

[wrog]

There's a deeper logical problem, even supposing for the sake of argument it were actually possible to read minds/brain-structure to the necessary level of detail (something I already have **cough**UncertaintyPrinciple**cough** issues with):

Having sufficient information to replicate a personality inevitably also means having sufficient information to undetectably pretend to be that person.

And it doesn't matter what one might have doing the pretending (some kind of AI/LLM thing or some Different Actual Person, whatever your definition of "person” might be); the point is you'll never be able to tell the difference.

Or to put it another way:
There is no such thing as a "soul detector" and the problem is not one of science or technology, it's definitional.

Re: You had my attention at Ponies

[solarbird]

And following from that is that if all the information you have is exactly all of those things needed to that end and nothing more, then is there any difference between pretending to be that person and actually being that person or at least an instance thereof?

I don't really think so. And I think the author knows that, hence the destructive reading. Each bit of information exists in one place only at a time. I think that's them both making the stakes higher for the uploaded and dodging the existing-simultaniously-in-two-places problem of a survivable copy process.

Re: You had my attention at Ponies

[wrog]

is there any difference between pretending to be that person and actually being that person?

Most people take "pretending" to mean that there's some important way in which one is not actually that person, that one retains the capacity to stop pretending at some point

(... e.g., to pursue goals the original person would never have supported once some position of trust has been secured,... the canonical "sleeper agent" scenario)

Re: You had my attention at Ponies

[solarbird]

Yeah, but that first clauses changes it.

"if all the information you have is exactly all of those things needed to that end and nothing more"

To get that "sleeper agent" to "activate" it seems like you'd have to have an injection of a lot more data, at which point, you basically have another person entirely again.

Not that this wouldn't be possible in this kind of holy-shit-yikes nightmare environment we're talking about, but still.

What I'm doing/playing with here is the idea that if you have enough information to be a sleeper agent, that's enough information that it'll interfere with your ability to pretend to be someone else perfectly, and if you don't have that information and can "be" that person perfectly, maybe you just are that person? Or at least another instance of that person.

Re: You had my attention at Ponies

[wrog]

If you have the necessary information, then you can simulate.
If having additional information interferes with this then there are most likely internal architecture games one can play, e.g., tag the extra information, run the simulation in a sandbox, etc -- classic man-in-the-middle attack

If the internal architecture is unknown, and all we get to do is ask questions from outside, and all of the correct answers are available to the simulation, we're screwed. If there exist "questions" that the entity would "answer" differently -- read situations where the entity will ultimately behave differently -- but we don't know what they are, reliably finding them is solving the halting problem and we are again screwed.

The question of whether we indeed have "another instance of the same person" is fundamentally unobservable, and therefore, I would say, meaningless.