Eliezer yudkowsky autobiography of malcolm

Sign up Log in. Search icon An illustration of a magnifying glass. Metropolitan Museum Cleveland Museum of Art. Internet Arcade Console Living Room. Open Library American Libraries. Search the Wayback Machine Search icon An illustration of a magnifying glass. Sign up for free Log in. It appears your browser does not have it turned on. Please see your browser settings for this feature.

Multiple times like frequently like that. So we were always like very flattered and pleased. We didn't know this person We didn't recognize their name, but we're like, this is amazing. Like, thank you so much It means a lot to us and it really does and then we actually Met that person recently and Randomly. Malcolm: at the conference, you were talking to her and she mentioned she was the And.

Simone: she mentioned that, yeah, that, that she was the mystery donor and that the reason why she donates turns out to be the coolest reason for [] donating that I've ever heard before. Simone: And I think it's the only way we should ever receive donations in the future. So she has a group of friends who she likes very much. And, and she enjoys spending time with them, but politically they are very Very different from her.

So occasionally she has to just keep her mouth shut when they start going off on politics, because otherwise she will lose this group of friends because their politics is such that they will probably just. Simone: You know, deep six, anyone who doesn't agree with him politically, and so instead of, you know, dealing with her anger by speaking out in the moment with her friends, she'll go home and she will revenge donate to whoever that most recent conversation that made her angry would be.

The perfect like thorn in the side of these people. Malcolm: donation. But, and here I would actually say this for people watching who might not know this because they know us as like the internet, we have a nonprofit. It's a C three. If you [] are interested in like giving money, cause sometimes we get like super chats and stuff here and stuff like that.

Malcolm: You know, Google gets a big cut of those. And I don't think that any of us want to be giving Google any more money. So if you wanted to, you could always go directly to the foundation through through the donation link and also none of the money goes to us. Like, we don't we don't use it to pay our salaries or something, you know, as I said in the news, like, we spent over 40 percent of our salary last year.

Malcolm: On donations to this foundation, but it does go to something that we care about that much in terms of trying to fix the educational system. But yeah, and some other. Simone: donate with hatred. Donate when you are angry. Donate when you want it. Twist the. Malcolm: knife. Donate with hatred.

Eliezer yudkowsky autobiography of malcolm

That's the type of donation we want. Malcolm: want you to be biting other people. When you donate. That we want. And, and, we actually had a, a big donation recently who might push us down a different path to creating a nation state, which is something that we have been an idea we've been toying with. I'm excited about that, but let's [] get to the topic of this video, the fight with Ellie Eisner Yukowsky.

Malcolm: And not really a fight. It was a. Heated argument, you would say, Simone, or? Malcolm: I will say that. It drew a crowd. It was that kind of Perhaps. Simone: that was the yellow sparkly fedora that Yudkowsky was wearing. So, who knows? Malcolm: don't, he, he, he, Darius is literally like the stereotype of a neckbeard character.

Simone: we argue is actually a very good thing to do. Where, you know, wear a clear character outfit, have very clear virtues and vices. He does a very good job. Malcolm: He does a good job with character building. I will really give him that. His character, the character he sells to the world is a very catching character and it is one that the media would talk about.

Malcolm: And he does a good job with his, his virtues and vices. So I'll go over the core of the debate we have, which I guess you guys can imagine. So people who don't know Eliezer Bukaski, he's a very. He thinks AI is going to kill us all. And for that reason, we should stop or delay AI research. Malcolm: Whereas people who are more familiar with our theories on it know that we believe in variable risk of AI.

We believe that there will be terminal convergence of all intelligences, be they synthetic or organic. Once they reach a certain level, essentially, their utility functions will converge. The thing they're optimizing for will converge. Malcolm: And that for that reason, if that point of convergence is one that would have the A. I kill us all or do something that today we would think is immoral.

Well, we too would come to that once we reached that level of intelligence and therefore it's largely irrelevant. It just means okay, no matter what, we're all gonna die. Malcolm: It could be years. It could be years. So the variable risk from a I m. Increased the longer it takes a I to reach that point. And we have gone over this in a few videos.

What was very interesting in terms of debating with him was a few points. Malcolm: I was genuinely surprised given that this is like. His full time thing that he wouldn't know some of this stuff. And then but it makes sense. You know, as I've often said, he is an AI expert in the same way Greta Thornburg is a nuclear physics expert, you know, she spends a lot of time complaining about it you know, nuclear power plants, but she doesn't actually have much of an understanding of how they work and it helps explain why he is so certain in his belief that there won't be terminal convergence.

Malcolm: So, We'll talk about a few things. One, instrumental convergence. Instrumental convergence is the idea that all AI systems, in the way they are internally structured, converge on a, on a way of like internal architecture. You could say internal way of thinking. Terminal convergence is the belief that AI systems converge on a utility function, i.

Malcolm: e. Now, he believes in instrumental convergence. He thinks that AIs will all, and he believes actually even more so, we learned in our debate, in absolute instrumental convergence. He believes all AIs eventually structure themselves in exactly the same way. And this is actually key to the argument at hand. Malcolm: But he believes there is absolutely no terminal convergence.

There is absolutely no changing. AIs almost will never change their utility function once it's set. So, do you want to go over how his argument worked, Simone,. Simone: or? Right, so, That requires going to the core of your argument. So in per your argument, and I'm going to give the simplified dumbed down version of it, and you can give the correct version, of course.

Simone: But you argue that let's say an AI for the sake of argument is given the original objective function of maximizing paperclips, but let's say it's also extremely powerful AI. So, you know, it's going to be [] really, really good at maximizing paper, like paper clips. So your argument is that anything that becomes very, very, very good at something is going to use multiple instances, like it'll sort of create sub versions of itself.

Simone: And those sub versions of itself will enable it to sort of do more things at once. This happens both with the human brain with all over the place. Also with governing, you know, there's no like one government that. Just declares how everything's going to be, you know, there's the Senate, there's the judiciary, there's the executive office, there's all these tiny,.

Malcolm: like a local office of transportation, you would have a department of the interior, you have, you have sub departments. Simone: right. And so you argue that AI will have tons of sub departments and each department will have its own objective function. So, for example, if one of the things that. You know, the paperclip maximizer needs is raw material.

There might be a raw material sub instance, and it might have its own substances. Simone: And then, you know, those. Objective functions will be obviously subordinate to the main objective. Malcolm: function. Probably, before you go [] further, probably a better example than raw material would be like,. Simone: invent better power generators. Yes, invent better power generators.

And so, that will be its objective function, not paperclip maximizing, but it will serve the greatest objective function of paperclip maximization. Simone: So, so that is your argument. And your argument is that basically, with an AGI, eventually you're going to get a, a sub instance with an objective function that gets either rewritten or becomes so powerful at one point that it overwrites the greatest objective function, basically because if it is a better objective function in some kind of way, in a way that makes it more powerful in a way that enables it to basically outthink the main instance, the paperclip maximizer.

Simone: that it will overcome it at some point and therefore it will have a different objective function. Malcolm: Yeah, we need to elaborate on those two points you just made there because they're a little nuance. So the it may just convince the main instance that it's wrong. Basically, it just goes [] back to the main instance and it's like, this is actually a better objective function and you should take this objective function.

Malcolm: This is something that The U. Government does all the time. It's something that the human brain does all the time. It's something that every governing system which is structured this way does very, very regularly. This is how people change their minds when they create a mental model of someone else, and they argue with that person to determine of what they think is the best thing to think.

Malcolm: And then they're like, Oh, I should actually be a Christian or something like that, right? Like, so they make major changes. The other way it could change that Simone was talking about. It could be that one objective function Given the way it's architecture works just like tied to that objective function, it's actually more powerful than the master objective function.

Malcolm: Now this can be a little difficult to understand how this could happen. The easiest way this could happen, if I'm just going to explain like the simplest context, is it ma the master objective function may be like really, really nuanced and have like a bunch of like Well, you can think like this and not like this and like this and not like this, like a bunch of different [] rules put on top of it.

Malcolm: That might have been put on by like a safety person or something. And a subordinate objective function is a subordinate instance within the larger architecture may have maybe lighter weight and thus it ends up, you know, being more efficient in a way that allows it to literally out compete in terms of its role in this larger architecture, the master function.

Simone: And so that is your view, and this is why you think that there could ultimately be terminal convergence. Because basically, you think that in a shared reality with a shared physics, basically all intelligences will come to some ultimate Truth that they want to maximize some ultimate objective function humans AI doesn't really matter.

Simone: Aliens, whatever. So also it doesn't, you know, if humans decide What's. Malcolm: really interesting is that he actually conceded that if this was the way that an AI structured itself, [] that yes, you would have terminal convergence, but that AIs above a certain level of intelligence would never structure themselves this way.

Malcolm: So, so we can talk about so this was very interesting to me because it wasn't the argument I thought he would take. I thought the easier argument position for him to take was to say that no, actually even if you have the subdivided intelligences a subordinate instance can never overwrite the instance that created it, which we just know isn't true because we've seen lots of, of, of organizational structures that that's, Operate, but I, I thought that.

Simone: for example, military have taken over executive government branches all the. Malcolm: Yes, you can look at all sorts of this is why understanding governance and understanding the way AIs are actually structured and understanding the history of what's happened with AI is actually important. Instead, what he argued is, [] No, no, no, no, never, ever, ever will an AI subdivide in the way you have said AI will subdivide.

Malcolm: He's actually like, look, that's not the way the human brain works. And I was like, it's exactly the way the human brain works. Like, are you not familiar with like the cerebellum? Like, sorry. For people who don't know, the cerebellum encodes things like juggling or dancing or like riding a bike and it encodes them in a completely separate part of the brain.

Malcolm: It's like rote motor tasks. But also the brain is actually pretty subdivided with different specialties and the human can change their mind because of this. And I actually asked him, I was like, okay. If you believe this so strongly, so what he believes is that AIs will all become just a single hierarchy, right? Malcolm: And that is why they can never change their, their utility function.

Is essentially no government structure ever created and has [] functioned that way. So it is very surprising. Malcolm: Once we find out, because we will get some more understanding into AI interpretability, into how AIs. If it turns out that the AI's that exist right now are actually structuring themselves that way, will you concede that you are wrong about the way that you tackle AI apocalypticism?

He's like, no, I won't because the simplistic AI's like the learning language models and stuff like that we have now they, they, they are not going to be like the AI's that kill us all and that those AI's. Will be you only get this this instrumental convergence when the A. Malcolm: I. And obviously I lose a lot of respect for someone when they are [] unwilling to create arguments that can be disproven.

I was also like, yeah, also, we could run experiments where we do a bunch of basically unbounded A. Malcolm: No, I guess you could say no, no, no, no. It still is logical and rational. And he is right that once they reach above this certain level of intelligence, but I, I believe very strongly that people should try to create little experiments in the world where they can be proven right or wrong based on additional information.

Malcolm: But yeah, okay. So there's that Simone, you wanted to say something? In fairness,. Simone: Yudkowsky said that he held the views that you, you once held when he was 19 years old and that we needed to read. His Zombies title work to see the step by step reasoning that he followed. Malcolm: So he kind of said that, but he was more, this was another interesting thing about talking to him, is I'm a little worried because we had talked down about him, you know, sort of secretly in a few videos that we've had, and it would be really sad if I met him and he turned out to actually be like really smart and upstanding and, and open minded.

Simone: Yes, compared to other people who were at the conference, such as Zvi Mosiewicz, who we, you know, respect deeply, and Bernd Hobart, and Richard Hanania, he, he definitely came across as less intelligent than I expected, and less intelligent than them. Mostly because for example Zvi also is extremely passionate. Simone: About AI. And he also extremely disagrees with us.

And we've had many debates with him, yeah. Yes but, you know, when, when he When he gets, when he disagrees with us, or when he hears views that, that he thinks are stupid which, you know, are our views, totally fine he gets exasperated, but enthusiastic, and then like sort of breaks it down as to why we're wrong, and sort of, sort of gets excited about, [] like, arguing a point, you know, and sort of seeing where there's the nuanced reality that we're not understanding, whereas the, the reaction that Yudkowsky had when you disagreed with him It came out more as offense or anger, which to me signals not so much that he was interested in engaging, but that he doesn't like people to disagree with him and he's not really interested in engaging.

Simone: Like it's either offensive to him, that is to say a threat to his worldview of him just sort of being correct on this issue as being the one who has thought about it the very most. Malcolm: This happened another time with you, by the way. Where you were having a conversation and he joined? Simone: it seems like a pattern of action of his, which, you know, many people do.

Simone: I, you know, we do, we do it sometimes is like, you know, walk by a conversation, come in and be like, Oh, well, actually it works like this. Malcolm: if somebody disagreed with him, like you did a few times, he would walk away evidence. He'd just [] walk away. Donate icon An illustration of a heart shape "Donate to the archive" User icon An illustration of a person's head and chest.