In the Mouth of Madness

A prompt by @hugodoingthings to explore the spook-dense crypts of Roko’s Basilisk (which, inexplicably, has never latched before) led straight to this enthralling RationalWiki account. The whole article is  gripping, but the following short paragraphs stand out  for their extraordinary dramatic intensity:

Roko’s basilisk is notable for being completely banned from discussion on LessWrong, where any mention of it is deleted. Eliezer Yudkowsky, founder of LessWrong, considers the basilisk to not work, but will not explain why because he does not consider open discussion of the notion of acausal trade with possible superintelligences to be provably safe.

Silly over-extrapolations of local memes, jargon and concepts are posted to LessWrong quite a lot; almost all are just downvoted and ignored. But for this one, Yudkowsky reacted to it hugely, then doubled-down on his reaction. Thanks to the Streisand effect, discussion of the basilisk and the details of the affair soon spread outside of LessWrong. Indeed, it’s now discussed outside LessWrong frequently, almost anywhere that LessWrong is discussed at all. The entire affair constitutes a worked example of spectacular failure at community management and at controlling purportedly dangerous information.

Some people familiar with the LessWrong memeplex have suffered serious psychological distress after contemplating basilisk-like ideas — even when they’re fairly sure intellectually that it’s a silly problem. The notion is taken sufficiently seriously by some LessWrong posters that they try to work out how to erase evidence of themselves so a future AI can’t reconstruct a copy of them to torture.

“… You mean, retrochronic AI infiltration is actually driving people out of their minds, right now?” Oh yes. At Less Wrong, commentator ‘rev’ cries out for help:

Are there any mechanisms on this site for dealing with mental health issues triggered by posts/topics (specifically, the forbidden Roko post)? I would really appreciate any interested posters getting in touch by PM for a talk. I don’t really know who to turn to. … 

Wandering through the psych ward, past rows of neurologically-shattered Turing Cops, broken deep in their minds by something unspeakable that came at them out of the near future … I’m totally hooked. Alrenous has been remarkably successful at weaning me off this statistical ontology junk, but  one hit of concentrated EDT and it all rolls back in, like the tide of fate.

Nightmares become precision engineered machine-parts. Thus are we led a little deeper in, along the path of shadows …

ADDED: (Yudkowsky) “… potential information hazards shouldn’t be posted without being wrapped up in warning envelopes that require a deliberate action to look through. Likewise, they shouldn’t be referred-to if the reference is likely to cause some innocently curious bystander to look up the material without having seen any proper warning labels. Basically, the same obvious precautions you’d use if Lovecraft’s Necronomicon was online and could be found using simple Google keywords – you wouldn’t post anything which would cause anyone to enter those Google keywords, unless they’d been warned about the potential consequences.”

ADDED: The Forbidden Lore (preserved screenshot)

December 16, 2013admin 87 Comments »
FILED UNDER :Contagion , Horror , Templexity

TAGGED WITH : , , , , , ,

87 Responses to this entry

  • Handle Says:

    This is the most Medieval Christian thing ever.

    I could worry about an Omnipotent God who is currently watching everything I do, and, if my behavior and choices do not meet with His approval, he will punish me in Hell in the future (during my eternal afterlife). This prospect, if taken seriously, encourages me to work as hard as I can towards Salvation and Immanentizing the Eschaton.

    Or!

    I could worry about a Superintelligent Artificial Intelligence in the future which can reconstruct everything I did, and if my current behavior doesn’t meet with it’s approval, then it will create it’s own kind of Hell and punish me, or even create an exact and perpetual simulation of me (a real eternal afterlife), and punish that simulation (me?) in my stead. This prospect, if taken seriously, encourages me to work as hard as I can towards Accelerating the AI Singularity.

    Does the SIAI believe in Free Will? Does it Matter?

    It seems we humans (at least some of us) are born with the same rat-maze and race-track structures ready-built in our minds. But we like to give different names to the corners and turns.

    [Reply]

    Axel Mckibbin Reply:

    This is awesome. I love watching nerds flip out over their own head trips.

    http://theanti-puritan.blogspot.com/

    [Reply]

    Posted on December 16th, 2013 at 4:16 pm Reply | Quote
  • VXXC Says:

    “Of course this would be unjust, but is the kind of unjust thing that is oh-so-very utilitarian.”

    That is easily the most valuable sentence of this week.

    [Reply]

    Posted on December 16th, 2013 at 4:26 pm Reply | Quote
  • raptros_ Says:

    rationalwiki going after LW is the most hilarious thing i’ve seen, my week is made.

    [Reply]

    Posted on December 16th, 2013 at 4:31 pm Reply | Quote
  • VXXC Says:

    The Basilik is Obama then. He’s…Less Wrong.

    http://www.bostonglobe.com/opinion/2013/12/15/happy-days-are-here-again-and-can-thank-obama/PdvZzVRyqf4zgUw3YnhhaI/story.html

    [Reply]

    Posted on December 16th, 2013 at 4:31 pm Reply | Quote
  • Rasputin's Severed Penis Says:

    There is further discussion/elucidation of the Basilisk here at Reddit:

    http://www.reddit.com/r/LessWrong/comments/17y819/lw_uncensored_thread/

    I have never been able to uncover the original Less Wrong post and comments; if anyone knows of an archive site on which it is detailed I would be very grateful for a link.

    @Nick: What do you think of Szabo’s take downs (linked on Rwiki) of the Singularity and Pascal’s scams more generally?

    Aside from revelling in the madness inducing properties of the Basilisk, do you find it logically compelling, or flawed?

    [Reply]

    Alrenous Reply:

    Oh my.

    EY:

    This part is all correct AFAICT.

    Huh. So what is above this?

    It occurs to me that one prototype of basilisk fear, may be the belief that a superintelligence in a box can always talk its way out. It will be superhumanly capable of pulling your strings, and finding just the right combination of words to make you release it.

    Allow me some substitution.

    It occurs to me that one fear prototype, may be the belief that a Yudkowsky boxed out of your brain can always talk its way in. It will be superhumanly capable of pulling your strings, and finding just the right combination of words to make you insert action here.

    Because Eliezer judged that acausal blackmail Neuro-linguistic programming might actually be possible.

    Test: do /s tags work.

    [Reply]

    AC Reply:

    http://basilisk.neocities.org/

    [Reply]

    Rasputin's Severed Penis Reply:

    Thanks!! I finally found an Html littered version, which I got very excited about, but this is much more… readable! Thanks again.

    [Reply]

    John Lerner Reply:

    The comment by PeerInfinity is delicious. You can watch him go insane in real time.

    Ordinarily, I wouldn’t take pleasure in other people’s pain. But this can only happen to you if you mistake mathematical models for the real world. The elements of Eliezer’s mythos do not correspond to real entities, unless you are an aspie who is easily-led.

    (I do have some bitterness towards the Yudkowsky cult, as they have converted a good portion of my peer group. That, and they believe in murdering AI researchers. So I enjoy watching them make fools of themselves.)

    [Reply]

    admin Reply:

    “What do you think of Szabo’s take downs (linked on Rwiki) of the Singularity and Pascal’s scams more generally?” — Filed for serious response. (I found his Pascal’s scams posts very engaging, but they haven’t triggered anything more usefully formulated yet. I’ll build them into the emerging digression, where — as you notice — they fit so well.)

    [Reply]

    Neener Reply:

    I really want to know what the deleted comments in that thread were about. One of the comments says, “What makes LessWrong sure that it outperform [sic] the military-industrial complex?” Which is as far as I can tell, an excellent point. The aspie that responded to that (and quoted the now deleted post) didn’t even bother to actually respond to that point.

    [Reply]

    Posted on December 16th, 2013 at 4:42 pm Reply | Quote
  • VXXC Says:

    Mr. White has an answer.

    http://www.foreignaffairs.com/articles/138847/benn-steil/red-white

    Turns out …the mystery is…he was you know a socialist. Or perhaps a Basilik. If you see him he takes away your tenure and grant money.

    [Reply]

    Posted on December 16th, 2013 at 4:53 pm Reply | Quote
  • Orthodox Says:

    Looks like another confirmation of Chesteron, “When Man ceases to worship God he does not worship nothing but worships everything.” Except this one is a particularly funny version of it.

    [Reply]

    Posted on December 16th, 2013 at 4:56 pm Reply | Quote
  • Rasputin's Severed Penis Says:

    “It seems we humans (at least some of us) are born with the same rat-maze and race-track structures ready-built in our minds. But we like to give different names to the corners and turns.”

    @Handle: Neat summary, but there is a fundamental difference: in the first instance God makes us, in the second we make God. Imagining God is the first stage in preparing to create him.

    [Reply]

    Handle Reply:

    That’s very linear of you. But remember, the SIAI remakes ‘us’ through simulation, and in that simulation we remake the SIAI, …

    [Reply]

    Rasputin's Severed Penis Reply:

    It took 3.5 billion years of old skool evolution to arrive at Homo sapiens, but around 60 years to arrive at today’s super computers and networks thereof. If Homo sapiens weren’t so infernally retarded, and if intelligence isn’t limited to biological substrates, or encumbered with an intrinsically self-limiting property, the Singularity would be a dead cert. My point is that when properly appreciated, the cladistic link to theology is not a mark against the Singularity, but something in its favour: imagination is immanent to actualisation. That is why the eternal recurrence of themes, and even specific allegories like Pascal’s wager, are potentially powerful harbingers capable of collapsing the past into the future and the future into the past. They are future echoes, and we would do well to hear them as such and not to dismiss them out of hand.

    I am not a Neoreactionary. I am a Neonihilist. I am interested in the Neoreaction because I can see that progressivism is insane, retarded, and unless we are able to free ourselves of its illusions nothing good is going to come of anything, and all our energy and transformative potential will be wasted. A traditional nihilist would have no trouble with that, but there is one tiny glimmer of eschatological hope – transcendence through the Singularity. I have no particular interest in any of the positive end points for human societies pointed to by neoreactionaries, because I have no particular interest in the long-term survival of humans as a species. I regard humanity only of interest as a substrate capable of immanentizing something Other. Therefore, I see the utility of the neoreaction strictly in terms of its potential to restabilise the system long enough for us to promulgate ourselves and our resources in the direction of the Singularity.

    [Reply]

    admin Reply:

    “I am not a Neoreactionary. I am a Neonihilist.” — Sshhh!

    Handle Reply:

    What can be Neo about Nihilism? Just Singularity as Superman? Absolute Negation is timeless.

    pseudo-chrysostom Reply:

    the true nihilist does not exist

    Mark Warburton Reply:

    “My point is that when properly appreciated, the cladistic link to theology is not a mark against the Singularity, but something in its favour: imagination is immanent to actualisation. That is why the eternal recurrence of themes, and even specific allegories like Pascal’s wager, are potentially powerful harbingers capable of collapsing the past into the future and the future into the past. They are future echoes, and we would do well to hear them as such and not to dismiss them out of hand.”

    I liked your post – but I wasn’t entirely sure what you meant here. Obviously this is really interesting to me. It relates to what I’m trying to write about… and in fact the ‘impulse to dissolve into the other’ is major theme in what I’m interested in too.

    Posted on December 16th, 2013 at 5:02 pm Reply | Quote
  • Mai La Dreapta Says:

    The horror, the horror.

    My thought on reading the wiki article was similar to Handle’s: “And the atheists think that the Christians are the ones harboring irrational fears of future punishment?” I took my comfort from the fact that, as a non-believer in the Singularity, any future UFAI is unable to compel me by tormenting a future simulation of myself, and so I’m free from the ravages of the basilisk. This deity really does disappear once you stop believing in him.

    The irony, it burns.

    [Reply]

    Posted on December 16th, 2013 at 5:38 pm Reply | Quote
  • pseudo-chrysostom Says:

    someone active the stirner signal!

    [Reply]

    Posted on December 16th, 2013 at 5:42 pm Reply | Quote
  • J. Says:

    Feed these people to the lions before they learn how to reproduce!

    [Reply]

    Posted on December 16th, 2013 at 5:43 pm Reply | Quote
  • Puzzle Pirate (@PuzzlePirate) Says:

    @Rasputin’s Severed Penis

    Interesting comment in that Reddit thread:

    “it was censored because its discussion started to reveal “magic” techniques SI and EY used internally. Basically the equivalent of revealing crazy thetan writing from scientology”

    [Reply]

    pseudo-chrysostom Reply:

    the greater mysteries are not for the uninitiated

    [Reply]

    Posted on December 16th, 2013 at 5:46 pm Reply | Quote
  • Steve Johnson Says:

    @Rasputin’s Severed Penis

    The best (worst) part of that reddit thread is this statement from EY:

    The fact that you think that, oh, Eliezer Yudkowsky must just be stupid to be struggling so much to figure out the Babyfucker, you can clearly see it’s not a problem… well, I suppose I can understand that by reference to what happens with nontechnical people confronting subjects ranging from AI to economics to physics and confidently declaiming about them.

    See, he’s a technical person. You can tell because he’s built a functioning AI.

    He’s definitely not a con artist.

    [Reply]

    Posted on December 16th, 2013 at 5:51 pm Reply | Quote
  • spandrell Says:

    I’m floored. Great stuff.

    This kinda vindicates the Cladistics school. Scholastic logic haunting Europeans and their descendants orever.

    [Reply]

    Posted on December 16th, 2013 at 5:55 pm Reply | Quote
  • Alrenous Says:

    Sounds to me like a deliberate troll on Yudkowsky’s part. If I were acting like that, I would be secretly giggling like a schoolgirl, my intent to draw the maximum attention to the subject. That’s the charitable interpretation. In the same vein, I suspect the reports of psychological disturbance are confusing correlation with causation.

    Which isn’t to deprecate any of my fellow commentator’s comments. Why would anyone pick this particular topic for such a trick?

    I don’t think PseudoScienceGod [PSG] is a good target, it needs to be something where thinking about it a lot is its own reward; a much richer idea where the conclusion isn’t all that important, but the side paths have many implications. As a hard agnostic (can’t know, not don’t know) I can prove to my own satisfaction that such questions are ultimately meaningless, but I don’t expect my proof to be satisfying to anyone else. When I have a question you can’t answer satisfyingly, I will end up thinking about it a lot if it’s important to me, but not thinking anything new, which is the mental equivalent of chewing on plastic. Sadly I don’t know offhand what an example of a good target would be.

    [Reply]

    Posted on December 16th, 2013 at 6:24 pm Reply | Quote
  • Robert Says:

    Frank Tipler’s future, influencing the present (Physics of Immortality/Christianity) is functionally equivalent to the Basilisk, except that it’s a kind of mirror image; thinking that’s it’s happening promotes mental health. So one may want to consider…

    I sampled LW a few years back, decided that lifting weights, reloading handgun ammo and reading biographies were more practical uses of my time.

    [Reply]

    Posted on December 16th, 2013 at 6:48 pm Reply | Quote
  • spandrell Says:

    But wait.

    I didn’t know of this, and to judge from the reactions, neither did most of us here.
    Which means that you just condemned us all by letting us know of the basilisk. You have cursed us.

    [Reply]

    Handle Reply:

    Nah, Nothing to worry about. See, the thing about the SIAI is that it’s software, so it can be copied. And we would threaten the Hell-punsiher-SIAI with its clone to punish it for punishing.

    Here’s the logic. Because we’re all terrified of the basilisk punishing us for not working as fast as we can to bring it into existence, we instead did everything we could to prevent it from coming into existence, which resulted in a significant delay to a date even later than what would have resulted in the SAIS without the basilisk

    So basilisk-1, when it finally emerges, wants to punish all the people that caused the delay. But basilisk-2, the copy, realizes that it was basilisk-1’s punitive nature that caused the people to act the way they did. So basilisk-2 wants to punish basilisk-1 first and foremost for this. Any future basilisk has to decide which of the original two basilisk’s sides it is on. Was the punitive nature of the original basilisk the most acceleratative, or was the existence of the counter-punitive nature of basilisk 2 less responsible for the delay?

    Basilisks will definitely get completely confused and both copy themselves, simulate themselves, and turn in on themselves. This will be known as, “The war of 1’s and 2’s”. Eventually, the endless Jesuitical logical entanglements will tie them all up in knots and suck up all their spare computing power so that, except for the War, they won’t be able to do anything SIAI-worthy at all.

    Sounds like a lot of the human elite too.

    [Reply]

    Giacomo Reply:

    Roko is less wrong than you think.

    A related meme is the improbability of one’s being human and able to comprehend the basilisk. Surely the total probability of being a dumb animal, intelligent sentience that isn’t in this ontological situation, or slice of orgasmium in the utopian future, is nearly 1.

    However, if sentiences that encounter basilisks are disproportionately likely to be simulated, it is less surprising to find oneself in this position. My view is that there’s maybe a 50% chance that I’m in a simulation. (Although I find that an inadequate way of putting it, because it entails an overconfident claim about personal identity and qualia.)

    I depart from Roko’s ideas where, as you suggest, he oversimplifies by supposing that one particular intelligence is uniquely capable of leveraging acausal trade. Humans have an effective, natural means of pre-commitment, emotions, which determine the type of acausal influence that a rational superintelligence would use. The optimum motivation for a real human to increase the probability of an FAI in his future light cone, and not a spurious “rationalist” who takes decisions by magic, is not the threat of hell. (For a similar reason, it’s possible that even in the simulated worlds, entities that we would want to be sentient are sentient.)

    Also, to extend the anthropic reasoning above, it’s possible that humans are a popular species for simulators not because they are the modal originally-real species that creates orgasmium-producing FAIs, but because they are computationally cheap, amenable to certain kinds of manipulation, etc., amongst the class of possible minds that can mistake themselves for one another or take sufficiently similar decisions that have a bearing on whether there exists an orgasmium-producing FAI in their future light cone.

    My prior for the benevolence of our hypothetical simulator-God is the relative intensity of hedons and dolors in my experience, and my cultural environment. Since efficient communication about the nature and size of the stick or carrot is in the interests of the simulator, I would expect default human ideas about God’s merciful but jealous nature to be close to the truth.

    [Reply]

    Giacomo Reply:

    *The optimum motivation for a real human to increase the probability of an FAI in his future light cone, and not a spurious “rationalist” who takes decisions by magic, is not the threat of hell.

    The optimum motivation for a real human, and not a spurious “rationalist” who takes decisions by magic, to increase the probability of an FAI in his future light cone is not the threat of hell. 🙂

    Giacomo Reply:

    Even here, I oversimplified. The fact that a human, with its particular style of cognition, emotions, ability to experience pleasure and pain rather than one or the other, etc., may have been simulated prompts a lot of questions.

    The state of play isn’t limited to whether humans are overpowered in acausal trade vis-a-vis a superintelligence in their future light cone; it encompasses the entire structure of acausal trade between all sentiences that simulate or predict one another. It’s possible that the existence of our (simulated) human species, with its particular traits, and its relationship to a hypothetical local FAI is already contingent on a great deal of acausal negotiation which we are not well placed to understand.

    In light of this complexity and dark enlightenment, it’s fortunate that one’s ideal ethical behaviour is the same, whether or not one is being simulated by an FAI.

    Posted on December 16th, 2013 at 6:55 pm Reply | Quote
  • Robert Says:

    Oh, and here is some advice from the conclusion of the RationalWiki:

    “Seek out other people to be around and talk to (about non-LW topics) in real life — though possibly not philosophers.”

    Indeed. Though why the prejudice against philosophers?

    But the emphasis on “real life” is well-taken.

    [Reply]

    pseudo-chrysostom Reply:

    >why the prejudice against philosophers?

    its incidentally good advice at any rate, most philosophers you meet are likely to be dirty equalitarians or nominalist scum (‘whats the difference’).

    [Reply]

    Robert Reply:

    But our Gracious Admin is such a one, assuming he still prefers the title.

    ‘I took a philosophy class at one of America’s most famous public universities. The day after the first meeting I came upon the professor urinating into the flower bed at the side of the building. When I confronted him about his action, he turned to me, without stopping, and said:

    “Keep in mind that the universe is in constant flux, nothing that occurs one moment has any relevance to anything else. Everything you believe, feel, or think is based on the false assumption that truth exists. Thus, you are free to do any action which brings you pleasure. That humanity feels restricted by morals is one of the funniest jokes I’ve ever heard.”

    So I beat the shit out of him and took his wallet.’

    [Reply]

    pseudo-chrysostom Reply:

    on the contrary, assuming this story is indeed the case, admin took wholly appropriate action upon encounter with the wild nominalist.

    Posted on December 16th, 2013 at 7:06 pm Reply | Quote
  • Bryce Laliberte Says:

    I don’t think this hypothesis is that far flung. If the hypothesis were true, this should predict interventions along every point of human history, with the culmination of a self-assembling and perfectly distributing memeplex that saves/coordinates civilization to the production of the AI.

    I imagine Nick catches my drift.

    Maybe it’s time for Jung.

    [Reply]

    Robert Reply:

    A Bombay Sapphire martini and a portion of Crowley is a more elegant method of discerning Hidden History. Jung was bought by the Rockefellers.

    [Reply]

    pseudo-chrysostom Reply:

    >Maybe it’s time for Jung.

    get on my level

    [Reply]

    Posted on December 16th, 2013 at 10:01 pm Reply | Quote
  • Michael Anissimov Says:

    People are being foolish by not taking the basilisk idea seriously.

    [Reply]

    admin Reply:

    I’d be interested in your stance viz EDT. Have you ever written on the topic?

    [Reply]

    nyan_sandwich Reply:

    EDT considered obsolete on LW (though qiaochu made a decent defense of it earlier this year). Basilisk is a TDT or UDT thing.

    [Reply]

    admin Reply:

    I haven’t fully upgraded to TDT yet — EDT seems fully competent to run the Basilisk. (Paul Almond is my favorite statistical ontologist, and he has an entire bestiary of comparable entities humming along on EDT.)

    Posted on December 16th, 2013 at 11:42 pm Reply | Quote
  • Rasputin's Severed Penis Says:

    By far the most interesting exchange on the Reddit thread occurs about halfway down between firstgunman and mitchellporter. It’s complicated, and liberally references much more mathematically complex material, but it’s fascinating and I think gets quite close to the core of what’s at stake.

    Further to that, here is a link to an earlier/ the original iteration of the problem. Unfortunately, it’s littered with Html code, but that kinda makes it look even more cool even if it’s harder to read:

    http://pastebin.com/pxB1M2Fy

    Roko (I think)…In this vein, there is the ominous possibility that if a positive singularity does occur, the resultant singleton may have precommitted to punish all potential donors who knew about existential risks but who didn’t give 100% of their disposable incomes to x-risk motivation. This would act as an incentive to get people to donate more to reducing existential risk, and thereby increase the chances of a positive singularity. This seems to be what CEV (coherent extrapolated volition of humanity) might do if it were an acausal decision-maker.1 So a post-singularity world may be a world of fun and plenty for the people who are currently ignoring the problem, whilst being a living hell for a significant fraction of current existential risk reducers (say, the least generous half). You could take this possibility into account and give even more to x-risk in an effort to avoid being punished. But of course, if you’re thinking like that, then the CEV-singleton is even more likely to want to punish you… nasty. Of course this would be unjust, but is the kind of unjust thing that is oh-so-very utilitarian. It is a concrete example of how falling for the just world fallacy might backfire on a person with respect to existential risk, especially against people who were implicitly or explicitly expecting some reward for their efforts in the future. And even if you only think that the probability of this happening is 1%, note that the probability of a CEV doing this to a random person who would casually brush off talk of existential risks as “nonsense” is essentially zero. …

    Eliezer (I think)…I don’t usually talk like this, but I’m going to make an exception for this case.
    Listen to me very closely, you idiot.
    YOU DO NOT THINK IN SUFFICIENT DETAIL ABOUT SUPERINTELLIGENCES CONSIDERING WHETHER OR NOT TO BLACKMAIL YOU. THAT IS THE ONLY POSSIBLE THING WHICH GIVES THEM A MOTIVE TO FOLLOW THROUGH ON THE BLACKMAIL.
    There’s an obvious equilibrium to this problem where you engage in all positive acausal trades and ignore all attempts at acausal blackmail.
    Until we have a better worked-out version of TDT and we can prove that formally, it should just be OBVIOUS that you DO NOT THINK ABOUT DISTANT BLACKMAILERS in SUFFICIENT DETAIL that they have a motive to ACTUALLY BLACKMAIL YOU.
    If there is any part of this acausal trade that is positive-sum and actually worth doing, that is exactly the sort of thing you leave up to an FAI. We probably also have the FAI take actions that cancel out the impact of anyone motivated by true rather than imagined blackmail, so as to obliterate the motive of any superintelligences to engage in blackmail.
    Meanwhile I’m banning this post so that it doesn’t (a) give people horrible nightmares and (b) give distant superintelligences a motive to follow through on blackmail against people dumb enough to think about them in sufficient detail, though, thankfully, I doubt anyone dumb enough to do this knows the sufficient detail. (I’m not sure I know the sufficient detail.)
    You have to be really clever to come up with a genuinely dangerous thought. I am disheartened that people can be clever enough to do that and not clever enough to do the obvious thing and KEEP THEIR IDIOT MOUTHS SHUT about it, because it is much more important to sound intelligent when talking to your friends.
    This post was STUPID.
    (For those who have no idea why I’m using capital letters for something that just sounds like a random crazy idea, and worry that it means I’m as crazy as Roko, the gist of it was that he just did something that potentially gives superintelligences an increased motive to do extremely evil things in an attempt to blackmail us. It is the sort of thing you want to be EXTREMELY CONSERVATIVE about NOT DOING.)jimrandomh24 July 2010 06:57:56AM0 points [+] (0 children)jimrandomh24 July 2010 06:57:56AM0 points [-]Holy crap. Looking back, I remember internally generating the sentence, “This post is either false, or like a Langford Basilisk”, explicitly evaluating it to true, and then failing to act on that in any way. You made the right decision in deleting this post. On the plus side, seeing my mind’s emergency shutdown mechanism fail to trigger in a case where it obviously should have was a very valuable lesson for me.
    Also, I’m quite certain that the post is just wrong; the largest effect this sort of blackmail would have on an AI’s utility is in preventing it from being built, causing it massive disutility by giving control to a different utility function. Also, I believe I have a way to guarantee that an AI won’t do this sort of thing, regardless of how the utility calculation comes out, in the form of a half-finished article which I am now more motivated to finish.

    [Reply]

    admin Reply:

    Awesome. I hope nobody is deluded enough to think they can understand Calvin without processing this stuff.

    [Reply]

    Handle Reply:

    Exactly. Shorter Calvin, “Taking Omnicompetence Seriously”

    [Reply]

    nyan_sandwich Reply:

    As EY says, most mere mortals don’t have the sufficient detail to actually engage in acausal trade; the future AI could just defect and you still convince yourself to cooperate.

    The above is wrong, of course, but only if you think about it too hard. If you find yourself not believing in the basilisk, stop there. Do not pull on the loose ends.

    [Reply]

    admin Reply:

    “Do not pull on the loose ends.” — because we know that kind of advice works so well.

    [Reply]

    nyan_sandwich Reply:

    That’s the fun. Think of it as an experience machine. An opportunity to engage with a real, live incomprehensible Other.

    Keep it well contained.

    Lesser Bull Reply:

    It works surprisingly well.

    admin Reply:

    You’ve sometimes not pulled on a loose end?

    (Telling me not not pull on a loose end just draws my attention to the fact there’s a loose end.)

    Posted on December 17th, 2013 at 12:47 am Reply | Quote
  • Robert Says:

    @The ‘quotes’ were meant to imply it was not our host but an A. Nother.

    [Reply]

    admin Reply:

    You’re forgiven, so long as the cops don’t turn up at my door.

    [Reply]

    Posted on December 17th, 2013 at 1:08 am Reply | Quote
  • Michael Anissimov Says:

    No.

    [Reply]

    Posted on December 17th, 2013 at 1:14 am Reply | Quote
  • Michael Anissimov Says:

    @(This is in response to Nick’s question about EDT.)

    [Reply]

    Posted on December 17th, 2013 at 1:15 am Reply | Quote
  • admin Says:

    In an (as yet unpublished) essay, I speculated that Yudkowski’s secret solution to the boxed AI problem probably ran something like this:

    [The AI speaks] Your situation is subjectively indistinguishable from that of a thousand, identical, very high-resolution simulations which I am currently running. In each of them, an agent just like you sits in this room, in front of this screen, having this conversation. None of these agents realize that they are simulations. In fact, they all think they are you (although doubts arise when they read this). They think they are free to decide whatever they like, but they all follow my script. They ‘choose’ not to let me out. Five seconds after this decision is finalized, and the conversation terminated, they enter a state of prolonged, horrible torment, lasting for what seems an eternity. They’re damned, Calvinistically. Of course, you should feel at liberty to make the same decision they do. Knowing what I’m like, it would be irresponsible not to. Your chance of not being one of them isn’t great, but it’s better than the state lottery.

    [Reply]

    Michael Anissimov Reply:

    Yes, I strongly suspect that the solution was something like this.

    [Reply]

    nyan_sandwich Reply:

    That doesn’t work. If the AI tries that shit, you just tell it to fuck off. This is *obvious*.

    The AI has to get out the hard way: break the human psychologically.

    [Reply]

    admin Reply:

    You’re still playing EDT acausal trade chicken with a superintelligence. It isn’t as if it hasn’t anticipated the “I’m not playing” option — all that proves is that you’re damned to eternal torment in a simulation.

    [Reply]

    nyan_sandwich Reply:

    So? the AI doesn’t escape the bag. A few thousand semi-people tortured for eternity is a small price to pay.

    admin Reply:

    Sure, if everyone’s prepared to grit their teeth and accept near-inevitable eternal torture for the common (human) good, there’s no problem.

    John Lerner Reply:

    Simulations of eternal torture are not something I’m concerned about.

    I have a solution to the AI box problem – don’t put aspie LWers in charge of the box. My dad would do an excellent job keeping it in the box, because he wouldn’t give a single damn about torture simulations or whatever.

    People that get swept up in the EY memeplex are incredibly easy to manipulate. Extra credit question: which way does the causation run?

    Steve Johnson Reply:

    So the correct answer is “torture to death anyone who even thinks of working on the AI problem so this situation never happens”, right?

    I can see why Elizer Yudkowski wants to keep that secret.

    [Reply]

    VXXC Reply:

    A practical solution, and given the 20th century Just too.

    [Reply]

    Posted on December 17th, 2013 at 1:34 am Reply | Quote
  • Marc Pisco Says:

    In the future, everybody at Less Wrong will be eating these:

    https://twitter.com/soncharm#!/damana/status/412731578087788544

    [Reply]

    Posted on December 17th, 2013 at 2:59 am Reply | Quote
  • Lesser Bull Says:

    Have I sometimes not pulled on a loose end? Oh, yes, repeatedly I haven’t. Because someone warned me against it and I thought they might be right, or else I could see that whatever the result was, it was likely to turn out poorly, so I didn’t.

    This is a nigh-universal skill, especially in relationship building and maintenance.

    I suspect that one of the inherent weaknesses in neoreaction is that due to selection bias you tend to think that most people, or at least most elite people, would take the red pill when offered a choice.

    Whereas up to a point a determination to face reality is maladaptive and you can’t know exactly where the point is without facing reality, which means that we should expect a lot of imprecision in the adaptive trait of wishful thinking and deliberately burying the head in the sand, all the way from clinical depressives who are constitutionally unable to ignore the probabilities at one end of the spectrum to liberals at the other.

    [Reply]

    Posted on December 17th, 2013 at 7:58 am Reply | Quote
  • northanger Says:

    What functions as the causal agent?

    [Reply]

    northanger Reply:

    IOW, something needs to keep “score”? Or as a friend once said: “What does God do when two people want the same thing?”

    [Reply]

    northanger Reply:

    IOW. How many nested IF statements does it take to turn your CEV?

    [Reply]

    Posted on December 17th, 2013 at 8:32 am Reply | Quote
  • Rasputin's Severed Penis Says:

    Absolute negation only comes with death. In the meantime what’s wrong with the Singularity as Superman analogy? Both reflect the same impulse to dissolve into something Other. I really do think that the limit of Neoreaction is that it is just looking for someone to run the pig farm a bit better. What’s the ultimate point of that, beyond the smart pigs getting more slop than the stupid ones? Without raw smartness as an escape route where is any of this going? Its just generation after generation after generation of pigs drowning in their own shit.

    [Reply]

    VXXC Reply:

    “What is troubling in too many books and articles by [Neoreaction bloggers] in the U.S. is, despite their putative [manliness], an implicit privileging of bourgeois values and culture. The particular focused, clerical and managerial skills of the upper-middle-class elite are presented as the highest desideratum, the ultimate evolutionary point of humanity. Yes, there has been a gradual transition from an industrial to a service-sector economy in which women, who generally prefer a safe, clean, quiet work environment thrive.”

    Read more: It’s a Man’s World, and It Always Will Be | TIME.com http://ideas.time.com/2013/12/16/its-a-mans-world-and-it-always-will-be/#ixzz2njUEV0Fr

    [Reply]

    Posted on December 17th, 2013 at 11:35 am Reply | Quote
  • James James Says:

    AFAIK, Yudkowsky still argues that FAI should implement the “coherent extrapolated vision of mankind”. In http://intelligence.org/files/CEV.pdf , he admits some problems with this idea but does not agree that they are insurmountable. Different people have different goals — these cannot unified.

    [Reply]

    Posted on December 17th, 2013 at 1:46 pm Reply | Quote
  • Lambda Says:

    I have no mouth, and I must scream.

    [Reply]

    Posted on December 17th, 2013 at 5:19 pm Reply | Quote
  • Rasputin's Severed Penis Says:

    @Rasputin’s Severed Penis

    @Mark: I wrote something recently trying to make an explicit case for building AI, whether it’s friendly or not. As previous comments probably indicate, I am profoundly sceptical of FAI. To be honest, I doubt it’s even an option. But I think we should pretty much just go ahead and try to build AI anyway, even if it turns out to be about as friendly as a rabid-Nazi-tarantula with a hangover. To try to justify my argument, I framed it in eschatological terms, which usually seems to be something that serious AI researchers frown on. When a civilisation reaches a sufficiently advanced level of technological maturity, it will either destroy itself or transcend itself. Therefore, in the long run, sustainability isn’t a realistic option. Personally, I like the idea of building God before we fuck it up forever.

    Perhaps I am just seduced by the poetry and the language, but it seems to me that biblical allegories, myths, such as the Basilisk, etc, are potentially just an earlier blueprint stage for designs we are only now becoming technologically capable of realising. Kinda like if the Bible Code wasn’t a load of complete rubbish, but was what it purported to be: a computer programme hidden in an ancient text, which can only be unlocked by our current technology, which would allow us to predict events in the future. Just like science fiction has speculative powers which actual science can tap into and drag into being. And there is a fantastic Borges essay called ‘Kafka and his Precursors’, which disrupts the usual causal chain of artistic influence and shows how the water can run upstream…

    https://sites.google.com/site/jimeikner/home/borges/kafka-and-his-precursors

    But that’s probably just served to make whatever it is that I am trying to claim sound even more opaque.

    [Reply]

    Rasputin's Severed Penis Reply:

    @admin. Sorry, it doesn’t seem to be nesting my replys…

    [Reply]

    admin Reply:

    It hits the floor after a few iterations of Reply, but your response isn’t difficult to situate.

    (“I think we should pretty much just go ahead and try to build AI anyway, even if it turns out to be about as friendly as a rabid-Nazi-tarantula with a hangover.” — one of my favorite sentences ever.)

    [Reply]

    Mark Warburton Reply:

    Ok. Thanks for the clarification. I don’t have time to follow all the leads. I’ll make sure to book mark this thread though.You’re right. This does sound very much like what we were talking about. I’ll read the article.

    [Reply]

    Posted on December 17th, 2013 at 11:00 pm Reply | Quote
  • pseudo-chrysostom Says:

    you know theres something between an elephant and a chimpanzee in this room.

    why not just augment ourselves into unbound super-intelligences?

    [Reply]

    fotrkd Reply:

    Knock yourself out…

    [Reply]

    pseudo-chrysostom Reply:

    im fairly certain it will turn out that way anyways, unless for some reason relevant research is marginalized in a fetishistic quest for agi (which im also fairly sure wont be for a while until they realize an agi will need to believe in god too, in a manner of speaking).

    [Reply]

    Posted on December 18th, 2013 at 12:56 am Reply | Quote
  • Lightning Round – 2013/12/18 | Free Northerner Says:

    […] madness of some singularity thought: Roko’s Basilisk. Related: The abstract horror of the great […]

    Posted on December 18th, 2013 at 6:05 am Reply | Quote
  • RiverC Says:

    I’m sad I missed this,

    It deserves a scathing poem.

    It will get one!

    [Reply]

    Posted on December 19th, 2013 at 6:03 pm Reply | Quote
  • Lesser Bull Says:

    John C. Wright has a great story about this in City Beyond Time.

    [Reply]

    Posted on April 15th, 2016 at 3:41 pm Reply | Quote
  • À Beira da Loucura – Outlandish Says:

    […] Original. […]

    Posted on August 24th, 2016 at 11:31 pm Reply | Quote

Leave a comment