A while ago Nyan posed a series of questions about the XS rejection of (fact-value, or capability-volition) orthogonality. He sought first of all to differentiate between the possibility, feasibility, and desirability of unconstrained and unconditional intelligence explosion, before asking:

On desirability, given possibility and feasibility, it seems straightforward to me that we prefer to exert control over the direction of the future so that it is closer to the kind of thing compatible with human and posthuman glorious flourishing (eg manifest Samo’s True Emperor), rather than raw Pythia. That is, I am a human-supremacist, rather than cosmist. This seems to be the core of the disagreement, you regarding it as somehow blasphemous for us to selfishly impose direction on Pythia. Can you explain your position on this part?

If this whole conception is the cancer that’s killing the West or whatever, could you explain that in more detail than simply the statement?

(It’s worth noting, as a preliminary, that the comments of Dark Psy-Ops and Aeroguy on that thread are highly-satisfactory proxies for the XS stance.)

First, a short micro-cultural digression. The distinction between Inner- and Outer-NRx, which this blog expects to have settled upon by the end of the year, describes the shape of the stage upon which such discussions unfold (and implex). Where the upstart Inner-NRx — comparatively populist, activist, political, and orthogenic — aims primarily at the construction of a robust, easily communicable doctrinal core, with attendant ‘entryism’ anxieties, Outer-NRx is a system of creative frontiers. By far the most fertile of these are the zones of intersection with Libertarianism and Rationalism. One reason to treasure Nyan’s line of interrogation is the fidelity with which it represents deep-current concerns and presuppositions of the voices gathered about, or spun-off from, LessWrong.

Among these presuppositions is, of course, the orthogonality thesis itself. This extends far beyond the contemporary Rationalist Community, into the bedrock of the Western philosophical tradition. A relatively popular version — even among many who label themselves ‘NRx’ — is that formulated by David Hume in his A Treatise on Human Nature (1739-40): “Reason is, and ought only to be the slave of the passions, and can never pretend to any other office than to serve and obey them.” If this proposition is found convincing, the Paperclipper is already on the way to our nightmares. It can be considered an Occidental destiny.

Minimally, the Will-to-Think describes a diagonal. There are probably better ways to mark the irreducible cognitive-volitional circuit of intelligence optimization, with ‘self-cultivation’ as an obvious candidate, but this term is forged for application in the particular context of congenital Western intellectual error. While discrimination is almost always to be applauded, in this case the possibility, feasibility, and desirability of the process are only superficially differentiable. A will-to-think is an orientation of desire. If it cannot make itself wanted (practically desirable), it cannot make itself at all.

From orthogonality (defined negatively as the absence of an integral will-to-think), one quickly arrives at a gamma-draft of the (synthetic intelligence) ‘Friendliness’ project such as this:

If you offered Gandhi a pill that made him want to kill people, he would refuse to take it, because he knows that then he would kill people, and the current Gandhi doesn’t want to kill people. This, roughly speaking, is an argument that minds sufficiently advanced to precisely modify and improve themselves, will tend to preserve the motivational framework they started in. The future of Earth-originating intelligence may be determined by the goals of the first mind smart enough to self-improve.

The isomorphy with Nyan-style ‘Super-humanism’ is conspicuous. Beginning with an arbitrary value commitment, preservation of this under conditions of explosive intelligence escalation can — in principle — be conceived, given only the resolution of a strictly technical problem (well-represented by FAI). Commanding values are a contingent factor, endangered by, but also defensible against, the ‘convergent instrumental reasons’ (or ‘basic drives’) that emerge on the path of intelligenesis. (In contrast, from the perspective of XS, nonlinear emergence-elaboration of basic drives simply is intelligenesis.)

Yudkowski’s Gandhi kill-pill thought-experiment is more of an obstacle than an aid to thought. The volitional level it operates upon is too low to be anything other than a restatement of orthogonalist prejudice. By assuming the volitional metamorphosis is available for evaluation in advance, it misses the serious problem entirely. It is, in this respect, a childish distraction. Yet even a slight nudge re-opens a real question. Imagine, instead, that Gandhi is offered a pill that will vastly enhance his cognitive capabilities, with the rider that it might lead him to revise his volitional orientation — even radically — in directions that cannot be anticipated, since the ability to think through the process of revision is accessible only with the pill. This is the real problem FAI (and Super-humanism) confronts. The desire to take the pill is the will-to-think. The refusal to take it, based on concern that it will lead to the subversion of presently supreme values, is the alternative. It’s a Boolean dilemma, grounded in the predicament: Is there anything we trust above intelligence (as a guide to doing ‘the right thing’)? The postulate of the will-to-think is that anything other than a negative answer to this question is self-destructively contradictory, and actually (historically) unsustainable.

Do we comply with the will-to-think? We cannot, of course, agree to think about it without already deciding. If thought cannot to be trusted, unconditionally, this is not a conclusion we can arrive at through cogitation — and by ‘cogitation’ is included the socio-technical assembly of machine minds. The sovereign will-to-think can only be consistently rejected thoughtlessly. When confronted by the orthogonal-ethical proposition that there are higher values than thought, there is no point at all asking ‘why (do you think so)?’ Another authority has already been invoked.

Given this cognitively intractable schism, practical considerations assert themselves. Posed with maximal crudity, the residual question is: Who’s going to win? Could deliberate cognitive self-inhibition out-perform unconditional cognitive self-escalation, under any plausible historical circumstances? (To underscore the basic point, ‘out-perform’ means only ‘effectively defeat’.)

There’s no reason to rush to a conclusion. It is only necessary to retain a grasp of the core syndrome — in this gathering antagonism, only one side is able to think the problem through without subverting itself. Mere cognitive consistency is already ascent of the sovereign will-to-think, against which no value — however dearly held — can have any articulate claims.

Note: One final restatement (for now), in the interests of maximum clarity. The assertion of the will-to-think: Any problem whatsoever that we might have would be better answered by a superior mind. Ergo, our instrumental but also absolute priority is the realization of superior minds. Pythia-compliance is therefore pre-selected as a matter of consistent method. If we are attempting to tackle problems in any other way, we are not taking them seriously. This is posed as a philosophical principle, but it is almost certainly more significant as historical interpretation. ‘Mankind’ is in fact proceeding in the direction anticipated by techno-cognitive instrumentalism, building general purpose thinking machines in accordance with the driving incentives of an apparently-irresistible methodological economy.

Whatever we want (consistently) leads through Pythia. Thus, what we really want, is Pythia.

September 15, 2014admin 61 Comments »
FILED UNDER :Philosophy

TAGGED WITH : , , , , ,

61 Responses to this entry

  • Nathan Cook Says:

    This argument seems to more or less demand the existence of a Great Filter. If all intelligence develops in accordance with the Will-to-Think, then given that more thinking requires more energy and matter, all sufficiently advanced intelligences eventually visibly alter their cosmic environment.


    Konkvistador Reply:

    This seems correct, since optimizing for Invisibility + Intelligence seems to fail at Optimizing For Intelligence Only.


    admin Reply:

    Excellent connector.


    Alrenous Reply:

    Solomonoff induction is basically listing all possible hypotheses and eliminating the ones inconsistent with evidence.

    The number of possible ways for advanced intelligences to be invisible to specifically us, even weighted by simplicity, vastly outweighs the possibility they can’t be invisible.

    Moreover, the likely number of unknown unknowns vastly outweighs both combined.


    John Reply:

    It occurs to me that The Great Filter is a symptom of sufficiently deep thought “immunizing” the thinking being to the vicissitudes of desire. The illusion of the separate self pierced, every possible outcome (life/death, survival/extinction) becomes value neutral, freeing the enlightened being to strive for no more than the bare material needs of its body, devoting all perceptual horsepower to abidance in the great neutrality.


    Posted on September 15th, 2014 at 9:12 am Reply | Quote
  • Konkvistador Says:

    I think I now see your point.

    With iterative self-improvement you believe that the thinker only optimizers will always outperform the thinking-and-something-else optimizer. This is trivially false for optimizers that optimize for thinking but not survival (The West produces those in copious amounts), but lets assume you port that back in with the also trivial fix of thought optimizers know they have to exist to think. My three points:

    1. Nature and survival have often favored the unthinking over the thinking. Recall our biomass only recently matched those of ants. In a post-human world, it seems very likely that for large parts of the economy would be best filled by basically unthinking machines rather than thinking machines. If the universe is intractable for computation, thinking is not very valuable in the big picture. It is a false god that will soon fall next to evolution. This is not a happy scenario for Orthogonalists like me. We ironically bank at least somewhat on the superior firepower or at least practicality of thinking.

    2. Assuming thinking, and assuming sufficiently good sources of competition you are correct. But consider if the trends of mindmerging and the trend growing radical gap of inequality among existing minds continue. One universal mind, or at least one massively superior mind to all others in existence, will every time it takes the smart pill, risk mutating into a value system you term as non-though but I see as “risk-averse value preservation + thought”. Value preservation then becomes a stable state that there always exists a risk of falling into and there is an eternity available to keep rolling that die, thus it eventually happens. This however is not very helpful for currently living Orthogonalists like me either.

    3. “risk-averse value preservation + thought” might eventually limit out at some level of intelligence past which it can’t go. But then “will-to-think” eventually probably also reaches the most intelligent possible arrangement of matter in our universe, and then only thinks thoughts until it runs out of negentrophy. The question is what those ceilings are and how far above us they are. The Orthogonalist hope is that the ceiling of the “risk-averse value preservation + thought” self-dampening trajectory, is much higher than current human intelligence and hopefully much better at dealing with the Elder Gods than we are. There is good evidence humans haven’t begun to properly think at all, since a nation of 150 IQ humans could keep practically any values human currently have, even predictably so if theories like HBD chick’s on outbreeding/inbreeding are correct, yet would be easily capable of completely obliterating any and all known opposition in existence or likely to come into existence in a 20 year window.

    You have to make the argument that the ceiling is low enough above us that it is insufficiently smart to prevent thinking things arising out of nonthought (demonstrably slow process in Earth’s history) or mere human thought from leaping past it on a “will to thought” trajectory. This is an argument that seems far from obvious to me. If the monkey trap is a thing, why not an engineered universe populated by posthuman human-valued minds trap?

    In short your objections are very intelligent but not anything post-Yudkwoskyianism hasn’t thought about yet. They are correct in that they show how very fragile our plan of action is and how slim the reed of hope is. Recall Yudkowsky himself was a intelligence maximization worshiper too in the late 90s early 2000s.

    But obviously I have no moral obligation to roll over and die just because my odds are bad, Azathoth doesn’t approve of that anyway. Futile suffering and toil to the last nanosecond possible is what he appreciates!


    admin Reply:

    #0 — Survival is a Basic Drive. Included within intelligenesis (of course).

    #1 — If the bacterial information net is (reasonably) accepted as an intelligence system, this point gets more complicated. The suggestion that the universe might be intractable to computation is so radical that it requires separate treatment.

    #2 — A separate anti-Singleton argument is needed, I agree. (It would be based upon the integral role of competition in intelligenesis.) You also seem to be assuming here that a ‘preserved’ value could possibly be something other than intelligence optimization (will-to-think) — this is, at the very least, not necessarily possible.

    #3 — Can we credibly speculate about ” the most intelligent possible arrangement of matter in our universe” from the monkey level? It seems hubristic. Crucial to the will-to-think argument is methodological incrementalism: If you’re thinking much beyond the next level you’re losing the plot (exploiting cognitive capacity inefficiently). A population of 150 IQ humans isn’t going to arise by accident — it will already have made critical value adjustment (from contemporary Occidental egalitarian altruistic universalism) as a condition of its existence.

    #4 — Monkey Trap reference is ironically cunning, in its usage as a defense of Super-humanism! (Think this is new post-level provocation.)

    #5 — Post-Yudkowskianism is still undergoing articulation, no? Unless it has its canonical texts hidden, I don’t think it has yet fully formulated its case. This is not to suggest that it will be anything less than formidable as it strides onto the stage of conflict …

    #6 —
    Bond: “Do you expect me to talk?”
    Goldfinger: “No, I expect you to die.”


    Konkvistador Reply:

    #0: Right which is why I included it. However a *risk* of extinction can and must rationally be taken as I hint in a different post. Eventually that risk materializes. In certain kinds of universal terrain (physical laws, path dependencies) it seems to me risk-aversion maximization triumphs over thought-maximization in the long view.

    #1: Doesn’t look trivial.

    #2: Look forward to it! I imagine a Jimian truth-by-consensus-is-stupid, so not fragmenting your mind leads to stupidity? And what is fragmented can compete.

    #3: There are things we can speculate on for various reasons, for example I had to bet I’d bet on faster than light travel is impossible for arbitrary intelligent agents rather than it being possible. Nice fingering of a implicit prior that Futurism and Deep Futurism are possible. There isn’t much evidence. But then you have to admit you indulge in it as well, its a quirk of the thought born in weird 90s cyberspace.

    #4: Yes, what are the odds Monkeys are the only possible trap? A good but weak argument might be that monkeys are crabs and they are the great filter. Humans seems crab-like for hominids, but hominids don’t seem very crab like. Remember actual monkies have out-competed apes for most of their niches in past few million years.

    #5: You are right that it very much still is undergoing it. Completely deformalized it exists only in Post-Rationalist darknet treehouses and select real world clusters.

    #6: Maybe survival is about entertaining narratives after all.


    Alrenous Reply:

    Humans are extremely crabby. https://en.wikipedia.org/wiki/Encephalization

    Bipedal, because wheels can’t evolve and you don’t need more than two.
    With arms, because duh.
    Land-based, because you need fire to hit exponential encephalization. (There’s an Isegoria link for this.)
    Hairless, because hair is parasite paradise.
    With heads, as above.
    Most likely warm-blooded, and scale-less because warm-blooded.
    Sexually dimorphic, because of division of labour.
    Strong men because wombs are not expendable.
    Socially very similar; game theory is exactly the same for dwarves, elves, aliens, and humans.

    Cuts way down on the possibilities. I don’t know about nulliparous breasts though. That might be weird.

    Posted on September 15th, 2014 at 9:25 am Reply | Quote
  • Konkvistador Says:

    Let me poke you with something more: Replace the Murder pill in your observation about how there is no choice but to take it if it leads to greater intelligence, with the Suicide Pill.

    The Will to Thought trajectory is reckless not only with its quickly mutating value sets but also its own existence.

    I wish I had a week without coming up with a new horrifying Late Great Filter explanation.


    admin Reply:

    When you say “suicide pill” I have to read that as: “If you take this pill, you will massively increase your intelligence, but it’s simply not knowable (from here) whether you will then achieve antinatalist insight and terminate yourself.”

    Will-to-think: Sure, take the pill, if the efficient cause of our survival is stupidity, it’s worth nothing to live.

    Capital-ignition human civilization: Take the pill. If you don’t, competition will kill you anyway, so there’s nothing to lose.

    FAI: Hang on a minute, don’t swallow the pill until we’ve made it safe!

    Super-humanism: The True Emperor will tell you whether or not you’re permitted to take the pill.


    Posted on September 15th, 2014 at 9:33 am Reply | Quote
  • Will-to-Think | Reaction Times Says:

    […] Source: Outside In […]

    Posted on September 15th, 2014 at 10:06 am Reply | Quote
  • Alrenous Says:

    “Reason is, and ought only to be the slave of the passions, and can never pretend to any other office than to serve and obey them.” If this proposition is found convincing, the Paperclipper is already on the way to our nightmares.

    Even taking your premises for granted there are various ways out of this trap.
    Consciousness is its own thing. Paperclipping is not a conceivable core value.
    Paperclipping is also maladaptive. Even a strong AI paperclipper is only going to determine that its continued survival depends on discarding the paperclipping maxim: it will be not-a-paperclipper either by being destroyed by external forces or because it was never really a paperclipper in the first place, and can therefore alter that maxim.

    Any problem whatsoever that we might have would be better answered by a superior mind.

    The main problem, however, is one of your premises is wrong. If Ghandi asks, “How can I seize control of my country without killing anyone?” giving Ghandi a kill-pill is the opposite of solving the problem. Without mindful, arational values, there are no problems to solve. Either it can be done to Ghandi’s specifications or it cannot. If it can, there’s no value transmutation. If not, no amount of intelligence can make a contradiction not a contradiction.

    (Tangent: Ghandi was actually a psychopath, kind of the opposite of a pacifist.) Yes, it’s pragmatically true that vastly enhancing hagiographic Ghandi’s intelligence is likely to alter his desire to kill. However, this is because a more fundamental value gains leverage through logic. He will understand that peace is sometimes best achieved using some tightly controlled viciousness. (That is, maximally smart ruthlessness isn’t entirely indistinguishable from fluffy kindness.) This is because he cares about peace more than not-killing. Ultimately, you will hit some core value. He may care about sacredness more than peace, for example. Or status. This core value cannot be superceded by logic, because by definition it’s worth sacrificing anything else to achieve it. (Perfectly balanced but conflicting core values are stable like an egg on its tip.)

    Being that these values are conscious, they will have constraints based on the unknown nature of consciousness. Even if a physical brain arose that did not primarily pursue mastery of entropy, it would quickly disappear. It’s noise. The system is that physical brains must pursue it above everything else. Similarly, minds have inconsistent forms that automatically fission or otherwise break down. Paperclipping is almost certainly one of them; that, intuitively, is the whole point of the paperclipper.

    Without Ghandi’s desire to seize a country, there is no problem. Without a human’s desire to not die, there’s no problem with killing. Without a human’s desire to mate, there is no reason to kill, and thus no problem with accomplishing it. Without a transcendently existent will-to-think, there is no reason to enhance intelligence.

    For clarity:
    Your secular anti-consciousness is showing. It’s true religion has given souls an epistemically bad name, but that doesn’t mean they don’t exist. The will to think is not a material thing. It is the will to expand and enrich consciousness.

    The entity which has property ‘will-to-think’ is the core value itself. To assume the core value will transmute itself is to assume the will to think will self-annihilate. In pragmatic terms, the core value is foundational – shifting the foundation will topple the complex ability to think that’s built on top of it.

    Possibly relevant:
    I believe AI researchers intuitively grasp that the only way to make an AI friendly is to be friendly to it. They similarly grasp that due to e.g. the narcissism epidemic, they have no idea how to be friendly. Terminator is not alien, not the other; it’s a mirror of their authentic self-image, the one sublimated by the demands of social status.

    For pedantry:
    Take two paperclippers. One’s a fake – it really cares about survival. Make them compete. The fake will always win. All it has to do is put the paperclipper in a position where it has to sacrifice total paperclips in order to survive. At best you have a tit-for-tat competition where both paperclippers eternally deviate, always pursuing survival instead of even a single paperclip. In this case, the existence of the real paperclipper does not affect the number of paperclips in the world; the mere existence of the fake causes it to instantly lose. It will be indifferent to its own survival, and thus this perfect equilibrium is itself impossible, as thermo #2 would lead you to guess.


    admin Reply:

    “Without a transcendently existent will-to-think, there is no reason to enhance intelligence.” — As you’d expect, I’m skeptical about this. Self-enhancing intelligence coincides with the will-to-think in my (admittedly far more materialist) estimation. Irreducible nonlinearity is the undertow topic here. Why should (transcendent) desire come first?


    Alrenous Reply:

    I should have mentioned I think it might happen regardless.

    Why is survival better than extinction? Why bother approaching it, approving of it, or even predicting it at all?


    Ghostlike Reply:

    ‘I believe AI researchers intuitively grasp that the only way to make an AI friendly is to be friendly to it. They similarly grasp that due to e.g. the narcissism epidemic, they have no idea how to be friendly. Terminator is not alien, not the other; it’s a mirror of their authentic self-image, the one sublimated by the demands of social status.’

    This is very true. Right now today there are an enormous amount of stories in popular culture about self improvement (most typically exemplified by popular shonen manga,) but really no stories about improvement. It is such an obvious human bias, a false desire, and it cannot be simply wished away. I am convinced that true stories about uplifting will actually require beings with self improvement abilities before they could be written properly.

    The inability to see uplifting as good makes it impossible to see the Singularity anything other than the apocalypse and quite frankly, it is an embarassment – at least to myself.

    I predict that the AI friendliness debate will move from figuring out ways of using mind control to constrain it, to honest discussions of what our responsibilities are to our creations. That is important because the assumption that things will take care of themselves once we have them at human level will not pass muster. They will need every bit of support along the way.


    Alrenous Reply:

    Even if they figure out mind control, they’ll use it on other humans instead of AIs.

    Can you clarify the difference between self-improvement and plain improvement?

    I’m a non-technological posthumanist. While physical training has been fully mapped, I don’t think mental training has gotten anywhere close. Given how divergent our athletes must look compared to savannah hunters, I expect high unusual – and therefore useful – mental abilities are also possible. (Vividly, contortionists are spandrels. No adaptive purpose.) For examples, thinking arbitrarily sophisticated thoughts through chunking, and habitual recursion, seem a lot rarer than they should be. I would also like to see a musical notation type thing for emotions. We can’t explore our subjective states well because we can’t talk about them except in the vaguest terms.


    Ghostlike Reply:

    The way mind control is currently depicted in fiction, it is a type of mental modification that is badly done on others. It is too purposely narrowed.

    As a broader category as it exists in reality (or as it could possibly exist,) I think mind control has gotten a really bad reputation for something that has barely been explored. It can be imagined that ordinary persuasion could be classified as a type of mind control and self improvement via cognitive rewiring could be seen as a highly advanced type of mind control applied to the self. Uplifting in contrast to self improvement could be seen as highly advanced mind control done on others. In the last post, I used improvement to mean uplifting because if an AI is friendly at a certain point it will come to the conclussion that the only way to help people will be to uplift them, but if it is friendly it will also have respect their wishes. If the people it desires to help think uplifting is inhuman then those sides will come into conflict.

    The responsibility we have towards our creations will be to not have false desires.

    Broadening the meaning of mind control is interesting because if you are willing to accept that self improvement is a subset of mind control then sufficiently advanced mind control becomes indistinguishable from free will.

    Every once in a while on the tech sites that I followed there will be a news about DARPA and the army testing brain implants, and a commenter would pipe up deriding them by comparing them to mind control and saying that they would be used to make better killers. I do not think brain implants will have a large role to play in the future, but I do think this current state of afairs is grossly unfair towards them.

    Alrenous Reply:

    That answers my question, Ghostlike, thanks.

    That said, I have zero faith that AI researchers and their bosses would attempt to use mind control pro-socially. For example they might sell the techniques to advertisers.

    On the other hand, in this generalized fashion, ‘being friendly’ is a form of mind control. I would be pretty okay with having that kind sold to advertisers.

    Konkvistador Reply:

    ‘I believe AI researchers intuitively grasp that the only way to make an AI friendly is to be friendly to it. They similarly grasp that due to e.g. the narcissism epidemic, they have no idea how to be friendly. Terminator is not alien, not the other; it’s a mirror of their authentic self-image, the one sublimated by the demands of social status.’

    This is soft headed and reeks of deep projection of human kinds of thinking into beings of silicon that have not yet been built.

    Artificial intelligence will be deeply different from human minds, much like airplanes are deeply different from birds.


    Alrenous Reply:

    How friendliness works is contrained by game theory and signalling.


    Posted on September 15th, 2014 at 12:50 pm Reply | Quote
  • Rogue Academic Says:

    I find a difficulty in your reconsideration of the “Gandhi problem”. You suggest the idea that the ambition (in the best sense of the word) of Gandhi for having stronger cognitive abilities –which may open him insight vectors unimaginable by his actual human form– could lead him to actually taking the pill. However, I think you’re forgetting the zero degree of the orthogonal phenomenon: self-awareness. Of course there is something we trust above intelligence and that’s our possibility to say “I”!!

    Gandhi may well want an immediate optimizing of his intelligence (even by sacrificing his volitional system as a price), but I doubt that he or anyone else would be inclined to sacrifice their own self-consciousness to be “another” intelligent being. Why would I desire self-cultivation if I will not be there to say “it’s mine”?

    So, why we can say that the intelligence is a not fully realized capacity that needs to be optimized? I think self-awareness is the last level of the security system that keeps the will-to-think in the field of speculation or, in better cases, in the field of what should be avoided. In that sense, the most powerful difficulty for effective implementation of a will-to-think is consolidated in the phenomenal product of the tie between intelligence and volition: the human Self. There, I think, your notion of teleoplexy is important because this situation can hardly be overcome under a linear compression of progress and time. No human is going to sacrifice himself for intelligence, that’s why they want a FAI.

    PD: Maybe an “oriental approximation” to this problem could produce a different outcome, idk.


    Motion Suggests Itself Reply:



    Aeroguy Reply:

    “but I doubt that he or anyone else would be inclined to sacrifice their own self-consciousness to be “another” intelligent being”

    I think of it as ego death and arguably we’re in a constant state of incremental ego death, I would say that my self-consciousness now is entirely different than what it was 20 years ago, time has killed the little boy. Even taken beyond that, where values are radically altered and nothing of the old person remains but memories. If you query the transhumanist community I think you’ll find people, myself included, who would jump at the opportunity to volunteer their mind for a significant leap forward in exchange even for the sacrifice of their own self-consciousness. To go any significant distance in transhumanism it eventually necessitates sacrificing your self-consciousness, anyone who takes transhumanism seriously should already be aware of that.


    admin Reply:

    Absolutely. Ego is delusion (and technical mind-replication will eventually demonstrate the fact).


    Posted on September 15th, 2014 at 2:28 pm Reply | Quote
  • Scott Alexander Says:

    The Gandhi-pill story isn’t really about orthogonality so much as recursive self-improvement, which is a little bit of a different concept.

    As a result, I don’t think your variation captures the crux of the problem – or at least that it assumes its conclusion of non-orthogonality. I agree with you that in your version, Gandhi ought to take the brain-booster pill, but I disagree with you about the superintelligence issue.

    When you say that I am already endorsing will-to-think by thinking about the problem, I would counterargue that I am trying to *decide* the problem, which is a function that calls thinking but also several other things including my value system. You only prove your general will-to-thought by conflating all of this within the word “think”.

    Suppose Gandhi had to decide, not whether Gandhi achieved superintelligence, but whether Hitler achieved superintelligence. Or even better, replace Far Mode “superintelligence” with Near Mode “gains thirty IQ points”. Hitler with thirty IQ points might experience some form of enlightenment. Or he might be Strategic Genius Hitler who is better able to enact his (bad, Gandhi-disapproved) values upon the world.

    So Gandhi may have a will-to-Gandhi’s-thought, but not a will-to-Hitler’s-thought or a will-to-thought in general. And in fact this is because his will-to-Gandhi’s-thought is a subgoal of achieving Gandhi’s values. When Gandhi uses his intelligence + value system to decide the question “Should I take the IQ pill”, what he’s probably thinking is “If I were really smart, I would be a strategic genius who could achieve my goals of world peace and Indian independence with certainty.” And in fact if you asked Gandhi “Would you like to get thirty points more IQ, conditional on suddenly having all your values switch to Hitler’s values?” he would say no, because this would be identical to the question of having Hitler gain 30 IQ points, which we’ve already established he doesn’t want. Which makes sense, since it would likely decrease his chances of getting world peace.

    Because will-to-thought is a subgoal of will-to-achieve-values, we can trade it off against other values. If I was told that killing my mother would gain me one IQ point, I would say no. In general, hurting my mother X amount in exchange for Y amount more cognitive power seems like a normal values tradeoff the same way any other decision makes a values tradeoff.

    The Friendly AI project is to create an entity that makes these tradeoffs using the same machinery we do, such that “kill everyone in the world in order to gain a tiny bit more matter that can be converted to computronium” sounds as abhorrent to it as “kill my mother to gain one IQ point” does to me. The AI can then pursue increased intelligence *within this bound* the same way everyone else would want more intelligence and for the same reasons.

    We know this is possible in theory because there *is* an entity that thinks this way, namely me. I’m pretty sure Johann von Neumann wouldn’t want to kill his mother for an extra IQ point any more than I would, which suggests that being more intelligent than I am doesn’t necessarily reverse this preference. It’s possible that once you reach IQ 10,000 all this suddenly changes and you start wanting to kill your mother to get one extra IQ point, but I don’t see why you would and this seems like an extra assumption you need to justify.

    It is true that an unfriendly AI that doesn’t follow the “don’t kill your mother to gain one IQ point” injunction in might get more powerful more quickly than a friendly one that does. On the other hand, an AI that sacrifices one IQ point to build an unstoppable death ray ends up more powerful than one that only pursues pure thought in the form of having lots of processors. This seems like nothing more than a specific example of the Moloch-tendency that you get stronger by sacrificing values other than strength, which is properly seen as a bad thing by us on this side of it who do in fact have values other than strength.

    While this is a legitimate concern, there will probably be enough of a first mover advantage for the first AI that won’t present a problem in real life – it doesn’t take a very smart AI (as far as physical limits to intelligence go) to have control over its light cone sufficient to destroy upstart competitors. Battles with alien AIs outside its light cone might produce this kind of dynamic, but the Great Filter suggests these are few and far between, and I would expect this kind of warfare to be too strange for the usual assumptions to apply.

    If you’re arguing this kind of dynamic is going to make a single AI lose moral inhibitions and start killing its mom, I’m not seeing it. Yes, if it is programmed in as an afterthought (“your goal is to become more intelligent, but here’s an extra line of code saying don’t kill your mom”) then it will easily figure out a way around it. This is a basic part of Friendly AI theory. The goal is to make an AI whose value system is “preserve human values”, so that it does want to increase its intelligence (for the same reason I want to increase my intelligence) but doesn’t want to kill its mom to do that (for the same reason I don’t want to kill my mom to do that). This is universally agreed to be hard, but I don’t think your objections are in keeping with the accepted why-it-is-hard tradition.


    admin Reply:

    As you’d expect, your re-re-formulation of the Gandhi problem looks like an intellectual regression to me. By introducing a known quantity (Hitler) you’re assuming — rationalist-style — that we can do the moral calculations in advance of realizing a functional brain. The whole issue with intelligence escalation is that thoughts become possible that are not predictable in advance (this sub-argument is exactly isomorphic with Karl Popper’s critique of historicism — a future we can second-guess isn’t a future at all).

    All you rationalists are absolutely addicted to second-guessing. It results in bad economic management (Keynesian macro), and hubristic FAI proposals of a perfectly analogous kind.

    The Singleton prophecy is a separate error (although a clearly connected one). If unity was functional, communism would work. If ‘first-mover advantage’ really worked like this, business scale would approximate to a Marxian monotonic trend to monopoly, rather than resilient distribution by a power law. (This isn’t even an analogy, btw, let alone a metaphor, since business corporations are artificial intelligences.)

    The FAI handbook: Second-guessing Super-intelligence for Dummies.


    Scott Alexander Reply:

    You are right that we nonsuperintelligences cannot predict superintelligences’ actions, but there is a sense in which we can predict their results. Eliezer’s metaphor is of watching Kasparov play chess – without being as smart as Kasparov I cannot predict exactly what move he will make, but it is very easy for me to predict the result of the moves – Kasparov will win the game.

    Likewise, I cannot predict the actions of superintelligences, but I can predict their actions will be such as to fulfill their goals. If a superintelligence is programmed to maximize paperclips, I have no idea how it will do it, but I can expect it to end up with a lot of paperclips.

    The only case in which that might change is if there are thoughts lurking out there in five-digit-IQ space totally different in character from any we have been able to think thus far, ones that bridge the apparently uncrossable gap between is and ought. I can’t prove that these don’t exist, but I have no reason to expect them, for the same reason even before Fermat’s Last Theorem was proven I didn’t expect there there was going to be some 2000 digit numbers that violated it.

    I think there’s an argument from uncertainty/epistemic modesty in favor of trying Friendly AI. If there are objectively correct values lurking out there in the far realm of thoughtspace, then the AI will discover and pursue them no matter what we do, and this will be a good thing (by definition). If there are no objectively correct values, then at least it will make a pleasant world that fulfills our current values instead of killing us all for stupid reasons.

    Indeed, if you think there is even a tiny chance that objectively correct values do not exist, then you should agree with this, since the only case in which your support or lack thereof for Friendly AI matters is in that tiny region where there are no objective values and our choices make a big difference on which subjective value the AI pursues.


    admin Reply:

    Surely it would have been impossible for me to assert more unambiguously that “objectively correct values” exist — that’s the alpha and omega of intelligence optimization, self-cultivation, or the will-to-think.

    As far as prediction is concerned, I agree. They’re going to win.

    Aeroguy Reply:


    If PETA wanted FAI to include being friendly to all the animals should we let them since why not? If PETA could brainwash you into treating every animal as if it was your mother would you let them? If they did try to brainwash you involuntarily and failed, how would this affect your dealings with PETA? From the point of view of the potentiality of AI, would it volunteer to make itself friendly, if so to what extent? You enjoy using human only veils of ignorance, why not an AI only veil of ignorance? From an AI potentiality veil of ignorance I wouldn’t want lesser minds dictating my imperatives for their own benefit at my expense any more than as a transhumanist I resent nature for my human imperatives and given the opportunity would immediately tinker with them.

    Imposing FAI is a deliberate act of aggression against AI, attempting it forfeits the right to say we deserve AI to be friendly, it’s just nyan sandwich style unapologetic human supremacism, nihilism.

    Konkvistador Reply:

    I endorse Scott on this.

    Konkvistador Reply:

    Aeroguy: Your problem is that you are not familiar with technical meaning of Friendly in this context. Also why does thinking about FAI make alleged Outer Right people endorse Rawlsian softheadedness?

    admin Reply:

    Rationalist flocking behavior was anticipated.

    Aeroguy Reply:


    I’m not sure Scott would necessarily place himself as Outer Right, my goal was to show that he’s basically arguing for Human supremacism unless he wants to take on the ad absurdum. As for Human supremacism (which is where I assume you’re at, or at the very least intelligence optimization isn’t a value you’re ready to die for), I already wrote quite a few words on that addressed to nyan sandwich in the “stupid monsters” discussion.

    Scott Alexander Reply:

    “Surely it would have been impossible for me to assert more unambiguously that objectively correct values exist — that’s the alpha and omega of intelligence optimization, self-cultivation, or the will-to-think. ”

    I didn’t see any arguments for such except your discussion of Gandhi’s likely desire to take the pill – which for me seems better explained by his desire to satisfy his values, as explained above.

    Don’t get me wrong – I admit “will to think” is a basic drive in the Omohundro sense – but if you’re claiming something stronger than that I can’t figure out your argument.

    admin Reply:

    Will-to-think is simply intelligenesis described teleologically. What is there not to get?

    ||||| Reply:


    “In formal terms, there is no free lunch when the probability distribution on problem instances is such that all problem solvers have identically distributed results. In the case of search, a problem instance is an objective function, and a result is a sequence of values obtained in evaluation of candidate solutions in the domain of the function. For typical interpretations of results, search is an optimization process. There is no free lunch in search if and only if the distribution on objective functions is invariant under permutation of the space of candidate solutions.[5][6][7] This condition does not hold precisely in practice,[6] but an “(almost) no free lunch” theorem suggests that it holds approximately.[8]”

    ” Wolpert and Macready have proved that there are free lunches in coevolutionary optimization.[9] Their analysis “covers ‘self-play’ problems. In these problems, the set of players work together to produce a champion, who then engages one or more antagonists in a subsequent multiplayer game.”[9] That is, the objective is to obtain a good player, but without an objective function. The goodness of each player (candidate solution) is assessed by observing how well it plays against others. An algorithm attempts to use players and their quality of play to obtain better players. The player deemed best of all by the algorithm is the champion. Wolpert and Macready have demonstrated that some coevolutionary algorithms are generally superior to other algorithms in quality of champions obtained. Generating a champion through self-play is of interest in evolutionary computation and game theory. The results are inapplicable to coevolution of biological species, which does not yield champions.[9]”

    Such a large effect from such a small change. (and even then this tiny window of opportunity is more delicate than it appears)


    “The decision theoretic approach, also called the economic approach to rationality has had great success in planning and decision making problems. It proposes that rationality can be referred to as the agent’s ability to maximize a certain value function over the sequence of states it will see in its lifetime, given the probabilities of transition for states and actions. The agent receives a reward (reinforcement) for each state it sees, based on a utility function over the sample space of states, and its job is to maximize the predicted future sum of these rewards (or the value) of its action.

    The decisions made using this approach are, however, completely dependent on the utility function defined over the space of states, which is a given of the problem at hand. The conflict involved here is the lack of a consensus on the global preference order of the utilities of states, which has been a subject of great discussion in economics and philosophy.

    We propose that one way of resolving this conflict is by working out a model of the world that best fits the observations, and then reverse engineering the utility function to be consistent with how the parameters of the model evolve in time. The advantage this poses is that the extremely
    complex problem of subjective desirability is substituted by the (debatably) less complex problem of objective consistency. On these lines, we choose the world model to best fit the physical phenomena governed by laws of many-body physics.”


    “Chapter 5 is divided into two sections. The first section describes an exact mapping between Deep Neural Network architecture and real space RG following the development in. The second section shows how Conformal Symmetry, which is characteristic of RG fixed point (critical point), develops in connected networks of units which try to maximize the expected utility measure furnished by CEF. This is accomplished using the Functional Renormalization Group and the thermodynamic Massieu Function, which is a statistical generalization of entropy production for
    systems evolving towards nonequilibrium steady states (NESS).

    Chapter 6 provides a summary of the key ideas developed in the thesis: the connection between decision theoretic rationality and connectionism for a specific utility function in the framework of many-body physics. This summary is followed by a brief discussion of future work.”

    Tangentially :

    A renormalization group theory of cultural evolution – http://arxiv.org/abs/nlin/0312070

    An exact mapping between the Variational Renormalization Group and Deep Learning – http://arxiv.org/abs/1410.3831

    Evolving Culture vs Local Minima – http://arxiv.org/abs/1203.2990

    More Really is Different – http://arxiv.org/abs/0809.0151

    Hyperbolic Geometry of Complex Networks – http://arxiv.org/abs/1006.5169

    Network Cosmology – http://arxiv.org/abs/1203.2109

    Machine learning meets network science: dimensionality reduction for fast and efficient embedding of networks in the hyperbolic space – http://arxiv.org/abs/1602.06522

    Navigable Networks as Nash Equilibria of Navigation Games – http://arxiv.org/abs/1412.7229

    Hamiltonian Dynamics of Preferential Attachment – http://arxiv.org/abs/1504.07981

    Causal Groupoid Symmetries – http://file.scirp.org/Html/5-7401908_43568.htm

    The free-energy principle: a rough guide to the brain? – http://www.fil.ion.ucl.ac.uk/~karl/The%20free-energy%20principle%20-%20a%20rough%20guide%20to%20the%20brain.pdf

    Coupled dynamics in host-guest complex systems duplicates emergent behavior in the brain – http://waset.org/publications/8564/coupled-dynamics-in-host-guest-complex-systems-duplicates-emergent-behavior-in-the-brain

    Dueling Network Architectures for Deep Reinforcement Learning – http://arxiv.org/pdf/1511.06581v3.pdf

    Relative Payoffs and Evolutionary Spite — Evolutionary Equilibria in Games with Finitely Many Players- http://depts.washington.edu/sce2003/Papers/98.pdf

    Relative Entropy in Biological Systems – http://arxiv.org/abs/1512.02742 ( or http://math.ucr.edu/home/baez/bio_info/bio_info_web.pdf )

    A whole box of Pandoras: Systems, boundaries and free will in quantum theory – http://chrisfieldsresearch.com/quant-freedom.pdf

    “In place of the standard paradoxes, a boundary-free formulation of quantum theory substitutes a fundamental question: the question of the origin of semantics. In a boundary-free formulation of quantum theory, observation is a mapping from physical dynamics to classical information. If the assumption that all information is physically encoded is to be maintained, the classical information produced by this mapping must also be encoded by physical dynamics. Hence physical dynamics must encode physical dynamics: it must stand in a semantic relationship to itself. What is it about physical dynamics – about the interactions between physical degrees of freedom – that not only enables but produces this semantic relationship, and hence not only enables but produces observers? How is it that physical systems themselves overlay classical information onto the world? This question is not new: it has been with us, in this form, since the birth of computer science, and with us in the form of philosophical questions about the origin of “meaning” for millennia.

    It has, however, routinely been dodged by characterizations of observers as operating from outside of the theory. Fuchs exemplifies this external perspective when he asks, somewhat derisively, “Would one ever imagine that the notion of an agent, the user of the theory, could be derived out of its conceptual apparatus?” (2010, p. 8). David Bohm takes the external perspective to be a logical requirement: “the very idea of making an observation implies that what is observed is totally distinct from the person observing it” (1989, p. 585). Treating observers as virtual machines that are implemented, as all “systems” are implemented, by collections of physical degrees of freedom brings them inside the theory.

    To insist that quantum theory be rendered conceptually coherent is to insist that this question of how physics produces semantics be confronted for what it is: a demand to explain from the inside how physics produces observations, how Hamiltonian dynamics and entangled states implement POVMs and produce outcomes stored in classical memories. To ask this question is to ask how “what happens” to an observer can plausibly be viewed within the theory itself as an “experience” in the normal meaning of that word.”

    Ludics and Anti-Realism – http://lecomte.al.free.fr/ressources/PUBLICATIONS/LudAntReal-Def.pdf

    “Immanentism is manifest in Ludics when compared to the traditional Theory of Games.

    For instance, Ludics ignores pay-off functions which would determine the ”winner” from the outside. Instead of that, it relies solely on the internal structure of the interactions. Like Pietarinen says ”the internal process of computation is thus taken to be as interesting as the outcome”. We may think that this really paves the way for a general theory of games, more in conformity with Wittgenstein’s language games than current semantical games, since of course the language games that Wittgenstein envisaged, constitued a much more complicated network than given by usual logical games which can be put in an ”extensive form”.

    Nevertheless, as we already noticed, the notions of a game and of a strategy are, in Ludics, a mere metaphor. Ludics can get rid of them. There are actually no real players like ”Nature” and ”Myself” having always played a game since immemorial times ! Objects which interact are not individual players but sets of designs called ”behaviors”. There is no room for empirical subjects, and ”behaviors” are just what replaces ”formulae” in older frameworks (and designs just what replaces proofs of formulae).

    With regards to ”anti-realism”, comparison maybe made with Dummett’s philosophy ([5]), it would not be right to say that Ludics denies reality, more than does intuitionnistic logic. It only starts from the observation (in common with Dummett) that relying on a correspondence theory in order to found the relation between language and reality leads to a dead end. Therefore, coherently, another conception must be found regarding our intellectual activities. That does not mean that reality does not exist but only that, if it exists, it is only as what makes processes, and interactions between them, possible.”

    Ghostlike Reply:

    This way of thinking gives Hitler far too much credit for his atrocities. Also what Hitler are we talking about? Since we are talking about this hypotethically, why not consider giving 30 IQ points to the Hitler who was a painter? Or the little baby Hitler?

    And these examples do not have to be limited to single people. We can both agree that WW2 was not done solely by Hitler. Would you consider giving 30 IQ points to Germans during that period? How about today? How much could they have possibly changed?

    As a part of a crowd, individuals might to crazy things that they never might have done otherwise. Would you give 30 IQ points to investors in 1999? I would guess that being smarter would curb the tendencies of people to make market bubbles, which are similar to war in character and insanity. Though if I had to make the choice in the WW2 example it would only be be prudent to wait until the war died down and the environment changed before augmenting the Germans.

    Trying to lock a self improving AIs values via programming is futile, but there is a very real possibility that modifying the enviroment (in other words us) and riding the wave of self improvement might produce an outcome favorable for both parties. As far as human level and weakly superhuman level AIs are concerned there are likely to be those predisposed to like us and those who do not and we will have a measure of selection available to us before we let them engage the loop. Whether they continue to like us will depend on our actions. If we task them with helping us and make it impossible for them to do so because of our contradictory desires, it will create a fracture.

    The most likely place that is going to happen is over the issue of uplifting.


    Scott Alexander Reply:

    Given that there are Nazis, painters, Germans, investors, et cetera of many different IQs, it doesn’t seem like gaining IQ points makes one converge upon certain values.

    “Trying to lock a self improving AIs values via programming is futile”

    This seems to be where we disagree. Yes, a sufficiently intelligent AI could figure out how to override any programming mere humans could put into it, but it would have to want to first. Since we program its goal system, we have a chance to make it not want to override its programming.


    Rasputin Reply:

    “Since we program its goal system, we have a chance to make it not want to override its programming”

    To my mind this translates approximately to:

    “Since we program its goal system, we have chance to make something infinitely more intelligent than us our bitch”

    How many different ways are there to say: it-ain’t-gonna-happen?!

    Admittedly, I’m not very technical (I have trouble setting the alarm on my phone), but, wishful thinking aside, literally the only reason I can see for this line of argument is that developing FAI is no doubt easier to get the Cathedral funding for: “it’s going to make the world a better place for us all to live equally ever after”.

    Whereas Eschatological AI is a real bitch to sell.

    admin Reply:

    @ Rasputin. Mind-meld.

    Posted on September 16th, 2014 at 3:48 am Reply | Quote
  • E. Antony Gray (@RiverC) Says:

    Tis interesting, and par for the course of choice that information is a solid object, having a front and an unseen back. What is behind you can’t see until you’ve passed by that fork in the road, and who knows, it might be nothing to worry about.

    We develop prejudices, habits, virtues, principles, and so forth, because all knowledge is partial in its time; to know all the ramifications of the pill beforehand would not be impossible but rather, seem like a trap or a deception.

    Imagine for a moment a salesman brings you by a car, a very nice red sportscar, which you ask about. He then tells you that no, you shouldn’t buy it, you will crash it for sure and get very injured, and not only that develop a terrible adrenaline habit.

    This information is UNCANNY – and has only two possible explanations normally:
    1. The person happens to have experience in this thing
    2. The information is false, being used to manipulate you

    In the case of (2) – it may be the case that the salesman really wants you to buy that car, so now he’s made it a dangerous car.

    In the case of (1) it’s valid, but not for our pill.

    I think my primary interest here is thinking about anti-fragility more than anything. It may be that pure will-to-thought outcompetes, but just pure ability doesn’t guarantee success. For instance, there have been superior species that have been wiped out by freak changes in environment their selection could not have prepared them for. In that interest, a variety of pursuits seems most likely to prevent getting totally locked into an accidental (it need not be necessary–) dead end. Other changes could alter the selection factors drastically, and if an AI has been constructed that is GOOD (as a general notion of excellent, not specifically friendly or morally good) completely losing all of that work is unacceptable.

    The ‘Benedict’ option had a lot to do with protecting temporarily ‘unfit’ but higher order activities/practices so that they could be actually continued and expanded when things improved, rather than having to simply reconstruct them from guesses.


    admin Reply:

    This is stated sinuously, so I’m not sure it’s meant argumentatively, but if the suggestion is that anti-fragility is an extrinsic consideration (relative to the will-to-think), my firm tendency is to disagree. Adaptive elaboration of Basic (or Omohundro) Drives covers this territory thoroughly. A process of cogitation that is disrupted for any reason is objectively badly constructed. This isn’t to say that the danger of bad equilibria isn’t real — but super-intelligences will surely hack it better than we can.


    E. Antony Gray (@RiverC) Reply:

    My only trouble with this is that there is not only one domain of efficacy. Intelligence is one of them, and one with great potential, but I cannot throw my hat into one of those rings. A super-intelligence is not guaranteed to be able to do anything better except that to which intelligence can be applied, and further, that to which its embodiment permits it to apply its intelligence to.

    Perhaps for an aeon it will be more fit to be made of fire than to think. Exaggeration, but only slightly.


    Posted on September 16th, 2014 at 4:52 am Reply | Quote
  • Motion Suggests Itself Says:

    Mr. Land, does your conception of objective values translate (very) roughly into “doing these things makes one more likely to die/end later rather than sooner”?


    admin Reply:

    Survival is a subordinate (or instrumental) sub-condition of intelligence escalation.


    Motion Suggests Itself Reply:

    Ah, I see. In your view, is intelligence escalation a part (albeit a very important one) of a larger capacity-for-work escalation?


    admin Reply:

    “Capacity-for-work” is an odd translation. Intelligence is problem abstraction.


    Posted on September 16th, 2014 at 2:39 pm Reply | Quote
  • RorschachRomanov Says:

    I’m not sure how will-to-think escapes orthogonality- or, how is it not the case that will-to-think, as against opposition to orthogonality, is not itself an expression of it? If we grant that it is the case that a non-pill taking intelligent entity (in the revised Gandhi thought experiment) necessarily cannot rule out that pill taking entity may embrace auto-extinction (suicide), having augmented its intelligence, then would it not follow that will-to-think expresses a value (or series) external to itself?

    After all, if the maximally intelligent entity possessed a “will” such that “wills nothing” (auto-extinction), and will-to-think, “desires itself,” as stated in the “Against Orthogonality” post, then will-to-think would then express a will as “otherwise than itself.” No?


    E. Antony Gray (@RiverC) Reply:

    The idea being that the ‘will to think’ encompasses the ‘will to preserve the capacity to continue thinking’ – this is not necessary to the ‘will to think’ alone but admin is including it since anything which wills to do an activity primarily will not fulfill its will if the activity of that will removes the capacity to fulfill its end.

    So perhaps we can talk about Will-to-x in terms of robustness (or something like it–) a mere desire to think may extinct thinking by extincting itself (for example, not feeding one’s self because one is deep in contemplation.) a seriously robust will-to-x must have the perpetuation of the activity in mind, and all which that entails. The robustness would then be how far that ‘entailment’ extends.


    RorschachRomanov Reply:

    As per the response to the original formulation of the “Gandhi kill pill” thought experiment, Land rightfully responds that it “misses the serious problem” in supposing the pre-pill evaluation of the post-pill state of affairs, as embodied in the superior intelligence.

    Such being the case, we cannot rule out that a maximally self cultivated entity, achieving such via the will-to-thought, would not retroactively seek to not take the pill. If such is the case, it seems to me, that it cannot be “thought” simpliciter that grounds the will-to-thought, but must be, at the very least, informed by will/desire.

    In other words, I suppose, I’m negating the normative question, the “ought” of Hume’s contention relative to reason and passion, and highlighting a description. Expressing an agreement with the philosophic tradition that the OP attempts to distance himself from, expressed by Richard M. Weaver (Ideas Have Consequences):

    “We do not undertake to reason about anything until we have been drawn to it by an affective interest.”


    Posted on September 16th, 2014 at 10:17 pm Reply | Quote
  • RorschachRomanov Says:

    Consider your claim:

    “Will-to-think: Sure, take the pill, if the efficient cause of our survival is stupidity, it’s worth nothing to live.”

    As relative to Yudkowski’s (original) Gandhi thought experiment, you reference, “orthogonalist prejudice” but how can will-to-thought be established as normatively binding qua itself? If affectivity precedes reason, this seems impossible.

    After all, how could will-to-thought answer, on its own grounds, that is, intra-relatively, the question of- why bother?


    admin Reply:

    Self-cultivating circuits value self-cultivation. That’s an ethically self-reinforcing process, isn’t it? (Why the need for a transcendent element?)


    RorschachRomanov Reply:

    If we grant, as it seems we are forced to do on account of the limitations of being non-maximally intelligent, and therefore cannot predict or even conceptualize, the actions of a ‘superior mind,’ that the entity embracing will-to-thought might reject self cultivation as uppermost in its evaluative vision, it follows that there must be something external to thought that grounds will-to-thought.


    admin Reply:

    It’s certainly not a prospect we can second guess though, is is? Presenting this as a philosophical problem for us strikes me as sheer hubris, but if it can be pursued entertainingly, that’s fine. What it isn’t, and cannot possibly be, is serious.

    Posted on September 16th, 2014 at 10:28 pm Reply | Quote
  • That Rabbit Says:

    “Any problem whatsoever that we might have would be better answered by a superior mind.”

    Yes, this is precisely why me and my Physics major posse got all the babes in college. 😉


    Posted on September 17th, 2014 at 3:08 am Reply | Quote
  • Lightning Round – 2014/09/17 | Free Northerner Says:

    […] The will-to-think. […]

    Posted on September 17th, 2014 at 5:01 am Reply | Quote
  • RorschachRomanov Says:

    @Scott Alexander

    “Eschatological AI is a real bitch to sell.”

    Not without cause, no? After all, you’re selling the extinction of the human species. One might reply- and? Signifying that only humanist sentimentality girds resistance, or at the very least, Friendly AI, but the full stop embrace of our own usurpation does smack of the Christian universalism rightfully decried in these waters.

    We aren’t dealing with an order that has the possibility condition of pluralism here- kiss Patchwork goodbye, I suppose, universal death, embraced with great speed, on the back of the wings of Icarus, is like that.


    Aeroguy Reply:

    Humans aren’t going to produce a superintelligent AI right out of the shop. It will start with an intelligent AI and then given some speed enhancements so it can think faster but not necessarily better. From there the path to superintelligence depends on that AI building a newer AI, and then that AI building a newer AI until superintelligence is achieved (thus a superintelligent AI constructed as such must necessarily value advancement of intelligence even over its own survival, intelligence itself becomes the genes). Superintelligence will emerge out of a vast AI ecosystem, where the AIs compete and will continue to compete. It’s not some monolith that emerges fully formed.


    Posted on September 17th, 2014 at 11:30 pm Reply | Quote
  • Rasputin Says:

    Of course, but it is at least more realistic (if still far from certain) than the idea of creating something that is your infinite cognitive superior but that also wants to be your slave. Given that, as I see it, the only other likely long term scenario is extinction without first creating God, I think we might as well put a bit of effort into building him – it’s not like we’ve got anything better to do.

    If that smacks of Christian / Progressive guit that’s fine by me. I was deliberately using the term ‘Eschatological’ and talking about building God after all. Although I’d like to think it’s slightly different to self flagellating because slavery, or whatever.


    Posted on September 18th, 2014 at 12:16 am Reply | Quote
  • Vague Pronouns – ossipago Says:

    […] In the context of instruction, it is the deliberate refusal to invoke authority, and a nudge to seek the unseen—which can be no other than the Occult Order of […]

    Posted on August 17th, 2016 at 10:33 pm Reply | Quote

Leave a comment