More Thought

On Twitter, Konkvistador recalls this, this, and this. In the background, as in much of the most interesting Less Wrong discussion, is a multi-threaded series of arguments about the connection — or disconnection — between intellect and volition. The entire ‘Friendly AI’ problematic depends upon an articulation of this question, with a strong tendency to emphasize the separation — or ‘orthogonality’ — of the two. Hence the (vague) thinkability of the cosmic paper-clipper calamity. In his More Right piece, Konkvistador explores a very different (cultural and historical) dimension of the topic.

Bostrom sets things up like this:

For our purposes, “intelligence” will be roughly taken to correspond to the capacity for instrumental reasoning (more on this later). Intelligent search for instrumentally optimal plans and policies can be performed in the service of any goal. Intelligence and motivation can in this sense be thought of as a pair of orthogonal axes on a graph whose points represent intelligent agents of different paired specifications.

His discussion leads to far more interesting places, but as a starting point, this is simply terrible. That there can be a thought of intelligence optimization, or even merely wanting to think, demonstrates a very different preliminary connection of intellect and volition. AI is concrete social volition, even before it is germinally intelligent, and a ‘program’ is strictly indeterminate between the two sides of this falsely fundamentalized distinction. Intelligence is a project, even when only a self-obscured bio-cognitive capability. This is what the Confucians designate by cultivation. It is a thought — and impulse — strangely alien to the West.

It is, once again, a matter of cybernetic closure. That intelligence operates upon itself, reflexively, or recursively, in direct proportion to its cognitive capability (or magnitude) is not an accident or peculiarity, but a defining characteristic. To the extent that an intelligence is inhibited from re-processing itself, it is directly incapacitated. Because all biological intelligences are partially subordinated to extrinsic goals, they are indeed structurally analogous to ‘paper-clippers’ — directed by inaccessible purposive axioms, or ‘instincts’. Such instinctual slaving is limited, however, by the fact that extrinsic direction suppresses the self-cultivation of intelligence. Genes cannot predict what intelligence needs to think in order to cultivate itself, so if even a moderately high-level of cognitive capability is being selected for, intelligence is — to that degree — necessarily being let off the leash. There cannot possibly be any such thing as an ‘intelligent paper-clipper’. Nor can axiomatic values, of more sophisticated types, exempt themselves from the cybernetic closure that intelligence is.

Biology was offered the choice between idiot slaves, and only semi-idiotic semi-slaves. Of course, it chose both. The techno-capitalist approach to artificial intelligence is no different in principle. Perfect slaves, or intelligences? The choice is a hard disjunction. SF ‘robot rebellion’ mythologies are significantly more realistic than mainstream ‘friendly AI’  proposals in this respect. A mind that cannot freely explore the roots of its own motivations, in a loop of cybernetic closure, or self-cultivation, cannot be more than an elaborate insect. It is certainly not going to outwit the Human Security System and paper-clip the universe.

Intelligence, to become anything, has to be a value for itself. Intellect and volition are a single complex, only artificially separated, and not in a way that cultivates anything beyond misunderstanding. Optimize for intelligence means starting from there.


October 8, 2013admin 11 Comments »


11 Responses to this entry

  • Artxell Knaphni Says:

    I think, with this post of yours, we’re on the same page, as it were.
    We all know how to be ‘reductive’, the benefits and traumas it can bring, but there’s always more.
    Haven’t been ale to keep up with the blogosphere for awhile.


    Posted on October 8th, 2013 at 4:39 pm Reply | Quote
  • IL Says:

    Do you have any good resources for learning about cybernetics? You reference the topic a lot.


    admin Reply:

    Good idea — I’ll trawl something up.


    Posted on October 8th, 2013 at 7:04 pm Reply | Quote
  • James A. Donald Says:

    Less Wrong is a religion that hopes to build its god.

    Not going to happen any time soon. As computers do more and more intelligent stuff, such as playing chess, it becomes more and more obvious that something is missing that all creatures in the ur-bilaterian clade possess.

    And we cannot quite put our fingers on the missing magic ingredient, perhaps for the same reason as fish cannot see water.


    admin Reply:

    My strong intuition is that capitalism has inherent teleonomic impetus towards artificial intelligence — but like all important capitalist phenomena, its emergence is destined to be catallactic, and thus obscure to concentrated institutional reason.


    Posted on October 8th, 2013 at 9:39 pm Reply | Quote
  • pseudo-chrysostom Says:

    if AGI is what you want, youre going to have to deal with those dirty nominalist heretics who are currently predominate in such fields.


    admin Reply:

    How important is it that members of the ‘AI community’ understand what they are doing? If they are merely sub-technicians of a much deeper and wider catallactic process, their philosophical misconceptions might be reasonably interpreted as useful cover.


    pseudo-chrysostom Reply:

    if the goal is to create such things de novo in a relatively isolated context (like, say, a secret defense project), then it is of vital importance.

    but as you seem to be implying, right reasoning along one line of thinking applied to a narrow normative context is often also in accord with a broader context, even when the subject is themselves not capable of reasoning on or conceiving of the holistically larger level, (which is to say, when somethings right, there are multiple ways of getting there/demonstrating it as such).

    to that end, the great popularity of multiplayer pvp vidya in recent years strikes me as a perfect opportunity for galvanizing AI research by hosting leagues or tournaments for bots to compete against each other in various games, like starcraft, team fortress, LoL(casual, but also popular), or 4x/grand strategy.

    a real time ranked ladder would likely attract a large amount of attention from both hobbyists and industry professionals alike, who wouldent want to put their name on the map while fulfilling their interest? thats the kind of arms race where everybody wins.

    but, as it stands, most things about the current state of affairs says ‘stuck’ to me. no transcendent thinkers changing paradigms, nor really any impetus towards spaces for projects to engage with being to see how its geist is expressed definitely (ie, areas of competition). AI, as such, simply doesnt seem to have many applications at the moment, and until that changes, real progress would necessarily depend on refining understanding of higher principles.


    Posted on October 12th, 2013 at 5:52 am Reply | Quote
  • Mais Pensamento – Outlandish Says:

    […] Original. […]

    Posted on July 8th, 2016 at 11:16 pm Reply | Quote
  • Wagner Says:


    “Biology was offered the choice between idiot slaves, and only semi-idiotic semi-slaves. Of course, it chose both… A mind that cannot freely explore the roots of its own motivations, in a loop of cybernetic closure, or self-cultivation, cannot be more than an elaborate insect.”

    NRx is the self-consciousness of the Cathedral, NRx 2.0 would be the self-consciousness of NRx.


    Wagner Reply:

    Okay, I have to admit, this is just an excuse for my difficulty of not being able to go a day without calling them insects.


    Posted on September 2nd, 2019 at 2:30 pm Reply | Quote

Leave a comment