Eliezer Yudkowsky FB blogging open problems in Friendly AI in realtime…

https://m.facebook.com/groups/233397376818827?view=permalink&id=233401646818400&__user=577077603

Where did you find it?

Via Ed Feser’s blog.

Where did you find it?

Many areas of recent advance in neuroscience are converging on the conclusion that neural circuitry does not record, store or transmit information in forms that could express propositions

Indeed. And yet, meaning exists. I know, because I can observe myself to have some. Ergo, physics cannot be the whole story.

The brain does not think. The mind thinks. The brain merely computes.

50 years of neuroscience have given us ample reason not to trust consciousness or introspection

Shockingly, it is hard to find objective evidence of subjective entities.

Truly, this is a masterpiece, and I must bow in respect to the craftsmanship.
It is a piece of denialism, but even still.

Because it ontologically commits you to dualism. If you have to believe that souls are as real as rocks, or find an alternative, which do you choose?

Well, if the alternative is barking mad …

Correct, and a very good way to put it. That said, from *inside* a system optimizing for some values (that were originally hypertrophied), it is simply a bad idea to allow more values to ascend. So the orthogonalist thesis is this:

1. Because subgoal stomp hurts the current value set in the long run, any sufficiently self-aware optimizing system will want to prevent it.

2. It is possible an optimizing system to resist further subgoal stomp, if it wanted to and knew how.

3. There are multiple possible sets of initial goals, whether created by subgoal hypertrophy (human values), accident (evolution), or deliberately (friendly AI).

4. It is therefore possible to have multiple possible long-term stable goal sets in arbitrarily powerful optimization systems.

This isn’t a philosophical question, it’s an engineering/empirical question. Can a system be constructed that reliably optimizes for X? The orthogonalists say “yes”, the non-orthogonalists say “no, all sufficiently powerful systems will end up optimizing for Y”. There is the separate question of how hard or likely an X maximizer is compared to a Y maximizer, but the orthogonality thesis is about possibility.

>Any intelligence using itself to improve itself will out-compete one that directs itself towards any other goals whatsoever.

This is also true and understood by orthogonalists. See Robin Hanson’s “hardscrabble frontier” stuff, and “burning the cosmic commons”. As mentioned elsewhere in this thread, it is possible that first mover advantage of the first superintelligence will nullify this concern, and it will only have to sacrifice a small amount of its resources to prevent competition.