The intermediate stages might be kind of rough, though. First, our wealth distribution system is not well-generalized enough to deal with post scarcity, and so as we approach it, our distribution system will become more and more deranged.

Similarly, if it is possible to make low-skill workers obsolete, while I think all workers will rapidly become obsolete at that point, there will be a 20-40 year gap where some are and some aren’t, and that will produce incredible wealth gradients.

Without such a community, elite cohesion and status competition would probably break down disastrously. Huxley thought a society of alphas wouldn’t work because no alpha would be willing to do beta and delta and gamma work. But the real reason is that an alpha by definition needs to feel himself on top of a heap. So there has to be a heap.

Him:

… Singularity University … spent a lot of time discussing two key topics:

1. The rise of artificial intelligence and the potential for a super AI to come to dominate society.

2. The impact of artificial intelligence and automation on jobs. Highly skilled workers continue to grab a greater share of the economic pie, while in many cases the value of unskilled labour is so low as to make them not employable at a living wage.

Expanding on that, in the past, as productivity gains were made, there was always some requirement to have unskilled labour, and as a result unskilled labour was able to capture a portion of productivity gains – the pool of unskilled labour was not unlimited, and therefore they had some bargaining power. If the minimum price you can pay for local unskilled labour is too high (minimum wages are too high), you will automate instead. There may be no value that those labourers can provide at a price that the market is willing to pay that is acceptable from a regulatory standpoint (if the minimum wage is higher than the clearing price for unskilled labour). In this case, you have to decide if, as a society, you simply pay off unskilled labour (give everyone welfare) or are indifferent to their plight. The option of indifference can take two forms: abandonment (see Camden from 2011 to early 2013) or extreme policing (see Camden in its current form).

At some point, however, and this was the point that was being made at [SU], artificial intelligence starts to eliminate middle income and high skilled jobs as well. How does society maintain its stability in the face of huge swathes of the population becoming redundant?

My point to you is the following: while you observe that most of the population is ignoring the concepts and issues that you think are most important, the wealthiest, smartest, and most powerful people in the world are not.

Me:

The question of what to do as everyone gradually (or rapidly) becomes economically obsolete is interesting. I think that at the point where everyone is obsolete, we’d want the machines to figure out what was best for human flourishing, because they’d be better at that sort of thing as well, conditional on being designed to actually care and do it right. There are a great many other things they could care about, though, and given the nature of moral philosophy, it would be a seriously difficult project to construct them to care about the right things, involving solving many major philosophy problems and reducing them to engineering. This is a hurdle that construction of killer indifferent AI would not have, so that part needs to get started early.

It is encouraging to know that elites are interested in the problem. I don’t know if the *Friendliness* aspect is widely understood among such people, though. Everything I see indicates that yes, people are interested in the problem, but don’t understand the difficulty and importance of getting friendliness right. In that sense, it still seems worth making noise about that aspect of the problem. I could be wrong about this, though; I’m going on public information, and haven’t made a serious research effort.

Him:

If the elites design the machine (and this is highly likely), then you are far more likely to end up with a machine that is built for the benefit of the elite than for the benefit of everyone. Don’t bet on the wealthiest people in society being altruistic. Our current system is a bargain between the rich and the poor where the rich agree to give up a considerable amount if income in return for not getting shot in the street or knifed in their sleep. They are purchasing security by keeping the poor happy. That has not always been the preferred method of purchasing security, and it may not be the preferred method in the future if the weaponry value of expensive technology increases versus the weaponry value of cheap technology. If the risk adjusted cost of subjugation declines below the cost of placation, do not bet on the rich choosing placation over subjugation.

Me:

I suspect that *if* they get it right by their own standards (indirect normativity, friendliness, turn over details to AI singleton), on reflection the sentimental value of saving and uplifting everyone else to some level would outweigh the cost. We’re talking about a fraction between 1/2 and 10^-9 of the total resources available (the entire universe) in return for *saving and uplifting everyone*, depending how extensively you want to fund them. I expect there would be some point in that range where the utility cost would no longer outweigh the altruistic opportunity. Still, point taken that the indifference fraction could be much lower than anyone would admit now when the balance of power is different.

But I have the advantage of having read him for years, and he’s been saying this for years.

That said, I don’t think you need real AI to automate a number of intellectual tasks that humans now do directly. I don’t see any particular reason the efforts of lots of smart people can’t be systematized to take over a number of tasks (lots of legal work is pretty rote, for example) with maybe ongoing human input here and there.

http://www.imdb.com/title/tt2209764/

Regarding your ADDED link, that is the real end game impact of Bitcoin. The currency is only round one. Ultimately everything is going in the Blockchain. Google got the ball rolling but Bitcoin is where it gets legs. The Mind of God bootstrapping itself before our eyes.

I think Hanson is evasive on purpose. That is a very dangerous subject and it is doubtful that a satisfactory answer can be given. Probably haven’t thought about it in-depth enough, but it certainly doesn’t look pretty. I am looking forward to your treatment of it.