If that smacks of Christian / Progressive guit that’s fine by me. I was deliberately using the term ‘Eschatological’ and talking about building God after all. Although I’d like to think it’s slightly different to self flagellating because slavery, or whatever.

Such being the case, we cannot rule out that a maximally self cultivated entity, achieving such via the will-to-thought, would not retroactively seek to not take the pill. If such is the case, it seems to me, that it cannot be “thought” simpliciter that grounds the will-to-thought, but must be, at the very least, informed by will/desire.

In other words, I suppose, I’m negating the normative question, the “ought” of Hume’s contention relative to reason and passion, and highlighting a description. Expressing an agreement with the philosophic tradition that the OP attempts to distance himself from, expressed by Richard M. Weaver (Ideas Have Consequences):

“We do not undertake to reason about anything until we have been drawn to it by an affective interest.”

“Eschatological AI is a real bitch to sell.”

Not without cause, no? After all, you’re selling the extinction of the human species. One might reply- and? Signifying that only humanist sentimentality girds resistance, or at the very least, Friendly AI, but the full stop embrace of our own usurpation does smack of the Christian universalism rightfully decried in these waters.

We aren’t dealing with an order that has the possibility condition of pluralism here- kiss Patchwork goodbye, I suppose, universal death, embraced with great speed, on the back of the wings of Icarus, is like that.

To my mind this translates approximately to:

“Since we program its goal system, we have chance to make something infinitely more intelligent than us our bitch”

How many different ways are there to say: it-ain’t-gonna-happen?!

Admittedly, I’m not very technical (I have trouble setting the alarm on my phone), but, wishful thinking aside, literally the only reason I can see for this line of argument is that developing FAI is no doubt easier to get the Cathedral funding for: “it’s going to make the world a better place for us all to live equally ever after”.

Whereas Eschatological AI is a real bitch to sell.

“Trying to lock a self improving AIs values via programming is futile”

This seems to be where we disagree. Yes, a sufficiently intelligent AI could figure out how to override any programming mere humans could put into it, but it would have to want to first. Since we program its goal system, we have a chance to make it not want to override its programming.