The Chinese Room argument convinced me that John Searle is likely a non-intelligent imitation of a philosophy professor.

One good framework to think about the problem of "Consciousness" in Humans vs. AI/AGI is the Samkhya School of Hindu Philosophy - https://en.wikipedia.org/wiki/Samkhya

See my past comments here for some background - https://news.ycombinator.com/item?id=40479388

This is how i work it out;

1) "Consciousness" defined as attribute-less pure awareness/witness and immutable "Self" (aka Purusha in Samkhya) will never be possible in AI/AGI.

2) All other human mental states which are mutable and considered as evolutes of Prakriti in Samkhya are possible in AI/AGI. A simple mapping would be;

2.1) Buddhi in Samkhya ~ Metacognition - https://en.wikipedia.org/wiki/Metacognition

2.2) Manas in Samkhya ~ Phenomenology - https://en.wikipedia.org/wiki/Phenomenology_(psychology)

2.3) Ahamkara in Samkhya ~ Emergent property from above due to "I Think therefore I am".

Now realize that (1) is purely experiential/internal and only (2) is observable from an external pov and you get the idea why AI/AGI may seem "Conscious" (in the common usage of the word) to external observers.

>...worried that an insufficiently cultivated mind ... would be perfectly imitable and thus replaceable by computers and other machines

I'm not sure cultivating our minds will buy much time there. But they will never be an exact replacement in the same way humans are not an exact replacement for cats and dogs in spite of being smarter.