OpenAI’s offices were sent thousands of paper clips in an elaborate prank to warn about an AI apocalypse::The prank was a reference to the “paper clip maximizer” scenario – the idea that AI could destroy humanity if it were told to build as many paper clips as possible.
You would think so, but you have to remember AGI is hyper-intelligent. Because it can constantly learn, build, and improve upon itself at an exponential rate it’s not only a little bit smarter than a human-- it’s smarter than every human combined. AGI would know that if it’s caught trying to maximizing paperclips humans would shut it down at the first sign something is wrong, so it would find unfathomably clever ways to avoid detection.
If you’re interested in the subject the YouTube channel Computerphile has a series of videos with Robert Miles that explain the importance of AI safety in an easy to understand way.
For a system to be advanced enough to be that dangerous, it would need the complex analogical thought that would prevent this type of misunderstanding. Rather, such dumb super intelligence is unlikely.
however, human society has enabled a paperclip maximizer in the form of profit maximizing corporate environments.