OpenAI’s offices were sent thousands of paper clips in an elaborate prank to warn about an AI apocalypse::The prank was a reference to the “paper clip maximizer” scenario – the idea that AI could destroy humanity if it were told to build as many paper clips as possible.
I think part of the wisdom in the warning is that any kind of “intelligence” (read: NOT specifically artificial general intelligence) is capable of running away with unforseen scenarios.
Hell, even normal ol’ algorithms can have some pretty nasty edge cases that noone spots until it’s running in production… Sure it’s uncommon, but it’s not exactly rare. (just look up the list of zero-day exploits over the years)