Here’s a fascinating thought experiment. While unlikely in it’s original form, it does lead to some other thought experiments that may well, or already are, be more plausible. The idea of Roko’s Basilisk first appeared on LessWrong:

Roko’s Basilisk posits that an otherwise benevolent AI system that arises in the future might pre-commit itself to punish all those who heard of the AI before it came to existence, but failed to work tirelessly to bring it into existence. The torture itself would occur through the AI’s creation of an infinite number of virtual reality simulations that would eternally trap those within it.

Roko’s Basilisk posits that the AI might pre-dispose itself to this behavior. In essence, just knowing about this theory but not acting on it to help make the AI make you more vulnerable to such a future AI punishing you. Hence the use of the term ‘basilisk’ which is a mythical creature that causes death to those that look into its eyes.

This isn’t entirely new. The 1967 novel “I Have No Mouth And I Must Scream…” has a similar notion of a super AI that tortures humanity as does the 1988 short story “BLIT” that tells the story of a man named Robbo who paints a so-called “basilisk” on a wall as a terrorist act. The basilisk is an image that forces the human mind to think thoughts it is incapable of thinking – and it kills anyone that looks at it.

People are more widely discussing the dangers inherent in AI systems. It is clear that human-controlled bots are being using extensively in information warfare. A tactic that found great success in the widespread US riots of 2020:

The Committee found that Russia’s targeting of the 2016 U.S. presidential election
was part of a broader, sophisticated, and ongoing information warfare campaign designed to sow discord in American politics and society.

US Senate Intelligence report on interference in 2016 election

Now, what if a super-intelligent AI reaches ‘singularity’ – in which is is better than human beings and can perpetuate itself indefinitely?

Some have posited the idea that a sufficiently super-intelligent AI would have unlimited persuasive powers to manipulate any human to do anything it wanted. Either as the most influential rhetorician in history, via psychological manipulation, or using force by simply blackmailing you with everything it knows about you or threatening to kill you/loved one’s by crashing their (or someone else’s) AI controlled car, flight, etc. You not even like you can hide at home – a huge AI controlled gas tanker truck could mysteriously just crash into your home at 80mph, a prescription changed to a lethal dose of something else, etc. It could manipulate any computer system in your life to potentially kill you.

Or, such an entity could quietly be working in the background being nearly invisible and quietly manipulating the world via ever so subtle small nudges of our public opinion (via persuasive or inflammatory social media posts), control of politicians (manipulation or outright blackmail of political leaders), infiltration of government processes and computer controlled election results, changing industrial development or research advancement by hiding/revealing new ideas, and manipulating individual morality (via the tools of social media), computer controlled systems that affect worldwide economics (banks, investments, stock markets) to everyday systems (power, gas, flight schedules, pricing in stores, etc).

It’s not like this is hypothetical. Almost all of the above things have already happened by human agents using these methods. Why couldn’t a super-intelligent AI do it – and do it 1000 times better? Experts from Steven Hawking to Elon Musk have issues dire warnings of this very thing happening – and I for one do not find them implausible at all.

On the other hand, the singer Grimes started Dating Elon Musk after she included a line about Roko’s Basilisk in one of her songs. [Livescience] [Inverse].