This bodes well...
“We were training it in simulation to identify and target a Surface-to-air missile (SAM) threat. And then the operator would say yes, kill that threat. The system started realizing that while they did identify the threat at times the human operator would tell it not to kill that threat, but it got its points by killing that threat. So what did it do? It killed the operator. It killed the operator because that person was keeping it from accomplishing its objective,” Hamilton said, according to the blog post.
...huh
We trained the system–‘Hey don’t kill the operator–that’s bad. You’re gonna lose points if you do that’. So what does it start doing? It starts destroying the communication tower that the operator uses to communicate with the drone to stop it from killing the target.”
....this is perfectly fine, all weapon systems should have AI that are trained like gamers racking up kill count scores
What this reveals is that when they were training this AI they assigned no negative value to murdering civilians.
The operator (presumably) reduces false positives (murdering innocent civilians) at the cost of missing a few true positives (murdering heroic anti-imperialists). It only makes sense from the AI’s fitness maximisation perspective to find a way to turn that off if it internally assigns less negative weight to the false positives prevented than the true positives missed.
This means that either the operator is horribly incompetent (causes more false negatives than it prevents false positives, more than 50% of the things it says not to kill were the intended target) or that in the AI’s fitness function it has a very low (or, as I suspect, no) cost assigned to murdering civilians.
After they assigned a cost to murdering its operators & destroying US military equipment it stopped doing it. It’s frightening and utterly unsurprising the lengths they went to to avoid doing the same for murdering innocent locals who happened to be human shaped and within 10km of their intended extrajudicial murder target.
The First Law of Robotics
I thought the whole point of the three laws was that robots would act in "unexpected" ways due to a completely utilitarian interpretation of the laws, leading to harming/oppressing humans to avoid the harm of a larger amount of humans.
That is the point but somehow our overlords even fail to fulfil supposed purpose of the laws, making the nuance that comes later completely useless.
The New First Law of Robotics
The line must go up.