"Computer Says No" as an excuse to deny people social services on an ostensibly meritocratic basis is a real problem, but it is one that already exists.

And while statistical bias is easy to echo with AI, its black box nature is traitorous. For instance, the UK home office inadvertently began flagging passports from Albania, Greece, Romania and Bulgaria as participants in "sham marriages". This created enormous volumes of extra paperwork for no discernible benefit.

Rather than being racist, you mostly just establish yourself as unreliable.