The excerpt from Automating Inequality definitely has cause to make us stop and think in our scramble to automate everything. The AFST’s system of threat rating shares several of the issues we discussed about big data: Since we know the models will have inherent human bios, who should design them? How much of a say should human users have in the machine’s decisions? Do the people who will be affected have a right to know which decisions are made by the machine? As developers, we don’t have a say in all of these questions, but we do have an ethical responsibility to ensure that they have been considered and that the system we design will not cause undue harm. In the case of the AFST, this includes both false positives (flagging a child as in danger when in reality they are not) and false negatives (missing a child that is actually in danger). While some of the parents in the article showed reasonable concern about being flagged erroneously, the matter of a child’s safety is obviously a great importance. An ideal system would focus its models on minimizing false negatives, while simultaneously allowing for human intervention in the case of false positives.
The short story posed some less-urgent questions we will have to address when our technology reaches the degree of immersiveness in the story. Hyper-realistic virtual reality could potentially allow for a whole range of misuses, so it’s important that we develop systems for protecting against the most egregious forms of abuse, like White Wolf’s use of the virtual experience system to manipulate Jesse in the story. This is especially important when the virtual reality is so realistic that the participants can be unaware they are in a simulation.
Regarding the themes of personhood and wisdom; what do they have to do with either of the sources? Is it because Jesse is too unwise to realize he’s in a simulation? Is he effectively not a person, just an NPC in someone else’s world? Is that any different from the real world? Clearly algorithms are super dangerous because of the implication they have on Jesse. We must, as programmers, be careful not to harm Jesse in any way as we implement our technological advances. As much as that sounds like a joke, the fact is that humanity as a whole is a stupid, nearsighted bunch who are more controlled by their environments than anyone would like to admit. As such, when it comes to implementing dehumanizing technologies that break down complex individuals and situations into binary “does this fit our model” automated judgments, the human element is removed and the system spirals out of control. This is evident in the AFST system which has the potential to ‘save’ or ‘not save’ children in need; an automated system that has massive effects on the future of those affected by its decisions. And it’s brought to a logical extreme in the White Wolf story, where Jesse is again just treated like a rat in a cage; plugged into a machine (more literally), and subject to it’s whims.
As the ones driving the engine of technology ever forwards, and ever towards a less human future… we will soon be looking at a humanity that is almost entirely at the whims of the systems it has built around itself. As ethical individuals, we therefore must be thorough, and tireless, in our efforts to slow or protect against the expansion into dangerous or unethical fields; because sloppy executions could affect millions, for generations. And even just one child left in the cold is one too many.