by David Samuels
If she fell below the right number of steps, it would lower her health and fitness rating, which is part of her social rating, which is monitored by the government.
A low social rating could
prevent her from working or traveling abroad. China's social rating
system, which was announced by the ruling Communist Party in 2014,
will soon be a fact of life for many more Chinese.
...by the state will
affect one's social rating.
will find themselves in a "green channel," where they can more
easily access social opportunities, while those who take actions
that are disapproved of by the state will be "unable to move a
Yet in the West, at least, the threat of government surveillance systems being integrated with the existing corporate surveillance capacities of big-data companies like,
...into one gigantic
all-seeing eye appears to trouble very few people - even as
countries like Venezuela have been quick to copy the Chinese model.
We are iPhone owners and
Amazon Prime members, not vassals of a one-party state. We are canny
consumers who know that Facebook is tracking our interactions and
Google is selling us stuff.
Among other things, it showed how the company engineered sneaky ways to obtain continually updated SMS and call data from Android phones.
In response, Facebook
claimed that users must "opt-in" for the company to gain access to
their texts and calls.
The best guides we have
to this emerging reality may be failed 20th-century totalitarian
experiments and science fiction. More on that a little later.
While US surveillance agencies do not have regular real-time access to the gigantic amounts of data collected by the likes of Google, Facebook, and Amazon - as far as we know, anyway - there is both anecdotal and hard evidence to suggest that the once-distant planets of consumer Big Tech and American surveillance agencies are fast merging into a single corporate-bureaucratic life-world, whose potential for,
...citizens may result in
a softer version of China's Big Brother.
One of the leading corporate spenders on lobbying services in Washington, DC, in 2017 was Google's parent company, Alphabet, which, according to the Center for Responsive Politics, spent more than $18 million.
Lobbying Congress and
government helps tech companies like Google win large government
contracts. Perhaps more importantly, it serves as a shield against
attempts to regulate their wildly lucrative businesses.
In locating one of
Amazon's two new headquarters in nearby Northern Virginia, Bezos
made the company a major employer in the area - with 25,000 jobs to
Last year, Amazon Web Services announced the opening of the new AWS Secret Region, the result of a 10-year, $600 million contract the company won from the CIA in 2014. This made Amazon the sole provider of cloud services across,
Once the CIA's Amazon-administered self-contained servers were up and running, the NSA was quick to follow suit, announcing its own integrated big-data project.
Last year the agency
moved most of its data into a new classified computing environment
known as the Intelligence Community GovCloud, an integrated
"big data fusion environment," as the news site NextGov described
it, that allows government analysts to "connect the dots" across all
available data sources, whether classified or not.
This is the threat that
is now emerging in plain sight. It is something we should reckon
with now, before it's too late.
...the company said in a
statement, which notably did not deny the activity, while perhaps
implying that turning over user data to government spy agencies is
At the same time as Yahoo was feeding user data to the NSA, Google was developing a search engine called Dragonfly in collaboration with the Communist Party of China.
In a letter obtained by
The Intercept, Google CEO Sundar Pichai told a group
of six US senators that Dragonfly could have "broad benefits inside
and outside of China" but refused to release other details of the
program, which the company's search engine chief, Ben Gomes,
informed Google staff would be released in early 2019.
...while linking online searches to a user's phone number and tracking their physical location and movements, all of which will presumably impact social ratings or worse - much worse, if you happen to be a Uighur or a member of another Muslim minority group inside China, more than 1 million of whom are now confined in re-education camps.
surveillance net is a key tool by which Chinese authorities identify
and track Muslims and others in need of re-education.
Under pressure from its
employees, Google said in June that it would not seek to renew its
Project Maven contract when it expires in 2019.
Cook didn't hesitate to name the process he was describing.
While Apple makes a point
of not unlocking its iPhones and SmartWatches even under pressure
from law enforcement and surveillance agencies, companies like
Google and Facebook that earn huge profits from analyzing and
packaging user data face a very different set of incentives.
According to a search warrant issued by a judge trying a double-murder case in New Hampshire, and obtained by TechCrunch, the court had,
Amazon told the Associated Press that it would not release such recordings
...a response that would
appear to suggest that the recordings in question exist.
Yet as Washington keeps
buying expensive tools and systems from companies like Google and
Amazon, it is hard to imagine that technologists on both ends of
these relationship aren't already seeking ways to further integrate
their tools, systems, and data.
Such fond hopes have a long history.
The all-seeing Amazon, Google, and Facebook
have every incentive to help the national security state
undermine privacy, free speech, and democracy.
We’ve read this book before.
animation by Casey Chin
Progressive techno-optimism goes back to the origins of the computer itself, in the correspondence between Charles Babbage, the 19th-century English inventor who imagined the "difference engine" - the first theoretical model for modern computers - and Ada Lovelace, the brilliant futurist and daughter of the English Romantic poet Lord Byron.
This is a pretty good
description of the principles of digitizing sound; it also eerily
prefigures and predicts the extent to which so much of our personal
information, even stuff we perceive of as having distinct natural
properties, could be converted to zeros and ones.
and early 20th-century European socialists, who
championed the organic strength of local communities, early 20th-century
American progressives like Herbert Croly and John Dewey
put their faith in the rise of a new class of educated
scientist-priests who would re-engineer society from the top down
according to a strict utilitarian calculus.
The 2008 election of Barack Obama, a well-credentialed technocrat who identified very strongly with the character of Spock from Star Trek, gave the old-time scientistic-progressive religion new currency on the left and ushered in a cozy relationship between the Democratic Party and billionaire techno-monopolists who had formerly fashioned themselves as government-skeptical libertarians.
Big Tech companies and
executives are happy to return the favor by donating to their
progressive friends, including Schumer.
Blaming the election
result on Russian bots or secret deals with Putin
betrayed a shock that what the left had regarded as their cultural
property had been turned against them by a right-wing populist whose
authoritarian leanings inspired fear and loathing among both the
technocratic elite and the Democratic party base.
Thinking machines can be taught to filter out bad information and socially negative thoughts. Good algorithms, as opposed to whatever Google and Facebook are currently using, could censor,
voters from electing more Trumps.
The result might be a
better social order, or as data scientist Emily Gorcenski put
Whether machines can
"think," or - to put it another way, whether people think like
machines - is a question that has been hotly debated for the past
five centuries. Those debates gave birth to modern liberal
societies, whose foundational assumptions and guarantees are now
being challenged by the rise of digital culture.
If you believe, like Searle and Leibnitz, that the answer is no, you understand thinking as a subjective experience, a biological process performed by human brains, which are located in human bodies.
By definition, then, the
human brain is not a machine, and machines can't think, even if they
can perform computational feats like multiplying large numbers at
Since you can build
machines that fix their own problems - debug themselves - these
machines are innately self-aware, and therefore there's nothing
stopping them from evolving until they reach
For Thomas Hobbes, who inspired the social-contract theorist John Locke, thinking was,
David Hume, who
extended Hobbes' ideas in his own theory of reason, believed that
all of our observations and perceptions were nothing more than
atomic-level "impressions" that we couldn't possibly make sense of
unless we interpreted them based on a utilitarian understanding of
our needs, meaning the attempt to derive the greatest benefit from a
A social order monitored and regulated by machines that have been programmed to be free of human prejudice while optimizing a utilitarian calculus is therefore a plausible-enough way to imagine a good society.
would be the better angels of our nature, helping to bend the arc of
history toward results that all human beings, in their purest, most
rational state, would, or should, desire.
Not coincidentally, these accounts evolved during the last time western societies were massively impacted by a revolution in communications technology, namely the introduction of the printing press, which brought both the text of the Bible and the writings of small circles of Italian and German humanists to all of Europe.
The spread of printing
technologies was accompanied by the proliferation of the simple hand
mirror, which allowed even ordinary individuals to gaze at a "true
reflection" of their own faces, in much the same way that we use
iPhones to take selfies.
What was once the
province of the few became available to the many, and the old social
order that had governed the lives of Europe for the better part of a
millennium was largely demolished.
Thirty Years' War, fought between
Catholic and Protestant believers and hired armies in Central and
Eastern Europe, remains the single most destructive conflict, on a
per capita basis, in European history, including the First and
Second World Wars.
Our inability to wrap our minds around a sweeping revolution in the way that information is gathered, analyzed, used, and controlled should scare us.
It is in this context
that both right- and left-leaning factions of the American elite
appear to accept the merger of the US military and intelligence
complex with Big Tech as a good thing, even as centralized control
over information creates new vulnerabilities for rivals to exploit.
Only two decades ago, the social and political power of the institutional press was still so great that it was often called "the Fourth Estate" - a meaningful check on the power of government.
The term is rarely used
anymore, because the monopoly over the printed and spoken word that
gave the press its power is now gone.
As a result, the value of "legacy" print brands has plummeted. Where the printed word was once a rare commodity, relative to the sum total of all the words that were written in manuscript form by someone, today nearly all the words that are being written anywhere are available somewhere online.
What's rare, and
therefore worth money, are not printed words but fractions
of our attention...
Facebook, probably the world's
premier publisher of fake news, was recently worth
$426 billion, and Newsweek changed hands in 2010 for $1, and why
many once-familiar magazine titles no longer exist in print at all.
It is the difference between all of those media people, old and new, and programmers and executives at companies like Google and Facebook.
A set of key social
functions - communicating ideas and information - has been
transferred from one set of companies, operating under one set of
laws and values, to another, much more powerful set of companies,
which operate under different laws and understand themselves in a
That has now been
replaced by the creation and amplification of extremes. The
overwhelming ugliness of our public discourse is not accidental; it
is a feature of the game, which is structured and run for the profit
of billionaire monopolists, and which encourages addictive use.
Ten years from now, thanks to AI, those networks, and the entities that control them - government agencies, private corporations, or a union of both - may take on a life of their own.
Perhaps the best way to
foresee how this future may play out is to look back at how some of
our most far-sighted science fiction writers have wrestled with the
future that is now in front of us.
Riffing on Darwin, Butler
proposed that if the species can evolve to the detriment of the
weak, so could machines, until they would eventually become
self-sufficient. Since then, science fiction has provided us with
our best guides to what human societies mediated or run by
intelligent machines might look like.
Interestingly, Capek's automatons aren't machines:
In the play, the humans are degenerates who stop procreating and succumb to their most selfish and strange whims - while the robots remain unerring in their calculations and indefatigable in their commitment to work.
The machines soon take
over, killing all humans except for a single engineer who happens to
work and think like a robot.
They each beg for the other's life, leading the engineer to understand that they have become human; he spares them, declaring them the new Adam and Eve.
This soulful theme of
self-awareness being the true measure of humanity was taken up by
dozens of later science fiction authors, most notably Philip K. Dick
in Do Androids Dream of Electric Sheep?, which became the film Blade
Perhaps we are reading the wrong books...
Instead of going back to Orwell for a sense of what a coming dystopia might look like, we might be better off reading We, which was written nearly a century ago by the Russian novelist Yevgeny Zamyatin.
We is the diary of state mathematician D-503, whose experience of the highly disruptive emotion of love for I-330, a woman whose combination of black eyes, white skin, and black hair strike him as beautiful.
This perception, which is
also a feeling, draws him into a conspiracy against the centralized
According to the Lex Sexualis, the government sex code,
Citizens, or numbers, are issued ration books of pink sex tickets.
Once both numbers sign
the ticket, they are permitted to spend a "sex hour" together and
lower the shades in their glass apartments.
The subjective human
perception of beauty, Zamyatin argued, along lines that Liebniz and
Searle might approve of, is innately human, and therefore not
ultimately reconcilable with the logic of machines or with any
utilitarian calculus of justice.
By eliminating freedom
and all causes of inequality and envy, the Only State claims to
guarantee infinite happiness to humankind - through a perfect
calculus that the Integral will spread throughout the solar system.
But the real threat to the ideal of happiness incarnated in the Integral is not inequality or envy or hidden power. It is beauty, which isn't rational or equal, and at the same time doesn't exclude anyone or restrict anyone else's pleasure, and therefore frustrates and undermines any utilitarian calculus.
For D-503, dance is beautiful, mathematics is beautiful, the contrast between I-330's black eyes and black hair and white skin is also beautiful.
Beauty is the answer to D-503's urgent question,
Beauty is the ultimate example of human un-freedom and un-reason, being a subjectivity that is rooted in our biology, yet at the same time rooted in external absolutes like mathematical ratios and the movement of time.
As the critic Giovanni Basile writes in an extraordinarily perceptive critical essay, "The Algebra of Happiness," the utopia implied by Zamyatin's dystopia is,
Against a centralized surveillance state that imposes a motionless and false order and an illusory happiness in the name of a utilitarian calculus of "justice," Basile concludes, Zamyatin envisages a different utopia:
Human beings will never stop seeking beauty, Zamyatin insists, because they are human.
They will reject and
destroy any attempt to reorder their desires according to the logic
The question now - as in previous such moments - is how long it will take before we admit that the riddle of human existence is not the answer to an equation.
It is something that we
must each make for ourselves, continually, out of our own materials,
in moments whose permanence is only a dream.