Big Data: How It Threatens Democracy

There are numerous organizations trying to learn as much about us as possible.  Everywhere we go, every website we visit, every book we read, every comment we make on social media, every email we write, who we communicate with by phone is being examined by people who wish to make a profit from information about us.  Everything known about you is fed into analysis routines that try to characterize you and predict how you are likely to behave as a consumer, as a voter, as an employee, as a member of an insurance plan, even as a lover.  Much of this activity is relatively harmless, such as helping vendors market their products to individuals.  In other cases the stakes are much higher: this data is being used in ways we have no control over and drawing conclusions about us that may be quite in error yet still determining whether or not we are worthy of a job, a loan, or even a prison sentence.

Cathy O’Neil addresses this “big data” environment and its uses in her book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.  We have discussed previously how big data is used to increase inequality.  Here the focus will be on how it is used to threaten democracy.

Big data has often been used to target people who have financial problems.  If a person is desperate for a solution to a problem, that is the best time to sell them some shady financial scheme that is likely to do them more harm than good.

“We are ranked, categorized, and scored in hundreds of models, on the basis of our revealed preferences and patterns.  This establishes a powerful basis for legitimate ad campaigns, but it also fuels their predatory cousins: ads that pinpoint people in great need and sell them false or overpriced promises.  They find inequality and feast on it.  The result is that they perpetuate our existing social stratification, with all of its injustices.”

The same techniques that identify people who might be receptive to the offer of an ultrahigh interest rate loan or an inducement to sign up at a for-profit college can be used to identify people who might be receptive to false or exaggerated political claims.  O’Neil provides this example.

“In 2015, the Center for Medical progress, an antiabortion group, posted videos featuring what they claimed was an aborted fetus at a Planned Parenthood clinic.  The videos asserted that Planned Parenthood doctors were selling baby parts for research, and they spurred a wave of protest, and a Republican push to eliminate the organization’s funding.”

“Research later showed that the video had been doctored: the so-called fetus was actually a photo of a stillborn baby born to a woman in rural Pennsylvania.  And Planned Parenthood does not sell fetal tissue.  The Center for Medical Progress admitted that the video contained misinformation.  That weakened its appeal for a mass market.  But with microtargeting, antiabortion activists could continue to build an audience for the video, despite the flawed premise, and use it to raise funds to fight Planned Parenthood.”

There are a large number of campaigns that are not widely seen because the general public would find them ridiculous or repulsive.  However, microtargeting can feed false or misleading information to vulnerable people who might be receptive, and thus propagate that information.

“According to Zeynep Tufekci, a techno-sociologist and professor at the University of North Carolina, these groups pinpoint vulnerable voters and then target them with fear-mongering campaigns, scaring them about their children’s safety or the rise of illegal immigration.  At the same time, they can keep these ads from the eyes of voters likely to be turned off (or even disgusted) by such messaging.”

“As this happens, it will become harder to access the political messages our neighbors are seeing—and as a result to understand why they believe what they do, often passionately.  Even a nosey journalist will struggle to track down the messaging.  It is not enough simply to visit the candidate’s web page, because they, too, automatically profile and target each visitor, weighing everything from their zip codes to the links they click on the page, even the photos they appear to look at.”

“Successful microtargeting, in part, explains why in 2015 more than 43 percent of Republicans, according to a survey, still believed the lie that President Obama is a Muslim.  And 20 percent of Americans believed he was born outside the United States and, consequently, an illegitimate president.”

As a result, different people will approach a political issue armed with different sets of supposed facts.  For a democracy to function properly, it must have people who are dealing with the same set of facts so that they can come to a compromise over how to address the consequences of that set of facts.  Having contending factions that possess different views of reality is a recipe for disaster.

“The result of these subterranean campaigns is a dangerous imbalance.  The political marketers maintain deep dossiers on us, feed us a trickle of information, and measure how we respond to it.  But we’re kept in the dark about what our neighbors are being fed.  This resembles a common tactic used by business negotiators.  They deal with different parties separately so that none of them knows what the other is hearing.  This asymmetry of information prevents the various parties from joining forces—which is precisely the point of a democratic government.”

O’Neil is also concerned because there are other big data companies that are intimately acquainted with us and our interests and preferences and could be responsible for biasing our views—perhaps without even intending to.  Consider the case of search engines such as Google.

“Two researchers Ronald Epstein and Ronald E. Robertson, recently asked undecided voters in both the United states and India to use a search engine to learn about upcoming elections.  The engines they used were programmed to skew the search results, favoring one party over another.  These results, they said, shifted voting preferences by 20 percent.”

“The effect was powerful, in part, because people widely trust search engines.  Some 73 percent of Americans, according to a Pew Research report, believe that search results are both accurate and impartial.”

O’Neil recognizes that it would be dangerous for a Google to purposely bias search results in order to promote some agenda, and there is no evidence that they do this.

“Then again, how would anyone know?  What we learn about these Internet giants comes mostly from the tiny proportion of their research that they share.  Their algorithms represent vital trade secrets.  They carry out their business in the dark.”

If a person with known liberal tendencies who has demonstrated a preference for reading liberal views searches on a topic with political relevance, does the Google algorithm provide a balanced mix of liberal and conservative views on the topic, or does it preferentially provide views that are more likely to be those desired by the searcher?  If it is intended to be the former, then it is difficult to see how that can actually be accomplished.  If the latter is intended, then the risk of positive feedback reinforcing established views becomes a problem.  As O’Neil points out “How would anyone know?”

O’Neil also fears the potential power of Facebook to bias opinions.  Facebook was accused by some of being a vehicle for the propagation of “fake news” in the 2016 presidential election.  Facebook, with its news feed, funnels information to its users in ways that are potentially biased just as search engines do.

“When we visit the site, we scroll through updates from our friends.  The machine appears to be only a neutral go-between.  Many people still believe it is.  In 2013, when a University of Illinois researcher named Karrie Karahalios carried out a survey on Facebook’s algorithm, she found that 62 percent of the people were unaware that the company tinkered with the news feed.  They believed that the system instantly shared everything they posted with all of their friends”

“While Facebook may feel like a modern town square, the company determines, according to its own interests, what we see and learn on its social network.  As I write this, about two-thirds of American adults have a profile on Facebook.  They spend thirty-nine minutes a day on the site, only four minutes less than they dedicate to face-to-face socializing.  Nearly half of them, according to a Pew Research Center report, count on Facebook to deliver at least some of their news, which leads to the question: By tweaking its algorithm and molding the news we see, can Facebook game the political system?”

Facebook itself seems quite interested in the answer to that question.  It has been running experiments on its users to find out how much influence it can wield.  With many millions of users available, it can run these experiments quickly and accurately.  O’Neil tells us that they admit to having been successful in increasing voting rates and in altering emotional states by manipulating the algorithm used in determining the news feed to a subset of users.

Once again, there is no evidence that Facebook has any intention other than to make a profit, but once again “How would anyone know?”  Even with the best of intentions, by operating as a news source, the company risks injecting bias into the views of its users.  If the news feed is intended to provide items that the particular user would be most likely to read, there is a real danger of reinforcing existing political views.  If it selects the most popular items to feed to its users, the risk is that the news will be dominated by those who mount the loudest and most outrageous campaigns.  If it tries to provide an unbiased set of news items, how can it actually accomplish that?

It is clear that voters will generally seek information that supports their preconceptions.  Voters who vote Republican prefer to get their news from Fox News.  That is a conscious choice they can make.  The information they are fed by Google and Facebook is determined by someone else.  Could the algorithms used by these companies be contributing to the extreme political polarization that has developed?

How would anyone know?

The interested reader might find these articles informative:

Big Data: Its Role in Increasing Inequality

Weapons of Math Destruction: The Higher Education Arms Race

Trump and Populism: Be Very Afraid

During the 2016 presidential election campaign, Bernie Sanders and Donald Trump were both described as populist candidates in the media.  It is difficult to imagine two more different individuals with two more different outlooks.  If they are both populists, then what does populism mean?  Understanding populism—and populists—has become more critical because Trump is now the president of the United States and it is necessary that we understand what populists do when they come to power.

Jan-Werner Müller is a German academic who is currently Professor of Politics at Princeton University.  His background provides him with intimate knowledge of the relevant developments in both Europe and the United States.  He provides us with insight into populism and its practitioners in his book What is Populism?  He bases his conclusions partly on academic studies and partly on the observation of recent populists who have strived for or gained power.  The book came out in August, 2016; consequently, Müller is well aware of the campaigns of the candidates for the presidency.

Müller recognizes that populism can from the left as well as from the right, but decides that Bernie Sanders does not qualify as a populist.  The hallmark of populist candidates is that they are against “elites” who are thought to be misusing their influence.  A true populist candidate will also practice a form of “exclusionary identity politics.”

“In addition to being antielitist, populists are always antipluralist.  Populists claim that they, and they alone, represent the people….The claim to exclusive representation is not an empirical one; it is always distinctly moral.  When running for office, populists portray their political competitors as part of the immoral, corrupt elite; when ruling, they refuse to recognize any opposition as legitimate.  The populist logic also implies that whoever does not support populist parties might not be a proper part of the people—always defined as righteous and morally pure.  Put simply, populists do not claim ‘We are the 99 percent.’  What they imply instead is ‘We are the 100 percent’.”

Populists from the right will generally be against “the elites,” but they also tend to be against minorities, immigrants and others at the bottom of the economic ladder—people deemed to be non-contributors.  As such, they are not necessarily to be provided the same benefits as “real people.”

“Populists pit the pure, innocent, always hardworking people against a corrupt elite who do not really work (other than to further their self-interest) and, in right-wing populism, also against the very bottom of society (those who also do not really work and live like parasites off the work of others).

“….populism is always a form of identity politics (though not all versions of identity politics are populist).  What follows from this understanding of populism as an exclusionary form of identity politics is that populism tends to pose a danger to democracy.  For democracy requires pluralism and the recognition that we need to find fair terms of living together as free, equal, but also as irreducibly diverse citizens.  The idea of the single, homogeneous, authentic people is a fantasy….”

Does Trump, who spent much of his campaign making derogatory remarks about various minorities, meet this antipluralist criterion?  Müller provides this comment.

“At a campaign rally in May, Trump announced that ‘The only important thing is the unification of the people—because the other people don’t mean anything’.”

What Trump seems to mean here is that his potential supporters are “the people” and everyone else is irrelevant.

Populist leaders also manage to form a more direct connection to followers, reaching them as individuals.  Müller refers to this interaction as being symbolic rather than specific with respect to policies.

“Apart from determining who really belongs to the people, populists therefore need to say something about the content of what the authentic people actually want.  What they usually suggest is that there is a singular common good, that the people can discern it and will it, and that a politician or a party….can unambiguously implement it as policy.”

Think “Make America Great Again.”

“Populists always want to cut out the middleman, so to speak, and to rely as little as possible on complex party organizations as intermediaries between citizens and politicians.  The same is true of wanting to be done with journalists: the media is routinely accused by populists of ‘mediating,’ which, as the very word indicates, is what they are actually supposed to do, but which is seen by populists as somehow distorting political reality.”

Trump has used Twitter to great advantage in talking directly to the people he wishes to reach.

“….’real Americans’ can be done with the media and have direct access (or, rather, the illusion of direct contact with) a man who is not just a celebrity; the self-declared ‘Hemingway of 140 characters’ uniquely tells it like it is.”

An insidious characteristic of populism is that it creates a ready and will-reinforcing explanation for failure that is capable of launching any number of conspiracy theories.

“….the problem is never the populist’s imperfect capacity to represent the people’s will; rather it’s always the institutions that somehow produce the wrong outcomes.  So even if they look properly democratic, there must be something going on behind the scenes that allows corrupt elites to continue to betray the people.  Conspiracy theories are thus not a curious addition to populist rhetoric; they are rooted in and emerge from the very logic of populism itself.”

Trump campaigned as if there was a handbook for populist politicians and he followed it item by item.  It is not too early in his administration to see that he continues to follow this populist script.

“Populist governance exhibits three features: attempts to hijack the state apparatus, corruption and ‘mass clientelism’ (trading material benefits or bureaucratic favors for political support by citizens who become the populists’ ‘clients’), and efforts systematically to suppress civil society.  Of course, many authoritarians will do similar things.  The difference is that populists justify their conduct by claiming that they alone represent the will of the people; this allows populists to avow their practices quite openly.  It also explains why revelations of corruption rarely seem to hurt populist leaders (think of Erdogan in Turkey or the far-right populist Jorg Haider in Austria).  In the eyes of their followers, ‘they’re doing it for us, ‘the one authentic people’.”

Think of Trump in the United States.

Trump is beginning his term in office heading in the same direction as other populist leaders.  An article in Bloomberg Businessweek by Marc Champion with Marek Strzelecki provides a look at Poland’s experience in the context of what we might expect from a Trump presidency.  In the print edition it was titled In Poland, the Stench of Swamp Clearing.  Online, the title was changed to What Happens If You #DrainTheSwamp? Poland May Have the Answer.

“Swept to power on a similar wave of anger against urban and political elites as President-elect Donald Trump in the U.S., Poland’s Law & Justice party has been purging the state of what, in their view, is the self-serving elite that misruled Poland for most of the last 27 years.”

“In the process, the government in Warsaw has run roughshod over the constitution and weakened its democracy, according to critics such as the European Union. After a year marked by shocks at the ballot box, Poland offers a cautionary tale for other countries with populist revolts.”

And here is the justification for their actions based on representing the “authentic people.”

“Law & Justice leader Jaroslaw Kaczynski and his loyalists fiercely deny accusations they are dismantling Poland’s democracy. They say that they are building a strong state and returning the country to its true historical path and Catholic values on behalf of ordinary Poles the old liberal elites ignored.”

And then we come to hijacking the state apparatus and suppressing civil society.

“Within months of winning power in October 2015, the government replaced more than 300 executives at state-run companies, records gathered by the Nowoczesna opposition party show. About 1,600 officials at state institutions were also uprooted, while the new candidates for the civil service no longer have to face the usual competitive hiring process.”

“About 130 journalists were fired from or left Poland’s public broadcasters, which were placed under direct government control. So was the prosecutor’s office.”

“In January, Standard & Poor’s downgraded Poland’s credit rating, citing concern that ‘Poland’s system of institutional checks and balances has been eroded significantly,’’ specifically the constitutional court, media and civil service.”

An article on the Voice of America website, Corruption Report: Turning to Populist Leaders May Make Things Worse, points out the tendency towards corruption inherent in governments led by populists.

“In countries with populist or autocratic leaders, we often see democracies in decline and a disturbing pattern of attempts to crack down on civil society, limit press freedom, and weaken the independence of the judiciary,” said Jose Ugaz, chair of Transparency International, as the group released its report Wednesday. ‘Instead of tackling crony capitalism, those leaders usually install even worse forms of corrupt systems’.”

This gives the reader some idea of where Trump seems to want to lead us.

Weapons of Math Destruction: The Higher Education Arms Race

In 1983, the magazine, U.S. News & World Report, took upon itself the chore of ranking the nation’s colleges and universities for excellence.  That effort grew and evolved algorithms for producing a ranking that has become an important factor in evaluating a school’s performance.  The producers of this report bragged in 2008 about the extent of their influence.

“When U.S. News started the college and university rankings 25 years ago, no one imagined that these lists would become what some consider to be the 800-pound gorilla of American higher education, important enough to be the subject of doctoral dissertations, academic papers and conferences, endless debate, and constant media coverage.  What began with little fanfare has spawned imitation college rankings in at least 21 countries, including Canada, China, Britain, Germany, Poland, Russia, Spain, and Taiwan.”

One might conclude from this that the ranking enterprise has been a roaring success.  Perhaps it has been, but only for the finances of those who produce it.  Instead, it serves as an example of what Cathy O’Neil refers to as a weapon of math destruction (WMD).  She includes this ranking exercise as one of numerous WMDs that afflict us in her book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.

U.S. News began their ranking activities by running a beauty contest basing its ranking on responses to a survey by university presidents.  Stanford won the first—a school one might expect to win a beauty contest.  But this was deemed to be too simpleminded—and could easily be copied by a competitor.

The U.S. News team had to arrive at a more sophisticated approach.  They knew which schools were expected to selected as the best, so whatever approach they chose had to favor the high-profile schools.  The natural thing to do then would be to assume all good schools would have the characteristics of the presumed best.  Since the actual characteristics of a great school are difficult to quantify, they had to use whatever was quantifiable.

“They couldn’t measure learning, happiness, confidence, friendships, or other aspects of a student’s four-year experience.”

“Instead they picked proxies that seemed to correlate with success.  They looked at SAT scores, student-teacher ratios, and acceptance rates.  They analyzed the percentage of incoming freshmen who made it to sophomore year and the percentage of those who graduated.  They calculated the percentage of living alumni who contributed money to their alma mater, surmising that if they gave a college money there was a good chance they appreciated the education there.  Three quarters of the ranking would be produced by an algorithm—an opinion formalized in code—that incorporated these proxies.  In the other quarter, they would factor in the subjective views of college officials throughout the country.”

A typical characteristic of a WMD is that there is no mechanism for feedback.  O’Neil uses the example of sports teams who can study the results of numerous interactions and formulate new strategies for obtaining statistically better results in those situations.  They can apply those strategies and observe any change in results.  These changes can then be used modify their procedures and try again.  This iteration is the process by which a good algorithm is produced.  But if you are ranking schools somewhat arbitrarily, there is no mechanism for feedback; the results are either credible or not.  If not credible, then the project likely dies.  But U.S. News already knew how to make their results credible.

“U.S. News’s first data-driven ranking came out in 1988, and the results seemed sensible.  However, as the ranking grew into a national standard, a vicious feedback loop materialized.  The trouble was that the rankings were self-reinforcing.  If a college fared badly in U.S. News, its reputation would suffer, and conditions would deteriorate.  Top students would avoid it, as would top professors.  Alumni would howl and cut back on contributions.  The ranking would tumble further.  The ranking, in short, was destiny.”

“Now the vast reputational ecosystem of colleges and universities was over shadowed by a single column of numbers.”

Since this ranking system became a national standard, whether it was worthy of that status or not, schools had no choice but to play by the rules the U.S. News journalists had established.  They had fifteen categories in which they could try to improve their grades—or cheat in order to inflate their scores.

“Some administrators have gone to desperate lengths to drive up their rank.  Baylor University paid the fee for admitted students to retake the SAT, hoping another try would boost their scores—and Baylor’s ranking.  Elite small schools, including Bucknell University in Pennsylvania and California’s Clarement McKenna, sent false data to U.S. News, inflating the SAT scores of their incoming freshmen.  And Iona College, in New York, acknowledged in 2011 that its employees had fudged numbers about nearly everything: test scores, acceptance and graduation rates, freshman retention, student-faculty ratio, and alumni giving”

Most schools invested money instead of cheating.  If they provided improved leisure activities, fancy student centers, better dorms, and better sports teams, they thought these would encourage more students to apply, so they could then turn more students away and improve their acceptance ratio (make it smaller).  If they invested in higher paid professors perhaps they could attract better students with higher SAT scores and up that grade.  If you wished to improve your ranking you had to spend more money—and spend it faster than the other schools were increasing their spending.

Incredibly, the U.S. News team never considered cost of education as a ranking criterion.  They claim they do this ranking for the benefit of the students so they can make better choices concerning schools, but did they think cost was of no interest to students?  Were there no students who were interested in a good education at a modest cost?  Of what benefit is it to have a market for higher education in which cost plays no role?

“By leaving cost out of the formula, it was as if U.S. News had handed college presidents a gilded checkbook.  They had a commandment to maximize performance in fifteen areas, and keeping costs low wasn’t one of them.  In fact, if they raised prices, they’d have more resources for addressing the areas where they were being measured.”

“Tuition has skyrocketed ever since.  Between 1985 and 2013, the cost of higher education rose by more than 500 percent, nearly four times the rate of inflation.”

The existence of the U.S. News rankings has required both schools and students to game the ranking system as best they can.

“All of this activity takes place within a vast ecosystem surrounding the U.S. News rankings, whose model functions as the de facto law of the land.  If the editors rejigger the weightings on the model, paying less attention to SAT scores, for example, or more to graduation rates, the entire ecosystem of education must adapt.  This extends from universities to consultancies, high school guidance departments, and, yes, the students.”

Colleges can no longer think in terms of students as independent entities, rather, they are members of an ensemble of students.  It is the ensemble against which schools are scored.  This makes acceptance decisions much more complicated.  Fortunately there are those who are prepared to provide the algorithms to help the school administrators maximize their scores. 

“As colleges position themselves to move up the U.S. News charts, they manage their student populations almost like an investment portfolio.”

Somewhat less directly, high school students are also driven by the U.S. News algorithm.

“Each college’s admissions model is derived, at least in part, from the U.S. News model, and each one is a mini-WMD.  These models lead students and their parents to run in frantic circles and spend obscene amounts of money.  And they’re opaque.  This leaves most of the participants (or victims) in the dark.  But it creates a big business for consultants….who manage to learn their secrets, either by cultivating sources at the universities or by reverse engineering their algorithms.”

The people who do not have a lot of money to spend are, as always, the victims.  They lose because the cost of education is being artificially driven up, and because the cost of getting admitted to a university is also being driven up.

“The victims, of course, are the vast majority of Americans, the poor and middle-class families who don’t have thousands of dollars to spend on courses and consultants.  They miss out on precious insider knowledge.  The result is an education system that favors the privileged.  It tilts against needy students, locking out the great majority of them—and pushing them down a path toward poverty.  It deepens the social divide.”

The Holocaust as History: People Like Us Killing Other People Like Us

Timothy Snyder has produced two extremely troubling books detailing the events that transpired in the lands caught between Hitler and Stalin.  The first, Bloodlands: Europe Between Hitler and Stalin, details the years between 1933 and 1945 and the actions taken by both Hitler and Stalin.  During that period 14 million non-combatants were killed in the bloodlands, a number far higher than the number of Jews killed.  In most cases, people died not in an impersonal, industrial mode, but because another person, often a neighbor, shot them.  The number of people who were capable of mass murder is truly frightening. 

Snyder’s intent is not to overshadow the fate of the Jews with the fates of others, but to point out that what we think of as the Holocaust was, as a stain on humanity, even worse than we thought.  He also tells us that the success the Germans had in killing Jews (and others) depended on the mass murders and deportations carried out by Stalin in Poland, Ukraine, Belarus, the Baltic states and western Russia before the Germans arrived. 

“In the name of defending and modernizing the Soviet Union, Stalin oversaw the starvation of millions and the shooting of three quarters of a million people in the 1930s.  Stalin killed his own citizens no less efficiently than Hitler killed the citizens of other countries.  Of the fourteen million people deliberately murdered in the bloodlands between 1933 and 1945, a third belong in the Soviet account.”

While the first book was focused on what happened, the second, Black Earth: The Holocaust as History and Warning, tried to explain why things happened the way they did—and found it necessary to remind us that what happened once can happen again.

The Holocaust and Auschwitz are so tightly coupled that the association tends to obscure critical facts surrounding that period of time and the events that occurred.  Most of the killing took place soon after the German invasion of Russian-controlled territories—before Auschwitz was established as a death factory.

“The word ‘Auschwitz’ has become a metonym for the Holocaust as a whole.  Yet the vast majority of the Jews had already been murdered, further east, by the time that Auschwitz became a major killing facility.  Yet while Auschwitz has been remembered, most of the Holocaust has been largely forgotten.”

“In the history of the Holocaust, Auschwitz was a place where the third technique of mass killing was developed, third in chronological order and also third in significance.  The most important technique, because it came first, because it killed the most Jews, and because it demonstrated that a Final Solution by mass killing was possible, was shooting over pits.  The next most important, and the next to be developed, was asphyxiation by the exhaust fumes of internal combustion engines.  At around the time that these carbon monoxide facilities were coming into use, in early 1942, the policy of murdering all Jews was extended from the occupied Soviet Union and occupied Poland to all lands that fell under German control.  Auschwitz became the major killing site for Jews in 1943 and 1944.”

The association of Auschwitz with the Holocaust is convenient for German memories.  It provides a view in which a few people were involved in an impersonal mechanism for sending people to their death, and suggests the false impression that many Germans could have been unaware of the slaughter of Jews.

“It is possible that some Germans did not know exactly what happened at Auschwitz.  It is not possible that many Germans did not know about the mass murder of Jews.  The mass murder of Jews was known and discussed in Germany, at least among family and friends, long before Auschwitz became a death facility.  In the East, where tens of thousands of Germans shot millions of Jews over hundreds of death pits over the course of three years, most people knew what was happening.  Hundreds of thousands of Germans witnessed killings, and millions of Germans on the eastern front knew about them.  During the war, wives and even children visited the killing sites; and soldiers and policemen and others wrote home to their families, sometimes with photographs, about the details.  German homes were enriched, millions of times over, by plunder from the murdered Jews, sent by post or brought back by soldiers and policemen on leave.”

Auschwitz is a convenient symbol of the Holocaust for the Russians as well because it was the only aspect of the Holocaust for which Soviets could deny any complicity.  This association allows the slaughter to be viewed as a German problem, when, in fact, most of the Jews killed were actually Soviet citizens who were often murdered by other Soviet citizens providing assistance to the German invaders.

“This historical reality remains thoroughly politicized….that tens of thousands of Soviet citizens could contribute to the murder of further millions of Soviet citizens on behalf of a totally alien system, has never been addressed.  It has instead been displaced.”

When we think of the Holocaust we should not focus on the image of Auschwitz, but rather on an image of an individual looking into the face of a man, woman, or child and pulling a trigger—perhaps hundreds or thousands of times.  Most often the killing was of Jews, but the killers were capable of murdering whoever was brought before them.

“When the mass murder of Jews is limited to an exceptional place and treated as the result of impersonal procedures, then we need not confront the fact that people not very different from us murdered other people not very different from us at close quarters.”

What was it about the conditions in the bloodlands, starting in 1941, that made it so easy to prosecute this highly personal form of mass murder?  Snyder says it cannot be explained as acts of rabid anti-Semitism.  Anti-Semitism was common in every country in which Christianity took hold.  It is comforting for some to believe that the inhabitants of the bloodlands were somehow more savage, and less civilized than themselves—a self-serving view that is difficult to justify.  The Germans did try to generate pogroms as they moved into new lands, but found the process to be not very productive by their standards.

“If the killing of 1941 involved locals, then perhaps it was a result of local antisemitism rather than German politics?  This is a popular way to explain the Holocaust without politics: as a historically predictable outburst of the barbarity of east Europeans.  This sort of explanation is reassuring, since it permits the thought that only peoples associated with extravagant antisemitism would indulge in disastrous violence.  This comforting and erroneous thought is a legacy of Nazi racism and colonialism.  The racist and colonial idea that the Holocaust began as an elemental explosion of primitive antisemitism arose as Nazi propaganda and apologetics.  The Germans wished to display the killing of Jews on the eastern front as the righteous anger of oppressed peoples against their supposed Jewish overlords.”

Snyder suggests that for understanding what transpired we must recognize the unique political situations that occurred in the various regions of the bloodlands.  The survivability of Jews depended a lot on which country claimed them as citizens.  Germans occupied several countries in Western Europe, such as France, Norway, Denmark, and the Netherlands, but retained the governments in them in order to run the countries.  In order for Germans to gain access to most of the resident Jews, the officials of the given country would have to provide them with the Jews.  This happened to a different degree from country to country, but as a rule, the stronger the occupied government was—meaning the degree to which it had maintained the prewar government—the greater the probability of survival.  The Germans in the occupied countries—and even in Germany itself—seemed driven by the need to obey laws.  Consider the fate of Victor Klemperer a German Jew who was a noted scholar.

“Because Klemperer was a German citizen with a non-Jewish wife, he was not subject to the general policy of the deportation and murder of German Jews.  Since his wife did not divorce him, he, like many such German Jewish men, survived.”

Consider the experience in France.

“The French placed Jews without French citizenship in camps.  The Germans wanted to take such people, but only insofar as the Germans themselves could consider them stateless.  Crucially, Nazi malice stopped at the passport: As much as Nazis might have imagined that states were artificial creations, they did not proceed with killing Jews until states were actually destroyed or had renounced their own Jews.  The French were willing to round up Jews from Hungary and Turkey, for example, but the Germans were unwilling to kill such people without the consent of the Hungarian and Turkish governments.  Germany was entirely willing to murder Jews of Polish and Soviet citizenship, since it considered those states to be defunct.  Germany was also willing to take and murder French Jews, but only under the condition that French authorities first stripped such people of citizenship.”

“A large majority of French Jews, about three-quarters survived the war.”

It would be in the regions where the prewar government was totally destroyed that the Germans received the most help from local residents, and were most efficient in their killing.  In fact, the bloodiest regions were those where a double governmental destruction occurred. 

Under the terms of the German-Russian nonaggression treaty, Russia was allowed to take control of not only the eastern regions of prewar Poland, but also Finland and the Baltic countries of Latvia, Lithuania, and Estonia.  The Russians occupied the Baltic States in 1940, destroyed the existing governments, and killed or shipped to work camps everyone who might be a problem.  Included in that category were members of government, past or present, anyone who might be a leader or opinion maker, and people who had too much property or money.  They formed a new government that lasted about a year until the Germans invaded and drove the Russians out, establishing yet another new government.  Estonia, having suffered this fate, could be used by Snyder as an example of how these regions differed from those who were merely occupied by the Germans.  In particular, he compares the fates of Jews in both Estonia and Denmark, similar countries with small Jewish populations.

“During the war both were under German occupation, both were subject to the Final Solution, and both were declared judenfrei—free of Jews—by their German occupiers.  And yet the history of the Holocaust in each land could hardly have been more different.  In Estonia, about 99 percent of the Jews who were present when German forces arrived were killed.  In Denmark, about 99 percent of Jews who had Danish citizenship survived.”

“In no country under German occupation did a higher percentage of Jews die than in Estonia, and in none did a higher proportion survive than in Denmark.”

Was the difference related to a more intense anti-Semitism in Estonia than in Denmark?

“In fact, Estonian Jews were equal citizens of the republic, which took in some Jewish refugees from Austria and Germany.  Denmark, by contrast, turned away Jewish refugees after 1935.”

Denmark was allowed to maintain their government in place after the German occupation.  The Germans had neither the manpower nor the will to coerce Denmark into treating Jewish citizens differently than other citizens.  Germany did wish to have Denmark declared free of Jews.  This was accomplished by Germany tacitly agreeing to let Denmark ship all of its Jews to Sweden, which, as a nominally neutral nation, but one aiding the Germans economically, wished to score points with the allied nations.

On the other hand, when the Germans moved into Estonia they found a country with no credible government.  The elected government and the participants were all dead or gone and replaced with what could be referred to as collaborators with an invader.  To many, Germany would arrive as a liberator.  Those who collaborated with the Russians were at least tacit accomplices in the murder and internment of their own countrymen, leaving them little option but to play the same role for the new invaders.  The Germans also noted that many of those the Russians had sent to far off camps were Jews with property that had been confiscated by their neighbors, who would have an interest in seeing that those Jews never return.  And there was also the long propagated Judeobolshevik myth which blamed the Jews for Russian communism.  In many countries the Germans took over from the Soviets, the Jews were a minority of the collaborators, but more than proportionately represented.  The final gift provided the Germans by the Soviets was to kill all the native prisoners and leave them behind as they retreated and left their collaborators to fend for themselves.  There was also the issue of food.  The German army was forced to live off the land.  The disruption of the multiple invasions did not help food production.  The argument that fewer mouths to feed would benefit everyone was easy to make.

Germans would often provoke pogroms when entering a new country.  They knew that this was inefficient as a method of killing Jews, but it was a useful technique for spotting people who were willing to kill.  If local recruiting methods failed, the Germans had accumulated millions of Russian prisoners during the war.  They began by letting them starve to death, but they soon realized that given the alternative of starving, many of these would be willing to participate in the killing.  The net result was that between the Germans themselves, the Russian prisoners, and locals, there were sufficient numbers of people willing to kill.  And it was difficult to ascribe this to anti-Semitism because the killing always involved other groups as well as Jews.  In Estonia, the Jews were a small fraction of those ordered killed, but the Germans still had little trouble finding willing killers.

“Because there were very few Jews in Estonia, the number of non-Jews was relatively more important than elsewhere.  All of the 963 Estonian Jews murdered under German occupation were killed by Estonians, usually policemen.  About ten times as many non-Jewish Estonians were killed by those same Estonian policemen.”

It is the availability of all those people the Germans found who were ready to become mass murderers that should concern us.  What exactly are the conditions under which “people like us” become capable of killing other people like us? 

Snyder talks long and often about political situations for which Estonia is an example and Denmark is a counterexample.  He emphasizes the “double occupation” scenario as being of particular importance in triggering such a response.  Clearly there is a correlation between mass murder and political considerations, and some truth in his hypothesis.  But it was Snyder who emphasized that it was important to focus not on the “impersonal procedures” associated with Auschwitz, but on the very personal procedure involved in putting a bullet in a person standing nearby and looking directly into one’s eyes. Snyder’s political theories seem just a bit too impersonal to be satisfying, and perhaps a bit too specific to the unique conditions of World War II.  And one yet has to explain the willingness of the Germans to become mass murderers.  They created the politics Snyder says is necessary for mass murder, they did not suffer from it.

A previous discussion suggested that the conditions that would explain the observed mass murders were some combination of peer pressure, respect for authority, and a sense of being mortally threatened.  One could attribute the motives of the killers the Germans recruited in the bloodlands to this combination of factors and arrive at a more useful paradigm than the more complex political construction of Snyder.

Consider this passage from Snyder discussing a letter written home by a German soldier.

“Even as the German army was advancing east in huge numbers, the German killers presented their actions as defensive.  To shoot Jewish babies in Mahileu was, as one German (Austrian) explained to his wife, to prevent something worse: ‘During the first try, my hand trembled a bit as I shot, but one gets used to it.  By the tenth try I aimed calmly and shot surely at the many women, children, and infants.  I kept in mind that I have two infants at home, whom these hordes would treat just the same, if not ten times worse.  The death that we gave them was a beautiful quick death, compared to the hellish torments of thousands and thousands in the jails of the GPU [Soviet secret police organization].  Infants flew in great arcs through the air and we shot them to pieces in flight, before their bodies fell into the pit and into the water.”

This soldier, after a decade of Nazi propaganda, could blame the capitalist Jews and the Bolshevik Jews for threatening Germany from all sides and forcing them to go to war.  He was also under severe peer pressure as his comrades were also participating in the killing, as well as having the excuse that he was just following orders. 

This soldier could have been a postman or a school teacher in civilian life, but conditions were capable of converting him into a mass murderer.  That is a lesson that must be learned from the Holocaust.

As an aside, Snyder provides us with a revealing quote from Hitler.

“The extermination of the Jews was a victory for the species, regardless of the defeat of the Germans.  As Hitler said at the very end, on April 29, 1945, Jews were the ‘world poisoners of all nations.’  He was sure of his legacy: ‘I have lanced the Jewish boil.  Posterity will be eternally grateful to us’.”

Take a man with some oratorical skills, an absolute certainty in his beliefs, and an obsession with a perceived enemy; give him power and terrible things can happen.  Hitler and the Holocaust are unlikely to be repeated.  But mass murder occurs regularly.  People like Hitler are always among us; the important thing is to not let them come to power.

Adam Gopnik wrote a piece for The New Yorker before the election directed at those who were worried about the rise of Donald Trump.  It included this warning—one we should continue to take to heart as we move forward.

“He’s not Hitler, as his wife recently said? Well, of course he isn’t. But then Hitler wasn’t Hitler—until he was. At each step of the way, the shock was tempered by acceptance. It depended on conservatives pretending he wasn’t so bad, compared with the Communists, while at the same time the militant left decided that their real enemies were the moderate leftists, who were really indistinguishable from the Nazis. The radical progressives decided that there was no difference between the democratic left and the totalitarian right and that an explosion of institutions was exactly the most thrilling thing imaginable.”

Interested readers might find these articles informative:

Humanity and Violence

The Fragility of Civilization

Are Humans Inherently Warlike?

The Popes Against the Jews: Providing a Path to the Holocaust

War and Remembrance: The Casualties of World War II

Big Data: Its Role in Increasing Inequality

We are living in the age of big data.  Everything we do, everywhere we go, every website we visit, every book we read, every comment we make on social media, every email we write, who we phone is being examined by people who wish to make a profit from information about us.  Everything known about you is fed into analysis routines that try to characterize you and predict how you are likely to behave as a consumer, as a voter, as an employee, as a member of an insurance plan, even as a lover.  This data is being used in ways we have no control over and drawing conclusions about us that may be quite in error yet still determining whether or not we are worthy of a job, a loan, or even a prison sentence.

Sue Halpern provided valuable insight into this issue in an article that appeared in The New York Review of Books: They Have, Right Now, Another You.  The accuracy of the profiles obtained from big data techniques is of great interest to her

She tells us that Facebook accumulates 98 data points that it uses to characterize an individual.  Some of these are self reported by the individual of interest, while most are extracted via other means.  For example, if you provide Facebook with a photo of yourself, its facial recognition software is good enough to pick you out of other peoples’ photographs.  It can clearly mine information from posts by you and those who it associates with you, but since they wish to make money by selling you to vendors, they need to learn more than you are likely to be willing to share.

“Facebook also follows users across the Internet, disregarding their ‘do not track’ settings as it stalks them. It knows every time a user visits a website that has a Facebook ‘like’ button, for example, which most websites do.”

“The company also buys personal information from some of the five thousand data brokers worldwide, who collect information from store loyalty cards, warranties, pharmacy records, pay stubs, and some of the ten million public data sets available for harvest. Municipalities also sell data—voter registrations and motor vehicle information, for example, and death notices, foreclosure declarations, and business registrations, to name a few. In theory, all these data points are being collected by Facebook in order to tailor ads to sell us stuff we want, but in fact they are being sold by Facebook to advertisers for the simple reason that the company can make a lot of money doing so.”

Halpern managed to delve into Facebook’s assumed knowledge about her and discovered that its information was often comically wrong.  This is what she learned:

“That I am interested in the categories of ‘farm, money, the Republican Party, happiness, gummy candy, and flight attendants’ based on what Facebook says I do on Facebook itself. Based on ads Facebook believes I’ve looked at somewhere—anywhere—in my Internet travels, I’m also interested in magnetic resonance imaging, The Cave of Forgotten Dreams, and thriller movies. Facebook also believes I have liked Facebook pages devoted to Tyrannosaurus rex, Puffy AmiYumi, cookie dough, and a wrestler named the Edge.”

“But I did not like any of those pages, as a quick scan of my ‘liked’ pages would show. Until I did this research, I had never heard of the Edge or the Japanese duo Puffy AmiYumi, and as someone with celiac disease, I am constitutionally unable to like cookie dough.”

If there is one thing Facebook should know about an individual it is her list of pages she has actively liked.  She then asks a troubling question: Is Facebook possibly this inaccurate, or has it decided she is more valuable as an asset if she is presented as a more marketable consumer.

“But maybe I am more valuable to Facebook if I am presented as someone who likes Puffy AmiYumi, with its tens of thousands of fans, rather than a local band called Dugway, which has less than a thousand. But I will never know, since the composition of Facebook’s algorithms, like Google’s and other tech companies’, is a closely guarded secret.”

Halpern also presents results from an encounter with a group of researchers at the Psychometrics Centre at Cambridge University.  This outfit attempts to derive a personality profile based on a person’s Facebook information.

“    [they] developed what they call a ‘predictor engine,’ fueled by algorithms using a subset of a person’s Facebook ‘likes’ that ‘can forecast a range of variables that includes happiness, intelligence, political orientation and more, as well as generate a big five personality profile.’ (The big five are extroversion, agreeableness, openness, conscientiousness, and neuroticism, and are used by, among others, employers to assess job applicants. The acronym for these is OCEAN.) According to the Cambridge researchers, ‘we always think beyond the mere clicks or Likes of an individual to consider the subtle attributes that really drive their behavior.’ The researchers sell their services to businesses with the promise of enabling ‘instant psychological assessment of your users based on their online behavior, so you can offer real-time feedback and recommendations that set your brand apart’.”

Again, Halpern was presented results that were bizarrely inaccurate.

“So here’s what their prediction engine came up with for me: that I am probably male, though ‘liking’ The New York Review of Books page makes me more ‘feminine’; that I am slightly more conservative than liberal—and this despite my stated affection for Bernie Sanders on Facebook; that I am much more contemplative than engaged with the outside world—and this though I have ‘liked’ a number of political and activist groups; and that, apparently, I am more relaxed and laid back than 62 percent of the population. (Questionable.)”

“Here’s what else I found out about myself. Not only am I male, but ‘six out of ten men with [my] likes are gay,’ which gives me ‘around an average probability’ of being not just male, but a gay male. The likes that make me appear ‘less gay’ are the product testing magazine Consumer Reports, the tech blog Gizmodo, and another website called Lifehacker. The ones that make me appear ‘more gay’ are The New York Times and the environmental group 350.org. Meanwhile, the likes that make me ‘appear less interested in politics’ are The New York Times and 350.org.”

“And there’s more. According to the algorithm of the Psychometrics Centre, ‘Your likes suggest you are single and not in a relationship.’ Why? Because I’ve liked the page for 350.org, an organization founded by the man with whom I’ve been in a relationship for thirty years!”

These results can be amusing, but one must not forget that these types of data interpretations are being used to determine peoples’ lives.  Halpern wrote her article partly as a review of the book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy by Cathy O’Neil, which she correctly describes as “insightful and disturbing.”

O’Neil has coined the phrase Weapons of Math Destruction (WMDs) to describe those algorithms that are either poorly constructed or misused in such a way that they are capable of causing extreme harm.  Algorithms and big data collection allow millions of people to be quickly evaluated, correctly or not, potentially spreading pain and suffering nationwide.

One of the presumed advantages of using mathematical algorithms to characterize individuals is that the particular biases of an individual are eliminated from the process.  However, the creation of the algorithm inevitably involves the biases of the creators.  While the errors in judgment made by an individual can be recognized because they are there for others to see, the false assumptions built into an algorithm are often hidden from view, protected as “proprietary information.”

If one wishes to evaluate applicants for a job, one might wish to evaluate an individual’s, honesty, conscientiousness, reliability, and creativity.  These are all quantities that are nearly impossible to quantify.  But algorithms can only deal with things that are quantifiable, therefore, they must select proxies for those attributes that can be converted to numbers that may or not be directly relevant.  Credit scores are a popular proxy that is used in evaluations, but a low credit score can be attained by a person with a poor sense of responsibility as well as by a highly responsible person who has just had a run of bad luck.  A personal evaluation could address those differences, but there is no way for the algorithm to know.  An employer who utilizes a mathematical routine that eliminates persons with low credit scores will likely be satisfied with the process and not realize that he is throwing away a number of potentially excellent employees.  There is no feedback possible to evaluate this process.  As long as employees are attained who allow the employer to continue to make money he will be happy.  Profit is the only scorecard—a crude one at best.

Another common technique used by algorithms is to judge people by who they digitally resemble.  An algorithm can create a collection of characteristics of a desirable person and a collection of attributes of an undesirable person.  An individual is then assumed to be similar to the type of person he most looks like.  This approach essentially eliminates any attempt to evaluate a person by their individual characteristics and renders them equal to their presumed look-alikes.  There are plenty of biases that can be built into this approach.  While bias based on race is technically illegal, there are many ways in which a person living in a poor neighborhood with a high crime rate and where interactions with police are frequent can be deemed undesirable even if he/she has an unblemished record.

O’Neil provides numerous examples of how this mass production approach tends to be biased against poor people.  Consider the case of the person who has been rejected for a job because a high medical bill or unemployment caused him to fall behind on payments and lowered his credit score.

“….the belief that bad credit correlates with bad job performance leaves those with low scores less likely to find work.  Joblessness pushes them toward poverty, which further worsens their scores, making it even harder for them to land a job.  It’s a downward spiral.  And employers never learn how many good employees they’ve missed out on by focusing on credit scores.  In WMDs, many poisonous assumptions are camouflaged by math and go largely untested and unquestioned.”

“This underscores another common feature of WMDs.  They tend to punish the poor.  This is, in part, because they are engineered to evaluate large numbers of people.  They specialize in bulk, and they’re cheap.  That’s part of their appeal.  The wealthy, by contrast, often benefit from personal input.  A white-shoe law firm or an exclusive prep school will lean far more on recommendations and face-to-face interviews than will a fast-food chain or a cash-strapped urban school district.  The privileged, we’ll see time and again, are processed more by people, the masses by machines.”

These algorithms that have become so prevalent are usually formulated with the best of intentions.  In many cases they will work quite well—on the average.  But what about the cases in which they make a wrong decision?  In these cases the victims have no recourse.  It is often the case that both the user of the tool and the victim have no idea why he/she was deselected.  And if a person is deselected in one area, they could find themselves deselected in other types of applications as well.

“….scale is what turns WMDs from local nuisances to tsunami forces, ones that define and delimit our lives.  As we’ll see, the developing WMDs in human resources, health, and banking, just to name a few, are quickly establishing broad norms that exert upon us something very close to the power of law.  If a bank’s model of a high risk borrower, for example, is applied to you, the world will treat you as just that, a deadbeat—even if you’re horribly misunderstood.  And when that model scales, as the credit model has, it affects your whole life—whether you can get an apartment or a job or a car to get from one to another.”

Inequality is enhanced in more subtle ways.  Big data can be used to pinpoint vulnerable people and take advantage of that vulnerability.

“We are ranked, categorized, and scored in hundreds of models, on the basis of our revealed preferences and patterns.  This establishes a powerful basis for legitimate ad campaigns, but it also fuels their predatory cousins: ads that pinpoint people in great need and sell them false or overpriced promises.  They find inequality and feast on it.  The result is that they perpetuate our existing social stratification, with all of its injustices.”

It has been demonstrated that degrees from for-profit colleges are of little value to students.  They are much more expensive than equivalent education from a community college and less highly valued by employers—in fact, little better than a high school education.  These schools make nearly all their money from government-guaranteed loans.  Whether students succeed or fail has little to do with their business plan.

“In education, they promise what’s usually a false road to prosperity, while also calculating how to maximize the dollars they draw from each prospect.  Their operations cause immense and nefarious feedback loops and leave their customers buried under mountains of debt.”

“Vatterott College, a career-training institute, is a particularly nasty example.  A 2012 Senate committee report on for-profit colleges described Vatterott’s recruiting manual, which sounds diabolical.  It directs recruiters to target ‘Welfare Mom w/Kids.  Pregnant Ladies.  Recent Divorce.  Low Self-Esteem. Low Income Jobs.  Experienced a Recent Death.  Physically/Mentally Abused.  Recent Incarceration.  Drug Rehabilitation.  Dead-End Jobs—No Future.’”

The word is gradually getting out about these organizations and a few of them have been forced to close. 

If one is concerned about the uses to which our personal data is being put, what can one do about it?  One possibility suggested by O’Neil is to produce legislation similar to that in place in Europe.

“If we want to bring out the big guns, we might consider moving toward the European model, which stipulates that any data collected must be approved by the user as an opt-in.  It also prohibits the reuse of data for other purposes.  The opt-in condition is all too often bypassed by having a user click on an inscrutable legal box.  But the ‘not reusable’ clause is very strong: it makes it illegal to sell user data.  This keeps it from the data brokers whose dossiers feed toxic e-scores and microtargeting campaigns.”

O’Neil points out that many WMDs could be turned into beneficial tools.  The knowledge that allows people to be deemed vulnerable and targeted for harmful advertizing could be used instead to identify people who are in need of social assistance.  One might even consider providing that social assistance.  What a concept!

Education Policy in the USA: Our Nation Was Not and Is Not At Risk!

We are living in the age of big data.  Everything we do, everywhere we go, every website we visit, every book we read, every comment we make on social media is being recorded by people who wish to make a profit out of information about us.  Everything known about you is fed into analysis routines that try to characterize you and predict how you are likely to behave as a consumer, as a voter, as an employee, as member of an insurance plan…..  The list of uses of your information is already long and continues to grow.  Cathy O’Neil has written a fascinating, and somewhat scary, evaluation of where this big data economy is taking us in her book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy.  She describes a number of troubling developments that have received little discussion.  One of the areas in which supposedly unbiased data analysis routines inject bias into evaluations of individuals is the ranking of teachers.  In a brief preface to that discussion O’Neil provided a startling revelation about a famous (now infamous) document that altered the history of education in this country and fueled a movement to replace public education by private education.  That revelation will be the topic here.

Back in 1983 the economy was not particularly healthy, and countries such as Japan and Germany seemed to have passed us by economically.  Someone had to be at fault.  A group was convened during Reagan’s administration to determine how our education system could be the problem.  This group issued a document titled A Nation at Risk.  One of the conclusions of the report was that average SAT (Scholastic Aptitude Test—used for college admission) scores had been falling for a considerable period; therefore our schools are failing us and putting our national security at risk.  It turns out those who wrote the report got the math all wrong.

“In fact, misinterpreted statistics run through the history of teacher evaluation.  The problem started with a momentous statistical boo-boo in the analysis of the original Nation at Risk report.  It turned out that the very researchers who were decrying a national catastrophe were basing their judgment on a fundamental error, something an undergrad should have caught.  In fact, if they wanted to serve up an example of America’s educational shortcomings, their own misreading of statistics could serve as exhibit A.”

There is something called Simpson’s Paradox, a situation where the average of a population can trend in one direction while subsets of the population can have averages that trend in the other direction.  This can occur when the numbers in the population subgroups are changing.  It is well known that performance on the SAT correlates strongly with family income.  Over the period studied, a large number of lower-income applicants began taking the SAT test causing the average to go down.  However, if the test takers were broken into income groups, then the data indicated that SAT scores were actually increasing for all income groups.  The education problem did not exist.

“Seven years after A Nation at Risk was published with such fanfare, researchers at Sandia National Laboratories took a second look at the data gathered for the report.  These people were no amateurs when it came to statistics—they build and maintain nuclear weapons—and they quickly found the error.  Yes, it was true that SAT scores had gone down on average.  However, the number of students taking the test had ballooned over the course of those seventeen years.  Universities were opening their doors to more poor students and minorities.  Opportunities were expanding.  This signaled social success.  But naturally, this influx of newcomers dragged down the average scores.  However, when statisticians broke down the population into income groups, scores for every single group were rising, from poor to the rich.”

“The damning conclusion in the Nation at Risk report, the one that spurred the entire teacher evaluation movement, was drawn from a grievous misinterpretation of data.”

O’Neil references an article by Tamim Ansary written in 2007, Education at Risk: Fallout from a Flawed Report, as the source for this information.  Ansary provides us with valuable background information.

“What we now call school reform isn't the product of a gradual consensus emerging among educators about how kids learn; it's a political movement that grew out of one seed planted in 1983. I became aware of this fact some years ago, when I started writing about education issues and found that every reform initiative I read about -- standards, testing, whatever -- referred me back to a seminal text entitled ‘A Nation at Risk’."

“Naturally, I assumed this bible of school reform was a scientific research study full of charts and data that proved something. Yet when I finally looked it up, I found a thirty-page political document issued by the National Commission on Excellence in Education, a group convened by Ronald Reagan's secretary of education, Terrell Bell.”

Ansary provides a direct quote from the document to illustrate the political hyperbole.

"Our Nation is at risk . . . . The educational foundations of our society are presently being eroded by a rising tide of mediocrity that threatens our very future as a Nation and a people . . . . If an unfriendly foreign power had attempted to impose on America the mediocre educational performance that exists today, we might well have viewed it as an act of war . . . . We have, in effect, been committing an act of unthinking, unilateral educational disarmament . . . ."

It seems Reagan was, as usual, clueless about the meaning of the report.

“As commission member Gerald Holton recalls, Reagan thanked the commissioners at a White House ceremony for endorsing school prayer, vouchers, and the elimination of the Department of Education. In fact, the newly printed blue-cover report never mentioned these pet passions of the president. ‘The one important reader of the report had apparently not read it after all,’ Holton said.”

However, Reagan’s handlers saw the advantage that could be had.

“Once launched, the report, which warned of ‘a rising level of mediocrity,’ took off like wildfire. During the next month, the Washington Post alone ran some two dozen stories about it, and the buzz kept spreading. Although Reagan counselor (and, later, attorney general) Edwin Meese III urged him to reject the report because it undermined the president's basic education agenda -- to get government out of education -- White House advisers Jim Baker and Michael Deaver argued that ‘A Nation at Risk’ provided good campaign fodder.”

“Reagan agreed, and, in his second run for the presidency, he gave fifty-one speeches calling for tough school reform. The ‘high political payoff,’ Bell wrote in his memoir, ‘stole the education issue from Walter Mondale -- and it cost us nothing’."

“What made ‘A Nation at Risk’ so useful to Reagan? For one thing, its language echoed the get-tough rhetoric of the growing conservative movement. For another, its diagnosis lent color to the charge that, under liberals, American education had dissolved into a mush of self-esteem classes.”

But what about the Sandia analysis that indicated that so many of the conclusions of the report were wrong?  Ansary tells us that, for reasons that are not clear, the secretary of energy, Admiral James Watkins, asked one of the Department of Energy National Laboratories, Sandia, to reevaluate the issue in 1990 using the relevant data. 

“Systems scientists there produced a study consisting almost entirely of charts, tables, and graphs, plus brief analyses of what the numbers signified, which amounted to a major ‘Oops!’ As their puzzled preface put it, ‘To our surprise, on nearly every measure, we found steady or slightly improving trends’."

So, the original conclusions were proven wrong.  There must have been some reaction.  Right?

“The government never released the Sandia report. It went into peer review and there died a quiet death. Hardly anyone else knew it even existed until, in 1993, the Journal of Educational Research, read by only a small group of specialists, printed the report.”

Ansary does not say when he became aware of the Sandia report, but one suspects it took him until near the time of his article, 2007, to realize it existed.  Meanwhile, in spite of the existence of the Sandia conclusions, each successive president has tried to demonstrate the ability to be tougher on our “failing” school systems. 

The Republican Party is still trying to eliminate the Department of Education, and it is still trying to take education out of the hands of educators and turn control over to profit-making or religious entities.  All the initiatives, such as vouchers to provide choice, increases in charter schools, and destruction of teachers’ unions, are aimed at funneling public education funds into private pockets.

The truth is that students with a good economic backing, either from family or from the state, do well in school compared to students from other countries.  Students from poor backgrounds, either due to family or to state circumstances, do poorly.  As long as the number of poorly-supported children is maintained or grows, we, as a country, will average out to something that could be called mediocrity.  The problem is not our education system; the problem is our political system.

The interested reader might find the following articles informative:

Could Our Schools Be a Threat to Our National Security?

How Your State’s Students Compare with Those from Other Countries