Tag Archives: Filter

Who’s Your Filter? (Nate Silver Edition)

“There are two ways to be fooled. One is to believe what isn’t true; the other is to refuse to believe what is true.” ~ Søren Kierkegaard

Back in 2010, I wrote a short post about some of the problems associated with getting all of your news and information from biased sources. It was essentially a call for people to hone their critical thinking skills and take steps toward establishing a more reality-based approach to decision-making.

Unfortunately, people don’t like challenging their existing beliefs very much because it can be pretty uncomfortable. They prefer sources of information that support their established worldviews and generally ignore or filter out those that don’t. In our modern society, this confirmation bias supports an entire ecosystem of publishers, news outlets, TV shows, bloggers, and radio announcers designed to serve up pre-filtered opinion disguised as fact.

For many people, the glossy veneer of the news entertainment complex is all they want or need. As David McRaney so succinctly states in his blog:

Whether or not pundits are telling the truth, or vetting their opinions, or thoroughly researching their topics is all beside the point. You watch them not for information, but for confirmation.

The problem with this approach is that — every now and then — fantasy runs into cold, hard reality and gets the sh*t kicked out of it.

This was what happened during the 2012 Presidential election cycle. Talking heads on both ends of the political spectrum had spent months trying to sway their audiences with confident declarations of victory and vicious denials of opposing statements. By the week of the election, the conservative media in particular had created such a self-reinforcing bubble of polls and opinions that any hints of trouble were shouted down and ignored. Pundits reserved particularly strong venom for statistician Nate Silver, whose FiveThirtyEight blog in the New York Times had upped the chances of an Obama win to a seemingly outrageous 91.4% the Monday before the election.

The furor reached its peak with Karl Rove’s famous on-air exchange with FOX news anchor, Megyn Kelly, and rippled through the conservative echo chamber after the polls closed. There was a lot of soul searching over the next few days, with many people taking direct aim at the conservative media for its failure to present accurate information to its audience. This frustration was summed up clearly by one commenter on RedState, a right-leaning blog:

“I can accept that my news is not really ‘news’ like news in Cronkite’s day, but a conservative take on the news. But it’s unacceptable that Rasmussen appears to have distinguished themselves from everyone else in their quest to shade the numbers to appease us, the base. I didn’t even look at other polls, to tell the truth, trusting that their methodology was more sound because it jived with what I was hearing on Fox and with people I talked to. It pains me to say this, but next time I want a dose of hard truth, I’m looking to Nate Silver, even if I don’t like the results.”

It was a teachable moment and Nate Silver — no fan of pundits — suggested that the fatal flaw in the approach taken by most of these political “experts” was that they based their forecasts less on evidence and more on a strong underlying ideology. Their core beliefs — “ideological priors” as Silver calls them — colored their views on everything and made it difficult to read such an uncertain situation correctly. It was time for something new.

In his book, The Signal and the Noise, Silver elaborates on the work of Philip Tetlock, who found that people with certain character traits typically made more accurate predictions than those without these traits. Tetlock identified these two different cognitive styles as either “fox” (someone who considers many approaches to a problem) or “hedgehog” (someone who believes in one Big Idea). There has been much debate about which one represents the best approach to forecasting but Tetlock’s research clearly favors the fox.

Tetlock’s ideas as summarized by Silver:

Fox-Like Characteristics Hedgehog-Like Characteristics
Multidisciplinary – Incorporates ideas from a range of disciplines Specialised – Often dedicated themselves to one or two big problems & are sceptical of outsiders
Adaptable – Try several approaches in parallel, or find a new one if things aren’t working Unshakable – New data is used to refine an original model
Self-critical – Willing to accept mistakes and adapt or even replace a model based on new data Stubborn – Mistakes are blamed on poor luck
Tolerant of complexity – Accept the world is complex, and that certain things cannot be reduced to a null hypothesis Order seeking – Once patterns are detected, assume relationships are relatively uniform
Cautious – Predictions are probabilistic, and qualified Confident – Rarely change or hedge their position
Empirical – Observable data is always preferred over theory or anecdote Ideological – Approach to predictive problems fits within a similar view of the wider world
Better Forecasters Weaker Forecasters

Nate Silver also prefers the fox-like approach to analysis and even chose a fox logo for the relaunch of his FiveThirtyEight blog. As befitting a fox’s multidisciplinary approach to problems, his manifesto for the site involves blending good old-fashioned journalism skills with statistical analysis, computer programming, and data visualization. (It is essentially a combination of everything we’ve been saying about data science + data-literate reporting.)

Nate Silver’s Four-Step Methodology for Data Journalism
This approach is very similar to the standard data science process.

  1. Data Collection – Performing interviews, research, first-person observation, polls, experiments, or data scraping
  2. Organization – Developing a storyline, running descriptive statistics, placing data in a relational database, or building a data visualization.
  3. Explanation – Performing traditional analysis or running statistical tests to look for relationships in the data.
  4. Generalization – Verifying hypotheses through predictions or repeated experiments.

Like data science, data journalism involves finding meaningful insights from a vast sea of information. And like data science, one of the biggest challenges to data-driven journalism is convincing people to actually listen to what the data is telling them. After FiveThirtyEight posted its prediction of a possible change in control of the Senate in 2014, Democrats have reacted with the same bluster as Republicans did back in 2012. At about the same time, economist Paul Krugman started a feud with Silver over — in my view — relatively minor journalistic differences. Meanwhile, conservatives gleeful at this apparent Leftie infighting continue to predict Silver’s ultimate failure because they still believe that politics is more art than science.

This seems to be a fundamental misunderstanding of what Silver and others like him are trying to do. Rather than look at how successful Silver’s forecasting methodology has been at predicting political results, most people seem to be treating him as just another pundit who has joined the political game. Lost in all of the fuss is his attempt to bring a little more scientific rigor to an arena that is dominated by people who generally operate on intuition and gut instinct. I’m certainly not trying to elevate statisticians and data journalists to god-like status here but it is my hope that people will start to recognize the value of unbiased evaluation and include it as one of their tools for gathering information. When it’s fantasy vs. reality, it is always better to be armed with the facts.

Update:

King Bhumibol is a Noob

In an effort to suppress disparaging remarks about the monarchy, the government of Thailand has recently established an official agency called the Office of Prevention and Suppression of Information Technology Crimes. The sole purpose of this department is to enforce the country’s lèse-majesté laws by combing the Internet for anything offensive to King Bhumibol Adulyadej and his family and then either eliminating or blocking the offensive material.

Agency technicians have apparently blocked over 70,000 pages so far, including those with pictures of the king with a foot above his head (considered very rude) and those that misuse informal pronouns before the king’s name.

Punishment for such disrespect for authority can be harsh. Under Thai law, even the digital distribution of information that threatens the “good morals of the people”  will get you five years in prison. For anyone who insults or defames the royal family, sentences can stretch to 15 years.

I am always surprised at the lengths to which repressive regimes will go in order to “safeguard” the sensibilities of its citizens while trying to maintain the openness and flexibility of the Internet. I’m even more surprised at this particular effort to shield a grown man from the forms of mild online abuse and disagreements that confront other world leaders every day.

This kind of experience can certainly be frustrating. However, is white-washing the Internet really the answer? Is it even possible? Wouldn’t everyone’s time be better spent teaching the king to deal with a few negative comments rather than censoring the entire Web? I understand the desire for people to protect someone they love from getting hurt but, in the long run, such heavy-handed tactics will probably fail. The Internet is just too irrepressible.

In an old discussion of online ethics, Simon Waldman notes:

“I find the views expressed on many organizations’ sites repellent. But one of the greatest achievements of the Internet has been to create the greatest gallery of human opinion in history, and that is something we should marvel at, rather than shake our heads in dismay.”

Would the people of Thailand deny their king access to such a place?

BTW: Sawatdee-krap, Mr. Surachai

Update:

Is Lying to the Public OK?

In a recent post, Lane Wallace discusses the pros and cons of a proposed amendment to Canada’s Broadcasting Act of 1986 which would allow broadcasters more leeway to broadcast false or misleading news. As you might imagine, that has generated some controversy, with free speech advocates saying the current law is too restrictive while others were concerned that the modifications would lead to a more toxic (and perhaps less accurate) American-style news environment.

Basically, the issue boiled down to whether or not people felt that TV and radio broadcasters had the same right to free speech as individuals. Because of their access to the public airwaves and the incredible power associated with holding a broadcast license, the CRTC (the equivalent to America’s FCC) determined that licence holders must be held to different standards and withdrew the proposed amendment,

Ms. Wallace wonders why we don’t have something similar in the U.S.:

Is it unacceptable censorship to require someone to be basically honest in what they broadcast as “news” — and which we are more likely to accept as truth, because it comes from a serious and authoritative-sounding news anchor?

Adding:

We prohibit people from lying in court, because the consequences of those lies are serious. That’s a form of censorship of free speech, but one we accept quite willingly. And while the consequences of what we hear on television and radio are not as instantly severe as in a court case, one could argue that the damage widely-disseminated false information does to the goal of a well-informed public and a working, thriving democracy is significant, as well.

One could counter that it is up to the individual to pick and choose their own news sources (something I have also discussed in the past) but she points out that:

In theory, we could all fact-check everything we hear on the TV or radio, of course. But few people have the time to do that, even if they had the contacts or resources.

Regulating U.S. broadcasters in such a manner would no doubt raise cries against  the “nanny state” from many circles but I suspect that these same folks are the ones that have raised obfuscation to a high art. In the long run, failing to hold news agencies responsible for their content does more harm to our society than good.

Propaganda vs. Disinformation vs. News

In his book Warriors of Disinformation, Alvin Snyder notes that, back in the 1980s, new developments in broadcast technology were making it difficult for Soviet and Warsaw Pact authorities to control the flow of information into their respective countries:

“The increasing accessibility of images, ideas, and information, facilitated by the rapid development and deployment of communications technology, presented closed societies with a Fustian bargain. On one hand, the new technology provided totalitarian governments with a new tool with which to control and manipulate public opinion, on an unprecedented scale. On the other, using the new technology meant those same governments would be subjected to greater domestic and international scrutiny, reducing their ability to control what their citizens saw and heard.”

These leaders knew that information coming from the outside world would clash with their own version of events and they were worried that the government-controlled media would lose its credibility if citizens were allowed to see and hear other opinions. They made it illegal to own things like satellite dishes and they jammed communication signals coming from across their borders. Meanwhile, the U.S. Government expanded its programs through the U.S. Information Agency’s Voice of America (VOA) and WORLDNET Television network and built ever more powerful radio and TV stations to broadcast American propaganda throughout the region. It was an arms race of sorts, only with the deployment of lies and exaggerated versions of the truth instead of tanks and missiles.

Although the Cold War is over, propaganda – officially defined as ideas, facts, or allegations spread deliberately to further one’s cause or to damage an opposing cause – is still big business. BP tripled its advertising budget during this summer’s oil spill in the Gulf of Mexico in an attempt to minimize the damage to its reputation. Estimated spending on advertising during the U.S. mid-terms topped $3 Billion as politicians fought to determine the outcome of the election. Lobbyists, non-profits, think tanks, and grassroots organizations all try to use propaganda to influence public opinion on any manner of topics — from tavern legislation to global warming.

In fact, propaganda is so pervasive in today’s culture that most people now assume that there is some level of bias behind almost every statement … even the straight presentation of facts. This loss of credibilty has been particularly damaging to traditional sources of information because they must now compete for attention with alternative media sources that are not bound by journalistic standards. In a world where accuracy of reporting must be balanced with speed of dissemination, the introduction of the Internet and modern social networks makes it much easier for false statements to spread unchecked.

The University of Wisconsin’s Center for Journalism Ethics sums up the issue facing journalists as follows:

“One of the greatest benefits of online journalism is its ability to reach millions of people almost instantaneously. But the pressure to keep news current — online within minutes of an event’s occurrence — can jeopardize the accurate reportingof even the most ethically-conscious journalist. Furthermore, the proliferation of news outlets — bloggers by the millions, of course, but also cable television, satellite television, web sites, and web broadcasts — has resulted in a multi-media race to get ‘the’ story 24 hours a day. As the pace intensifies, so does the pressure to cut corners.”

A prime example of this occurred recently when an unsubstantiated rumor about the cost of President Obama’s trip to Asia took hold in the blogosphere and then migrated to the talk show circuit and, finally, politicians eager to score some points with their constituents. It took an unusual stand by CNN’s Anderson Cooper to actually trace the rumor back to its source and call the whole charade into question. This neglect of basic fact-checking by pundits and major media outlets skirts dangerously close to disinformation — the deliberate spread of false information in order to influence public opinion or obscure the truth.

Thomas Freidman responded:

When widely followed public figures feel free to say anything, without any fact-checking, we have a problem. It becomes impossible for a democracy to think intelligently about big issues — deficit reduction, health care, taxes, energy/climate — let alone act on them. Facts, opinions and fabrications just blend together. But the carnival barkers that so dominate our public debate today are not going away — and neither is the Internet. All you can hope is that more people will do what Cooper did — so when the next crazy lie races around the world, people’s first instinct will be to doubt it, not repeat it.

I think it’s refreshing that Freidman places the onus of determining the validity of these rumors on the people receiving the information. Think of how futile it would be for any authority to try and block these sources. It would be similar to the Soviet government’s attempts to insulate its citizens during the Cold War. The challenge for our society than becomes: how do people prepare themselves to handle this influx of facts, opinions and outright fabrications?

The American Historical Association has published a great series of G.I. pamphlets that were originally prepared under the direction of the Army’s Division of Information and Education to help increase the effectiveness of American soldiers during and after World War II. One pamphlet deals directly with propaganda and tackles the concepts of “good” and “bad” propaganda as well as the difference between propaganda and news. This document also lays out the division of responsibility for interpreting information:

“The journalist of today has a responsibility to report facts as accurately, objectively, and disinterestedly as is humanly possible. The newspaperman who respects himself and his work — the average newspaperman — accepts this responsibility. The honest, self-disciplined, well-trained reporter seeks to be a propagandist for nothing but the truth.

“Of course propaganda does bet into the press. Sometimes it is presented in the guise of impartial fact because the newspaperman is not sufficiently trained — or smart enough — to recognize it for what it is. Sometimes the newspaper is a conscious propagandist — in news and headlines both. And sometimes propaganda is so obviously news, and so obviously a matter of importance to the newspaper’s readers, that the paper presents it knowing that the readers themselves will recognize it for what it is and evaluate it for themselves.

“All this imposes a responsibility upon the newspaper reader, and it is with him that the responsibility of judgment ultimately should and does lie. The good newspaperman does his best to confirm the news, to weed out propaganda that isn’t news, and to present whatever propaganda the citizenry ought to know about. Having done that, he leaves it up to the reader for evaluation and criticism. He knows that the critical reader — one decently supplied with facts and having some knowledge of propaganda methods and purposes — can do his own job of separating the wheat from the straw, the important from the unimportant. That is the citizen’s responsibility and his privilege in a democratic society.”

In other words, you are your own filter and you need to develop the skills and skeptical attitude that can help you cut through the BS. The GI pamphlet suggests a few simple questions that the “thinking citizen” can ask themselves when they see or hear a piece of information:

  1. Is it really propaganda? Is some individual or group consciously trying to influence opinion and action? Who? For what purpose?
  2. Is it true? Does a comparison of independent reports show that the facts are accurate? Does such a comparison show that the suggestions made are soundly based?
  3. Which fact or set of facts in a piece of promotion are really important and relevant? Which are irrelevant?
  4. If some individual or group is trying to influence opinion and action, is the effort selfish or is it unselfish? Will action resulting from the propaganda benefit the individual or group responsible for it? Or will it benefit those who act upon the suggestion given in the propaganda? Or will it benefit both?
  5. What is likely to be the effect of the action or of the opinion that the propaganda is trying to set in motion?

They also added:

All these points boil down to some very simple questions: What is the source of the propaganda? What is its authority? What purposes prompted it? Whom will it benefit? What does it really say?

This is timeless advice … maybe we should develop an advertising campaign to support the development of this skill.

Further Reading:

Your Opinions are Useless

Almost every time I read a particularly interesting article or blog post, I get sucked into the accompanying comments section hoping to find a stimulating conversation on the topic at hand. Almost every time I am disappointed. Does anybody ever find these threads useful? I know it’s silly to expect great discourse with random strangers but does it always have to devolve into such a wasteland of trolls, flamewars and embedded advertising? Is it even reasonable to expect a civilized dialog within such a framework?

I know, this is old news for anyone who lives and breathes social media but for less savvy folks trying to have a serious discussion online it can be a big barrier to participation. If I know that every conversation is going to end with obscenities‎ and insults, why join in? Many sites respond to the issue by introducing some form of moderation or requiring people to register before they can post anything. This reduces the level of anonymity and theoretically boosts the level of responsibility that people feel for their own comments.

However, some people feel that these approaches put an unnecessary muzzle on commentators. Matt Zoller Seitz over at Salon defends anonymous comments as a way to better understand our society:

“… for all the downsides of comments-thread anonymity, there’s a major upside: It shows us the American id in all its snaggletoothed, pustulent glory, with a transparency that didn’t exist before the Internet. And in its rather twisted way, that’s a public service.”

He argues:

“It’s impossible for anyone who reads unmoderated comments threads on large websites to argue that racism, sexism or anti-Semitism are no longer problems in America, or that the educational system is not as bad as people say or that deep down most people are good at heart.”

While I respect his reasoning, I don’t think that all sites should follow these guidelines. Surely, we as a society already know that there are plenty of blowhards out there who just want to insult people and stifle conversation. Do we really need to be reminded of that every time we watch a YouTube video or check the weather? Which is more of a problem … having someone self-censor themselves because their names are associated with what they write or having someone take themselves out of the conversation entirely because they are being harassed or annoyed?

Update:

Who’s Your Filter?

One of the ways in which people deal with information overload is to use a filter to screen out unwanted information and eliminate the amount of time spent on non-essential topics. Filters can take many forms but one of the most common is simply the “trusted source” — a person or an organization that has passed some internal test for trustworthiness. Everyone takes advantage of these personal filters because they provide shortcuts for many of our day-to-day activities.

Sore throat? Ask your sister the nurse. Problem with your PC?  Call up your buddy the software engineer. Car on the fritz? Invite your gearhead uncle over for a beer and friendly consultation.

People simply do not have the time to become an expert on every single subject and they also don’t have the time to vet every single source. They need a way to cut through all the BS and get information that allows them to take action. They find someone they trust who seems to know what they’re talking about and — boom — they’ve got their filter.

The problem with any filter is that there is always an element of bias involved. Your uncle may be great with cars but maybe he’s also a Ford guy who can’t stand GM and thinks BMWs are over-priced toys. No matter how well-meaning he may be, his opinions will color his advice — which could lead you to a different decision than if you were an expert auto mechanic and doing the research yourself.

This is the main complaint that people on the right side of the political spectrum have with mainstream media. They feel that the types of events that are presented and the way these events are covered reflect the liberal bias of the reporters and commentators telling the stories. The standards of journalism are supposed to prevent this kind of bias by requiring journalists to double-check claims, verify the reliability of sources and issue corrections if necessary. By casting doubt on the ability of the mainstream media to follow its own code of ethics, the right has reduced the number of people who feel comfortable using these news outlets as trusted sources of information. This has opened up the doors for a whole new slate of  alternative news filters. Of course, these organizations and their sources have their own biases and — in the case of bloggers — are not even required to follow standard journalistic practices of impartiality.

A prime example of the issue of source bias was on display during the controversy surrounding Shirley Sherrod and the speech she made to the NAACP that led to her firing from the USDA. Blogger Andrew Breitbart released an edited video clip of the speech that appeared to implicate Ms. Sherrod as a racist. After a firestorm of criticism, she lost her job … and then some bright bulb actually watched the whole video. Sherrod was exonerated and the White House, FOX news, Bill O’Reilly and others who had relied on Mr. Breitbart as their filter were forced to apologize.

Mr. Breitbart himself has not issued an apology and, as a political activist, is probably not required to do so. In an interesting turn, right-wing commentator Ann Coulter places the blame squarely on Mr. Breitbart’s own trusted filter:

“I think Breitbart ought to reveal his source, because he was set up. This was a fraud. The person who sent the edited tape has to know what the full speech said, and whomever sent only that segment to Andrew Breitbart is the one who should apologize to Shirley Sherrod.”

Certainly someone along the chain of information exchanges made the crucial edit that altered the tenor of her speech. Who is ultimately responsible, then, for errors that creep into these kinds of stories? Unfortunately, with the erosion of trust in the journalistic profession and the explosion of non-traditional news sources, the consumers of the information are themselves responsible. In a sense, we are all journalists now.

Update:

  • August 8, 2010 – There is an interesting article that touches on the difference between activists and journalists in the context of the WikiLeaks scandal here: http://reason.com/archives/2010/08/06/julian-assange-wikijournalist
  • November 2, 2012 – A good review of the movie “Hating Breitbart” discussing some of the changes going on in the media (link: http://reason.com/archives/2012/11/02/hating-breitbarts-post-partisan-message). I liked the author’s quote: “Internet-empowered journalism isn’t a Cosmic Cube-style weapon that will once and for all lay waste to all bullshit artists residing in low and high places. No, the new media that Breitbart helped operationalize are more like an over-the-counter Super Soldier Serum that helps us all fight for truth and justice as we define those terms.” The only problem I see with this metaphor is that being a superhero is a 24/7 job. Are we all going to be ready when the bat signal appears in the sky? Or are some of us going to sleep in?