The Bishop and the Butterfly: Murder, Politics, and the End of the Jazz Age

    How I Stopped Worrying and Learned to Love the Matrix

    By PeraclesPlease on Fri, 07/05/2013 - 2:07am |

    [copied in from a comment for further discussion re: the NSA]

    We can skip the word "evil" and just take examples of what citizens might want to know.

    While apparently the post office has been scanning envelopes a long time, the use of OCR means they have a huge searchable database of From, To and Date. The recent improvements in data analytics & Big Data means it's trivial to compile that into categories - especially if you link it to Facebook and collected phone metadata - who your friends are, where you work, where you bank, who you call, what organizations you're involved with. Depending on who's sending you mail, they can figure out if you're having credit problems, health issues, seeing a psychiatrist, getting a divorce.

    Oh, those license plate scans? One time might not say much - but aggregate over time, and we know which way you go to work, any clubs you frequent, where you go for lunch, what sports you play, where you drove on vacation last year and when you came back. Those Thursday 1500-1600 drives to a hotel? Looks like might be an affair - yep, we have that other license plate that's going there at those times too - let's see who that is... Uh-oh, she was just at an abortion clinic. Flag! Now maybe that's old school, because they also have phone companies turning over location records for your mobile phone - we can be pretty sure it was you driving that car, not your spouse.

    Of course they can grab all your banking & credit card information, and they have your tax info and employer ID. Emails would be nice - let's just get all the email headers going through Google, Microsoft, Apple, Comcast... Nice - we have all those identities, can sync it up with what you buy on Amazon, who you friend on Facebook, and with a bit more cooperation from your ISP (or just the big data taps we have at all the internet exchange points), we can tie this to what Web sites you go to, any anonymous posts you might have made on a blog, and whether you use encryption sometimes, which might give us the idea you have something to hide, so we can search further. Of course we can figure out the kind of porn you or your kids like and classify your political leanings. Hey, little Jimmy's been chatting on one of those grow-light and mushroom kit sites - guess we'll be talking to him downtown soon.

    Oh, now that everyone has their photos on-line in some form, this isn't faceless data gathering anymore - we've got you and family and friends all tagged and collated with appropriate metadata. Plus we can coordinate those street, bank & ATM cams with your cellphone, and do a bit of visual analytics if we need a bit of confirmation.

    How long can we keep all this information? Theoretically there are a few limits, but in practice, forever. Which is good, because there may be new methods we come up with in 6 months or 2 years that allow us to squeeze more info out. After all software and processing and visualization tools keep getting better and better.

    Oh, did I mention? - it's all legal! That's right, the NSA isn't doing anything wrong, Congress approved, we haven't even started talking FISA courts yet, so let's stop talking about this. Happy 5th of July, the Day After.

    [BTW, if you ever misplace your keys, you know who to call - it's not intrusion - it's a safety net!]

    Comments

    [for background data from another thread]

    I'm less talking about workers tracking than I am about technology tracking and spitting out info and indictments and apprehension orders at light speed. Workers are deathly slow compared to a speech-to-text algorithm searching for a set of dangerous keywords in different languages and then triggering a content analysis to see if the speech rises above a certain level of threat.

    In terms of network speeds, disk speeds, cloud computing, more power-efficient processors, the shift to GPU & co-processor, improved analytics, etc., it's just a matter of (short) time. Below is the push to exascale by end of decade. That's an exponential scale on the left, and as long as we exclude video and use character recognition for most images and docs, most people's yearly useful digital trails can probably be stuffed in 100Gigabytes max. Maybe 10 Gigabytes if it's just compressed location data, emails, blog posts, and digitized voice data.

    As for storage, the recent breakthroughs in 3D chips and HAMR storage (the latter better for write once, read many - like someone's history - means it's likely there can be a 60Terabyte disk in 3 years which would cover say 6000 people, so say 60,000 disks to handle everyone in the US if 100GB/person - 6000 disks if you low-ball with 10GB/person. That's getting few enough to put in a few racks managed by some fast search processor with next generation I/O tech. The newer SSD drivers are much much faster for read-write, but still too expensive for such a scaled application. Maybe by year 2020?

    From the last graph (all of these from The Register) you can see improvements in transfer time of data around the world just from one vendor's solution.

    These are just some of the technologies pulling together to make what we've got written in this legislation (and the absurd ways the executive branch is interpreting it - even Sensebrenner is outraged, ironically or sadly) very scary indeed much faster than many people would assume. (no, your home computer isn't improving that quickly)

    I'm still waiting for the sewage monitors that will tell who's home when and whether they're awake (though the thermal pattern you give off from your house is technically not protected either, so may be a simpler method to differentiate adults from kids and maybe use different volume of methane emissions to tell who's who in Beaver's house. By the way, all that falls under machine-2-machine or smarthomes, the technology that's been supposed to make us all happier in our residential nirvana, but like many a tech, can and likely will be used against us. Actually, all of this makes the mistake of assuming the NSA doesn't have early releases of all this tech paying above-commodity prices, so that this could be their actual capability now. (they had optical disks long before CDs became commoditized). N-Joy

     

    [oops - from 2 years ago, IBM was already doing something like this, but the scale I'm talking about is much more capacity for fewer devices and less energy consumption:

    A data repository almost 10 times bigger than any made before is being built by researchers at IBM’s Almaden, California, research lab. The 120 petabyte “drive”—that’s 120 million gigabytes—is made up of 200,000 conventional hard disk drives working together. The giant data container is expected to store around one trillion files and should provide the space needed to allow more powerful simulations of complex systems, like those used to model weather and climate.

    A 120 petabyte drive could hold 24 billion typical five-megabyte MP3 files or comfortably swallow 60 copies of the biggest backup of the Web, the 150 billion pages that make up the Internet Archive’s WayBack Machine.

    by PeraclesPlease 7/5/2013 - 8:10 am (re: ocean-kat)


    by PeraclesPlease on Fri, 07/05/2013 - 8:16am

    There are states openly taking measures to suppress votes.These efforts target minorities and the poor. States are making it more difficult for women to choose what they want to do with their bodies.by closing reproductive health clicks, poor women are losing access to routine health care provided at these facilities.

    Congress is making sure that fewer people can get educated by raising interest rates on student loans, Schools are being closed and teachers laid off. This is being done under the cover of fiscal responsibility. 

    In an era of urban food deserts and stagnant job growth, the government wants to decrease food stamps. This is supported by Christianists in Congress who quote the Biblical verse "the poor will always be with you." They omit the following verse and the context of the words that command Christian to go out of their way to care for the poor. 

    North Carolina holds a surprise vote to attack women.Texas shuts down debate during it attack on women. The Governor of Iowa becomes the sole decider on what women can do with their bodies. Alabama, Texas and North Carolina rush to suppress votes. Congress votes to let people starve.

    North Carolina' politics have been purchased by a wealthy man who now serves as state Treasurer.

    There are a variety of evils that are happening openly and not behind closed doors.Those waiting for you waiting for government cyber-attack, have missed the overt attacks.

    Amazon and Audible send emails telling me why a new movie or book should be of interest to me based on past purchase. They are very efficient in hitting their target. NSA missed the Boston Bombers. NSA missed that Snowden had download classified information. They also missed Manning.

    My point is that that government has already begun its attack on the populace. There is no real need for cyber-attacks because the population votes in people who decrease their rights willingly. Michelle Bachmann, Louie Gomert, Ted Cruz, Paul Ryan and Marco Rubio are all elected officials. 

    The forces of evil are using gerrymandering and a willing voting public to gain control over our lives. Devoting a large effort to metadata is a waste of resources.


    by rmrd0000 on Fri, 07/05/2013 - 8:34am

    Moderator: please delete comment above - it's way off-topic, hugely verbose, just a regurgitation of author's comments numerous other places on this blog, and distracts from any discussion I was trying to create.

    Thanks, PP


    by PeraclesPlease on Fri, 07/05/2013 - 8:40am

    Verbosity, irrelevance, and repetition don't fall under the ToS. But I encourage folks to show each other courtesy by trying to avoid hijacking threads.

    PS Please let's not have a discussion about who hijacked whose thread when. We've all done it from time to time. I'm just putting this out as a general request to try to be conscious of.


    by Michael Wolraich on Fri, 07/05/2013 - 11:05am

    No attempt to hijack. It seemed that corporations represent a threat by collecting data just as the government. Given that government goes to corporations to get data, the two seem intertwined. There are direct threats occurring in legislatures that have nothing to do with cyberspace. 

    There is theoretical abuse from cyberspace and actual abuse by legislators in various states.


    by rmrd0000 on Fri, 07/05/2013 - 11:19am

    The question is not whether you can tie your comment in some way to the blog post but whether your comment honestly attempts to address the subject of the post. Try to imagine that you wrote the post, and imagine how you would feel if someone else wrote the comment.

    I'm going to stop here because any further discussion would really hijack this thread. To reiterate, there's no ToS warning, and my request is not specifically targeted at you. Just a gentle reminder to all to try to be mindful of courtesy.


    by Michael Wolraich on Fri, 07/05/2013 - 11:41am

    We carry phones with GPS that can locate us when we are lost or ill. We have cars with On-Star. We walk in neighborhoods with video surveillance that aided in identifying the Boston Bombing suspects. We tell every minute of our lives on Twitter or Facebook. We post pictures on Instagram. We communicate our political views on public blogs. Corporations keep track of our calls, emails and purchases.

    The government and most corporations view our data as public. I have said many times that the public could be convinced to vote for ongoing electronic surveillance to feel safe. The courts will have to decide if the AP email capture was legal or if reporters can be singled out for surveillance because of reported leaked national security issues.

    We will continue to discuss political issues on a blog site open to anyone in the world.


    by rmrd0000 on Fri, 07/05/2013 - 9:09am

    This is again all irrelevant to this diary - I didn't go to the US government site to search, I didn't sign up for an account with the FBI, I didn't get terms of use from the NSA, I didn't agree to the post office scanning my envelope to tie to my account, and I'm not talking about emergency rescue procedures.

    Moderator - please delete this one as well.


    by PeraclesPlease on Fri, 07/05/2013 - 10:58am

    If I may be blunt, so what? Cataloging all the ways the government collects information does not show me why I should care.

    Maybe it's because I live in NYC where everyone is on top of each other, but I don't see why I should care that some civil servant has access to my facebook photos and records of doctor visits. The only people I worry about knowing things are people that I know. In that sense, I find the notion of small-town living, where everyone gets into everyone's business, far less private and more intrusive than an anonymous world where some faceless stranger has access to my data.


    by Michael Wolraich on Fri, 07/05/2013 - 11:13am

    Thanks for that. I expect that an email attacking a co- worker could land me in hot water even if done on a computer outside of the workplace.it would be tracking done by my job, not the government that would cause the problem.

    By capturing so much data, it makes it harder for government to focus on little old me. The large volume capture may be why they missed real threats,


    by rmrd0000 on Fri, 07/05/2013 - 11:25am

    Intent was not to show why should care.

    Intent was to show how easy it is to gather and cross-correlate all this information, and that it really isn't that hard to scale it to cover everyone - no longer just a 1980's Star Wars pipe dream.

    You're welcome to not care about it, though I hope you can acknowledge that other people do.

    Misuse of datamining like this can facilitate identity theft, blackmail, public embarassment & ridicule, false accusations (recent Ricin arrests had close but mistaken identity), financial theft or hardship, industrial / IP espionage, intimidation of minorities, shutdown & harassment of protesters, and probably some other significant abuses.

    Look at how well the military & BAH protected that data - a more criminal Snowden could take it out the back to the highest bidder or multiple bidders.

    Look at how well the FBI was able to hassle Aaron Swartz based on piling up charges - they only need a tiny excuse to make anyone's life hell. Look at the way some have harassed abortion clinics - how much consolidated information are we going to allow to how many million "trusted professionals" to use? When the Texas Democrats crossed state line to avoid a gerry mander vote, Tom "the Hammer" DeLay tried to use federal systems to bring them back - would be much easier now.

    I don't live in a small town, but this doesn't have to be either/or - you can have the small town busy bodies PLUS the faceless stranger accessing your data. Or worse, if that small town busy body has a clearance and some kind of crap excuse to go digging through your life. And note, in that "anonymous world" the faceless stranger is anonymous - not you.

    So I'm concerned. You're welcome to be blasé.

    And to rmrd0000's "By capturing so much data, it makes it harder for government to focus on little old me."  Not necessarily.

    If I tap your phone, I just have conversation on tape, maybe 2 phone numbers. I have to listen to it for same time as you spoke to really get info out of it (speech-to-text still only goes so far, though it can generally search for some loaded key words)

    If I grab all your metadata, I know when you usually call, who you call, can check all those people for suspicious on-line activity and hone in on certain ones, may decide to eliminate females or choose only blacks, may decided to focus only on those that go to liberal Web sites or are registered Democrat or said blogged something negative about the president, pull up any recent bank transactions of any over $500, can do a background check on arrests for any in that circle, can see if any donations to 1 of 20 groups, can get records on what vehicles are registered to you & can print out a map where each were caught on traffic cams the last week... if the telcos are providing all the GPS info can do the same with everyone's mobile numbers, and print out everyone's photo from Facebook or other social media site.

    Why I could just check about 10 or 12 boxes on a screen and it would go and do that for me and print it all out while I go get coffee. So no, it makes it much simpler for the government to focus on little old you. And with Big Data analysis, more and more obscure but helpful (to them) ways of using that data in aggregate will come up.


    by PeraclesPlease on Fri, 07/05/2013 - 12:11pm

    Instead of grabbing metadata on Progressives, one could just look at publicly provided voting patterns with red and blue colors like those posted in the newspapers after elections. Based on that data, I could gerrymander my party into Congressional seats that could only go to my party. It wouldn't matter what contributions where made to the opposition party, I've got the election locked.  Metadata seems labor intensive. I'm not saying that it would not or is not used by political parties, but it probably doesn't differ from lower tech data.


    by rmrd0000 on Fri, 07/05/2013 - 12:22pm

    I can see some danger from blackmail, overzealous prosecution, and other misuse of private data, but the scale of that threat seems limited, and I think it would better addressed by preventing the abuse of the data rather than the collection of the data. For example, arming the police facilitates police brutality, but we address it by trying to set up legal protection and punish bad cops, not by disarming them all.

    For the sake of argument, let's say the we effectively minimize or eliminate the abuse of information. It's my sense that most people still object to their data being collected--that it's inherently bad. And when you catalog all the ways information can be collected, you're banking on that reaction.

    So I think it's fair to ask, so what? Should we treat the desire for data privacy as an important right to be carefully protected, or should we regard it as a preference that some people have and that the government can ignore when its expedient to do so?

    I think that's the fundamental question, but I've yet to see anyone address it head on.


    by Michael Wolraich on Fri, 07/05/2013 - 12:31pm

    Don't think it's fair to drive by that thought so quickly.

    Many people object to how their data is being used, not necessarily collected.

    If I have a sense that the FISA court isn't a rubber stamp for every spook sniffing around, I have to think a bit harder about the tradeoff between a potential 9/11 and my personal info.

    But if the NSA/FBI/administration tells Congress and the EFF to fuck off anytime they ask about a program's operation, I'm going to side more with privacy.

    My point with detailing ways info can be correlated is to point out that that's part of the problem & the decision. How the government is correlating the information needs to be part of the permission given, and with new supercomputers and partitioning of data for more efficient processing and 20 different sources, it's no longer about "is a camera in a public place bad?" It's about a camera + video analytics + face recognition + bank records + Facebook + IP address and emails and blog history + ......" for the last 5 years rather than a single snapshot last night.

    "Should we treat the desire for data privacy as an important right to be carefully protected, or should we regard it as a preference that some people have and that the government can ignore when its expedient to do so?" - well after 230 years, I'd hope that we'd have some better sense of the intent of the framers on right to privacy and the despotism of governments whenever given the option of "expedient to do so".

    Of course data privacy is an important right to be carefully protected. If money is considered a form of "speech" in political terms, why isn't my speech and writings and details of my life that I haven't specifically divulged not part of my privacy? If I give $500 to Al Gore, you can't just take it and give it to Bush because I'd already given it away anyway. If I speak to you directly whether by phone or email or post, the details of that conversation should be private including whether it took place at all, unless one of the 2 participants divulges. We exhale, pee, sweat, bleed - I don't have a right to collect your fluids just because they left your body, but that's the way people seem to be interpreting the Constitution. Using thermal detectors on someone's house is equivalent in my eyes to using a sonic detector to listen through walls or wiretap on a phone. It should require a warrant for some good suspicion of breaking a law. If the only protected communication is where we hide in a closet, turn on a noisy fan and whisper to each other hoping no one has wiretapped the closet, we're fucked.

    And note - I don't trust people whether they're Americans or Chinese or whoever. Federal agents took out native americans in the 1800's, attacked wobblies when they were considered the "enemy" in the 20's, destroyed actors who had the wrong belief in the 50's, infiltrated & harassed the Civil Rights movement in the 1960's, had no problem supporting death squads and training goons in Latin America in the 70's-80's, and can blithely send out remote controlled drones with no-see-um civilian collateral damage in this decade while force feeding abused indefinite detainees at Gitmo or just simply tasering a lot of people for no good reason just because a cop's in a position of authority with a new toy. These people are assholes. We may have to trust them sometimes because we're threatened by bigger assholes - that's the basic issue all along - but I'm not going to sign on to how they're all noble spirits just doing their job. Some are, some aren't, some are halfway. I'll stay paranoid, thanks.


    by PeraclesPlease on Fri, 07/05/2013 - 1:11pm

    Agreed that we should know what data is being collected and how it's being used--so that we, not the some NSA bureaucrat, can decide what limitations are appropriate.

    On privacy, it doesn't actually go back 230 years. The first formulation of a right to privacy appeared the Harvard Law Review in 1890. The primary author was a Boston attorney named Louis Brandeis--you may have heard of him.

    But his argument was a little different from the Fourth Amendment justifications that we often hear about surveillance. He argued that the right to life included the right to be "let alone." When he wrote of privacy violations, he referred to publication of private data, as in a newspaper. The Supreme Court did not establish a right to privacy until Griswald v. Connecticut in 1965, but that was about contraceptives--again the right to be let alone.

    Private speech is clearly protected by Fourth Amendment precedent, which is why the NSA has not been able to collect email or voice communications between citizens without a warrant (though there's a little fuzziness around the edges). But the legality of collecting metadata is less clear. The government cannot open your mail without a warrant, but they have been capturing snail mail metadata for years.

    In any case, my point is not to make a legal argument--for which I am not qualified--but to point out that we have vague and evolving notions of privacy that are continually challenged by new technology. Putting the legal argument aside, I ask more broadly, why is it important that the government be prevented from collecting information about citizens' behavior (as opposed to speech), and where should we draw the line?


    by Michael Wolraich on Fri, 07/05/2013 - 2:31pm

    For rmrd0000, let's try another little thought experiment. Say Trayvon Martin comes through the hedge on the way back from 7-11, and in the new NSA paradigm, our trusted law-and-order buddy George Zimmerman can access mobile metadata for an ongoing call near a given address and search 3-degrees of freedom for connections to the neighborhood.

    The system spits out: Good news - Trayvon's dad's girlfriend lives in the neighborhood. Bad news - 17-year-old Trayvon was just suspended from school for 10 days - sounds bad - and his on-screen Twitter tag is "No_Limit_Nigga" and prints out a few whacked photos. So which way is George going to tilt - "oh, he lives here, I'll just be on my way" or "hmmmm, wonder what the punk will be up to around here?"

    My guess is it'll tilt even harder against Trayvon, and the info available will only be used against him. If you ever call the cops to your house to report a problem, the first thing they do is look around your house, frequently looking for drugs or something wrong - just their normal procedure, even if you were trying to be helpful. With a massive data mining procedure focused on finding problems, well, it's going to find and highlight them.


    by PeraclesPlease on Fri, 07/05/2013 - 12:40pm

    Bwahahahaha

    And you want to delete things I post as off topic?

    First wanna be cop Zimmerman accesses a program you say is too secret and cannot be accessed by mere mortals and now Zimmerman is justified in a death........WOW

    Since we are out in left field, wouldn't Julian Assange approve of Zimmerman having direct access to government information?

     


    by rmrd0000 on Fri, 07/05/2013 - 1:17pm

    There are 5 million people in the US with security clearance - 1.25 million above the lowest "confidential" category. It's possible that even Neighborhood Watch volunteers could be given access to these systems in the future if it's to save the Homeland or stop drugs from killing our children or some other patriotic excuse.

    "and now Zimmerman is justified in a death" - no, I intimated that Zimmerman would lean towards the negative side of the information rather than the positive. A privacy advocate would say "oh, he belongs here and he hasn't done anything wrong so I'll leave". But our law enforcement doesn't work like that - suspicions lead to more suspicions.

    Sorry I took the trouble to give you a practical example based on the topic discussed. Never fails to disappoint.


    by PeraclesPlease on Fri, 07/05/2013 - 1:28pm

    I'm afraid I have to agree with PeraclesPlease on this one. If I might put words in his mouth, he's not suggesting that the actual Zimmerman would've had access to this or would be likely to get access to it anytime soon [edit to add: after reading his reply, I guess he is suggesting the latter as a possibility], just that if he had, it would've made his actions more likely (not more justified, but more likely) and not less likely.

    I'm not going around screaming that the sky is falling (and near as I can tell, neither is PeraclesPlease), but I am concerned about what the NSA (and other countries' spy agencies) are doing. Unfortunately, just as with the post office gathering metadata from mail before it was even called metadata, I doubt this program is going to stop just because I don't like it.


    by Verified Atheist on Fri, 07/05/2013 - 1:34pm

    Thanks. Someone has decided the police departments across the country need drones and a bunch of leftover Iraq military gear and munitions to function properly. If you'd predicted that 10 years ago people would have likely laughed at you. That's why I don't consider it far-fetched (but not a prediction) that a Neighborhood Watch guy might be deputized to some kind of NSA/FBI information access.


    by PeraclesPlease on Fri, 07/05/2013 - 1:48pm

    Fair enough. I'm not laughing at your non-predicted possibility. Crying that I can't definitively say you're wrong, perhaps, but not laughing.


    by Verified Atheist on Fri, 07/05/2013 - 1:52pm

    I'm not a lawyer, but if metadata is available on Martin, the prosecutors would ask for the metadata on Zimmerman. Since Zimmerman had called the police department repeatedly, lied about being introduced to the concept of Stand Your Ground and appropriate force, and has a psychological profile when he tried to become a police officer, I'm sure it would all come out in the wash.

    Still not paranoid

     


    by rmrd0000 on Fri, 07/05/2013 - 3:04pm

    You appear to be assuming that metadata analysis equals something approximating truth. In reality, using your court example (which you'll note is not the original situation that PP introduced), metadata analysis is circumstantial evidence, and weak circumstantial evidence at that, not even as reliable as hearsay. I encourage you to read what I wrote last month on this issue, where I use another example to show how metadata analysis can be abused in a court of law.

    As for it all coming out in the wash, two bad pieces of information do not cancel each other out. In fact, they can frequently exacerbate the problem.

    As for paranoia, as a white, college-educated, middle-aged male with no political power, I personally feel that I have little to fear, but it's not me that I'm worried for.


    by Verified Atheist on Fri, 07/05/2013 - 3:31pm

    If metadata was used as the rationale for the shooting, would it not follow that the parents would request data on the shooter?

     


    by rmrd0000 on Fri, 07/05/2013 - 3:36pm

    Sure, but their son would still be dead.


    by Verified Atheist on Fri, 07/05/2013 - 3:38pm

    For fuck's sake, stop already.


    by PeraclesPlease on Fri, 07/05/2013 - 3:41pm

    I'm just assuming that defense lawyers and prosecutors would interpret the data to the benefit of the client and the victim represented by the state. Under your scenario , Zimmerman has information from the state that played a role in his decision  to approach Martin and shoot. Are you suggesting that the command on the phone would now be to attempt to apprehend a dangerous person rather than wait for police?  Would not the presence of a bigger threat make it more likely for Zimmerman to be told to stand down?


    by rmrd0000 on Fri, 07/05/2013 - 3:45pm

    In the example, as I understand, the assumption is that this hypothetical Zimmerman-esque neighborhood patrol guard has direct access to metadata on Martin, and that this direct access leads him to an analysis that makes him think that Martin is more likely a "thug" than not. As this is all very hypothetical, we don't know what it would lead to, but it's not a stretch (once one assumes that Zimmerman-esque has this direct access) to think that it might be the straw that broke the camel's back, such that if Zimmerman-esque could be slightly less likely to jump to conclusions than Zimmerman was, but that the metadata analysis could made up for that difference. I'm not assuming that anything different happens — i.e., the police still tell Zimmerman-esque to stand down, and he still ignores their command.

    (I think that the example I gave on my post might be easier to believe happening right now.)


    by Verified Atheist on Fri, 07/05/2013 - 4:14pm

    I think this Zimmerman scenario is a far fetched example. I see a more likely example in the current struggle over voters rights. I could easily see a peaceful group of activists engaged in acts of civil disobedience against Texas's voter id laws being targeted. Especially if those peaceful acts of civil disobedience were very disruptive. I can easily see the FBI or even the Texas police getting access to information about senior members of the group and using it to quell dissent.


    by ocean-kat on Fri, 07/05/2013 - 5:11pm

    The GOP has the legislature. Democrats can't block anything, so metadata isn't needed. The Texas voter suppression protesters are safe from the need for metadata scrutiny.

    I agree that the Zimmerman example is a stretch. What is amazing is that the Tdorashev death by the FBI has been forgotten, but that's a topic for another day.


    by rmrd0000 on Fri, 07/05/2013 - 5:23pm

    Your argument is contradicted by the evidence of history. Historically many groups have been targeted even though they were so powerless there was no need to target them. The Merton Center and the Catholic Worker were both targeted as little as siw years ago. Neither are powerful nor likely to be successful so there was no need to target them. One does not need to be powerful or successful. One merely needs to be sufficiently disruptive to annoy those in power or sometimes to simply exist in opposition to the status quo. In fact its easier to use this data to quell dissent of small powerless groups than large well organized groups with popular support.

     


    by ocean-kat on Fri, 07/05/2013 - 7:10pm

    I really don't disagree. I noted before, Boston was focused on OWS and not the Boston Bombers. Not sure if you are referencing vandalism charges or the nuclear plant trespass. Metadata was not used in either case.


    by rmrd0000 on Fri, 07/05/2013 - 8:17pm

    I think reality strikes more fear in me than the hypothetical. It took pressure to actual get the case to the courtroom. An unarmed child was dead and the killer was up walking around. The situation you lay out with metadata in use is no different than what actually happened. Dead Black Kid, nothing to see here. Prtests would be required with or without your scenario. Martin metadata would have been met in court with Zimmerman metadata. You can't frighten me with the possible. The real is scary enough.


    by rmrd0000 on Fri, 07/05/2013 - 3:21pm

    Unfortunately, I fear that abuses of metadata analysis will not receive enough exposure for them to become "real" to most Americans. As I discussed last month, real horror stories of the misapplication of statistics do exist. I agree that the real tragedy of Trayvon Martin's death is sad enough, but that doesn't nullify the tragedies caused by the misapplication of statistics, or the potential tragedies caused by the abuse of metadata analysis, just as we can't say that it's OK that Paula Deen wanted a plantation-themed wedding with all [black] wait staff because Don Imus called some female athletes "nappy-headed hos" and yet is still on the air.


    by Verified Atheist on Fri, 07/05/2013 - 3:37pm

    Repeating myself, I see the possibility of legal solutions that address privacy rights. Congress and plebiscites were not very good at securing rights for Gays. 


    by rmrd0000 on Fri, 07/05/2013 - 3:50pm

    Repeating myself, why are you still here? I see the possibility of space travel to black holes, which has nothing to do with the diary. Whether they're gay black holes or are subservient to supernovas is also light years away from the point of discussion. Why don't you write a diary for a change, and I'll show up and throw irrelevant shit all over it? Except I usually try to discuss the points brought up in the diary because I don't like wandering all over the universe.


    by PeraclesPlease on Fri, 07/05/2013 - 4:01pm

    Everything that is being done by PRISM, BLARNEY and FISC may be Constitutionally sound. Congress may have to declare metadata private property before there are legal limits. If Congress has to take action, nothing will happen until there is a major turnover.


    by rmrd0000 on Fri, 07/05/2013 - 4:55pm

    In the long run, perhaps:

    But this long run is a misleading guide to current affairs. In the long run we are all dead. — John Maynard Keynes, A Tract on Monetary Reform

    I don't dispute your second sentence, but I don't see how it helps your point. To be sure, metadata analysis is here to stay, and it's not all bad. I just want to make sure that people are educated about how it can be deliberately abused and accidentally misused, to help hasten the day that legal solutions do address the privacy rights and other issues surrounding metadata analysis.


    by Verified Atheist on Fri, 07/05/2013 - 4:03pm

    People may choose security over privacy. Unfortunately courts take time.


    by rmrd0000 on Fri, 07/05/2013 - 4:21pm

    In your case, you're probably right. No one is going to track you. No one is going to track me either. But the government is going to track some people and history convinces me that there will always be some abuse.

    http://abcnews.go.com/News/Blotter/fbi-spied-peta-greenpeace-anti-war-ac...

    The FBI improperly targeted Greenpeace, People for the Ethical Treatment of Animals (PETA) and two antiwar groups in domestic terrorism investigations between 2001 and 2006, the Inspector General of the Department of Justice said in a report released today.

    At least two of the investigations resulted in innocent people being placed on the domestic terror watch list for years,

    In addition to Greenpeace and PETA, the FBI targeted the Thomas Merton Center, a Pittsburgh anti-war organization, the Catholic Worker, another anti-war group, and an individual Quaker peace activist, according to the report.

    All my life I've been reading reports of abuse like this, from major news sources not nut case conspiracy web sites, for the last 40 years. This last one from is ABC in 2006, that's not ancient history. Abuse like this has a chilling effect on first amendment activity. Some groups are large enough or powerful enough to fight on in the face of such abuse, like King's civil rights group. But smaller less powerful groups could easily curtail their activity and probably have. If you were a volunteer with the Merton Center and you discovered that one of the higher ups was placed on a no fly list how would that affect you? Some would surely decide to end their efforts.

    I start from the premise that this data is or will likely be abused. I'm guessing you disagree but I think its a reasonable assumption based on a history of abuse. My predilection is to limit their power and the data they can gather to the minimum necessary to do their job. Before I even consider the question, what's the harm, I want to know the answer to this question.

    Why do you need that data? I have yet to see a convincing example of a terrorist being stopped using this metadata.

    http://www.cnn.com/2013/06/17/opinion/bergen-nsa-spying/index.html?hpt=o...

    Testifying before Congress on Wednesday, Gen. Keith Alexander, director of the National Security Agency, asserted that his agency's massive acquisition of U.S. phone data and the contents of overseas Internet traffic that is provided by American tech companies has helped prevent "dozens of terrorist events."

    On Thursday, Sens. Ron Wyden and Mark Udall, Democrats who both serve on the Senate Select Committee on Intelligence and have access to the nation's most sensitive secrets, released a statement contradicting this assertion. "Gen. Alexander's testimony yesterday suggested that the NSA's bulk phone records collection program helped thwart 'dozens' of terrorist attacks, but all of the plots that he mentioned appear to have been identified using other collection methods," the two senators said.

     


    by ocean-kat on Fri, 07/05/2013 - 3:41pm

    Ever since the Snowden/NSA story broke, I have been trying to understand why everyone is suddenly freaking out.

    Except for Snowden, what's new? Okay, maybe not even him. Since you bring up The Matrix I wonder, does he fancy himself  Neo-Neo?

    Since he left HK I imagine him as a much different pop-culture figure

     

    Again I ask, what's new? The Matrix, 24, Mission Impossible, NCIS LA.  Every other movie or tv show is about good guys using NSA-like technologies to save us all from the bad guys. That's going to be a hard meme to dispel from the popular imagination. No, hard is not the right word.  More like, impossible.

    For me the problem with Prism and similar programs is not the collection of all that data nor is scanning it for possible threats. That is what sentries have done since the dawn of time.

    My concerns about the program(s) are the competency of its designers and operators, the secrecy in which it is shrouded,  and its cost:benefit.

    1. My chief concern is with the competency of the people who built and run it. I have not been at all impressed by the ones revealed so far: Booz Allen, Clapper, Alexander, and Snowden - especially Snowden. He may be a whiz bang hacker but from what I've read his spycraft comes straight out of graphic novels comic books.

    2. Why so much secrecy? Posting visible sentries is a deterrent in itself. In a way Snowden may have enhanced  PRISM's deterrent effectiveness by revealing its scope.

    I distrust people who distrust other people. Secretive people who think they know what is best but are not willing to test their ideas in open exchange.

    3. Why plan to keep so much data essentially forever when the vast majority of us can easily be classified as non-threatening. Sure the capability may be there but why bother? A sensible retention schedule would be more cost effective.

    I have had The Transparent Society: Will Technology Force Us To Choose Between Privacy And Freedom? by David Brin on my to-read list for a while. Probably should read it soon.


    by EmmaZahn on Fri, 07/05/2013 - 12:07pm

    "A sensible retention schedule would be more cost effective." - not necessarily - the long-term data trends may give something more valuable than a short-term window. Plus cops & such are always looking to see if anything can be traced 10 years back, 20 years back. And there are new paradigms and algorithms that get invented that can let you look at your data archive in a completely different way. The reality is, you never know what will be useful in 5 years. And the mining techniques are making it easier to say, "screw it, we'll just buy a faster machine and analyze everything".


    by PeraclesPlease on Fri, 07/05/2013 - 12:44pm

    What about the energy costs? My reading indicates that the Cloud is going to be an energy-sucking monster -- one reason all the major players are setting up data centers nearby. Even so, I doubt they intend to keep everything forever. Not even Google since they just pulled the plug on my Reader. But hey, I am sure they all will be glad to work out some sort of reciprocal agreement the PRISM to supply the data in exchange for access to it.  Would save them oodles in storage and energy costs, an indirect subsidy for them; a hidden cost for the rest of us.

    Also, just because we can keep everything forever doesn't mean we should. There's a word for people who do: 

     

     


    by EmmaZahn on Fri, 07/05/2013 - 1:10pm

    Energy is a huge issue - the energy cost to reach exascale means the tech has to be made much more efficient or it will take up all the energy produced in the country. But that's happening.

    In terms of disk, first the move from moving disks to solid state greatly cuts down on energy, though at a big cost difference between drives. But if you spin down a disk when not using it, and can sit there for 2 years without cost until you need it. (may need to be spun up now and then to keep it operational).

    Processing is getting more efficient such as moving to GPU's and co-processors and also how problems are approached, and both Intel & ARM have come out with significantly more efficient chips energy-wise the last few years (to make a mobile phone run more and more requires it - miniaturization & efficiency at the same time - no room for a big fan in that handset). New 3D chips will improve the power consumption - distance energy travels,number of processors on a chip.... - even more. Networking is also an energy cost, and even all those mobile base stations with transmitters suck up a huge amount of juice, though new LTE data networks are much better than original data nets. All on the roadmap for "stuff that has to be improved".


    by PeraclesPlease on Fri, 07/05/2013 - 1:21pm

    An anecdote:

    Late 70s, very busy day in the stock market, all ADP systems crashed leaving many, many trades unfulfilled and lots of irate customers to either placate or fulfill their orders from the broker's own books.

    A couple of weeks late an apology letter from ADP arrived explaining what happened and outlining plans to expand future computing capability.

    Funny thing is what happened was that its air conditioning failed, its system overheated and shut down at around 80 degrees (iirc) and yet no where did the letter mention what its plans were for future a/c capacity.

    Question: will more or less a/c be needed for the future systems you describe?

     


    by EmmaZahn on Fri, 07/05/2013 - 2:34pm

    If it's spinning disk - more. If it's SSD less. If it's HAMR disk then fewer disks so less. If it's new Intel Atom CPU or ARM or optimized for GPU probably less. If the analytics processing requirements are greater then more. If the building's in Arizona then more. If it's in Maine then less.


    by PeraclesPlease on Fri, 07/05/2013 - 3:35pm

    Although if the building is in Arizona, they can use solar power and batteries as their energy source.


    by Verified Atheist on Fri, 07/05/2013 - 3:39pm

    Solar power is an energy source, esp. in Arizona. Batteries are just a storage device with energy loss due to heat.


    by PeraclesPlease on Fri, 07/05/2013 - 3:45pm

    Right, that was a poorly constructed sentence on my part. I added the "and batteries" to make it clear that solar power can be reliably used in Arizona, even when it's night out or on the relatively rare occasions it rains.


    by Verified Atheist on Fri, 07/05/2013 - 4:06pm

    Ah, I thought you meant "super espionage data center with encoder ring - batteries included" ;-)


    by PeraclesPlease on Fri, 07/05/2013 - 4:13pm

    I rarely agree with PeraclesPlease so strongly, but Moore's law is with him on this one. Not only is it getting easier to store more, it continues to require less energy per byte and per flop. If you're not already familiar with it, I encourage you to consider the Internet Archive AKA the Wayback Machine. This work is funded by a 501(c)(3) non-profit with what I assume is a tiny fraction of the NSA's budget.


    by Verified Atheist on Fri, 07/05/2013 - 1:28pm

    I'm concerned Clapper is beaming my metadata into space where any alien race out there can use it to to kidnap me, make an army of NCD clones, and in a sinister plan use them to furtively subvert and destroy the earth! I watch the skies friend, and worry a lot.

    Seriously, what if no one at NSA or anywhere cared about you or your metadata? Your divorces or your doctor visits, or where you go with your cell phone? Except perhaps Google or Microsoft who are beaming you customized advertisements?

    Maybe, after you're gone, your brilliance and uniqueness will be discovered, and your perpetually preserved metadata will be as highly valued as the Dead Sea Scrolls? Would you deny future generations that gift?


    by NCD on Fri, 07/05/2013 - 2:54pm

    You don't need metadata to get around People who support women's rights. You gerrymander districts. Once you have control, you don't care what voters think, you merely set criteria that abortion clinics can't meet and close them down.If you want to block voters, you don't need metadata, you simply use your gerrymandered legislature to make sure voting booths in areas that don't vote for your party have serve 10 X  more people.

     


    by rmrd0000 on Fri, 07/05/2013 - 3:11pm

    Your silly first paragraph doesn't do anything to contribute to a meaningful discussion of the issues and only makes it seem like you have no idea how this data can be misused.

    As for your second and third paragraph, it's not my metadata I'm concerned about. (I won't presume to speak for PP on this.) In his Zimmerman example, PP painted a reasonable picture about how misuse of metadata can lead to confirmation bias. I painted a similar picture in what I wrote here. In addition to the metadata that they admit to capturing, there also the other data that they're not currently admitting to (but that you're evidently also assuming they're gathering, based on your second sentence of your second paragraph) which could be used (as an example — I'm not claiming this is actually happening) to apply pressure on elected officials to continue their support of programs designed to continue to make the rich richer.


    by Verified Atheist on Fri, 07/05/2013 - 3:23pm

    The picture you painted:

    Now, consider a hypothetical: data mining statistics come back suggesting that the odds that John Doe is guilty of X is 99.999%. John Doe is a minority with a juvenile record that includes crimes similar to X, and he happens to live within 20 miles of Alan Street and 1st Avenue, in New York City where the crime happened. Although 99.999% seems pretty ironclad, it means that there are 1-in-100,000 odds that a non-guilty person would be also implicated. How many people live within 20 miles of Alan Street and 1st Avenue? How many of them have juvenile records? How many of those records might include similar crimes? Do we even know if the person actually guilty of the crime has a juvenile record? (No, we don't. This is my hypothetical, after all, and it would seem weird to assume otherwise regardless.)

    So, who here now thinks John Doe is most likely guilty? Who here thinks he would be found guilty in a court of law?

    I think your picture is a crock. I have been in courts of law, and they don't admit unrelated to the crime metadata as evidence.

    I think your scenario, and the beaming metadata to aliens scenario, are about equally likely.

    And the real downer with the aliens, is there will be no going to court at all, no ACLU to hire lawyers to defend you. No Bill of Rights.  So have a nice day and watch the skies. 


    by NCD on Fri, 07/05/2013 - 5:35pm

    Of course it wouldn't be called metadata in the courts. They'd merely point out that John Doe is a minority with a juvenile record, who lives within 20 miles of the crime scene, and facts X, Y, and Z that they happened to glean from metadata. They'll just probably fail to point out that John Doe was drawn from a sample of 7 million (or whatever the number is), so the fact that someone happened to have these criteria would actually be expected. People who don't sufficiently understand statistics (and that's the vast majority of people) aren't going to appreciate these non-coincidental coincidences.

    Again, your alien quip suggests a soupçon of insecurity in your ability to defend your own position without exaggerating the other side's position.


    by Verified Atheist on Fri, 07/05/2013 - 7:27pm

    I am tempted by Wolraich's blasé response to all of this but there's more to it, I think, than just my privacy. It's about power and who gets to use it. I don't think the NSA will ever have much direct bearing on my life but I also do not like that it can act as it pleases, deny it and then use it's own denial to say I lack the standing to even sue over it. In short, I deeply dislike the NSA's assumption of authority, the same way I hate and resent that a cop in NYC can stop and search anyone for any reason. It is not, frankly, deferential enough.


    by Michael Maiello on Fri, 07/05/2013 - 3:51pm

    Very well said.


    by Verified Atheist on Fri, 07/05/2013 - 4:04pm

    Ok, I'll buy that. Great comment.


    by Michael Wolraich on Fri, 07/05/2013 - 4:13pm

      Some of us who voted for Obama in 2008 thought he was going to restore civil liberties. That didn't work out so well.


    by Aaron Carine on Fri, 07/05/2013 - 4:56pm

    I thought that when he voted for telecom immunity that those hopes were pretty dashed.


    by Michael Maiello on Fri, 07/05/2013 - 7:07pm

    Excellent, well-reasoned post, Peracles. The key thing to understand is that technological advances, not law or conscious decision-making, now set the limits. What is done is what CAN be done.

    You talk about everybody's total data being compressible to about 10Gb. With ultrafast connections, that can now take less than a second to upload or download. Yet we're still hearing arguments that it's too labor-intensive to keep tabs on everyone.

    I make phone calls to the States; I've received packages from the States. So there's an NSA e-file with my name on it, ready for when I step out of line, maybe get too critical of U.S. foreign policy on some obscure blog.

    We've all been pre-investigated for the future crimes we might commit. That used to be the definition of a police state. Perhaps I've said too much.


    by acanuck on Fri, 07/05/2013 - 8:53pm

    Actually, I've been sending all your comments to the NSA as well. Sorry dude, but it's my patriotic duty to treat every Canadian as a potential terrorist threat. The Mounties will be smashing down knocking politely on your door in about 10 seconds.


    by Michael Wolraich on Fri, 07/05/2013 - 11:16pm

    I think the Canadians already know. They have their own metadata program.


    by rmrd0000 on Fri, 07/05/2013 - 11:23pm

    You bastard! It's unforgivable you ratted me out, but thanks for the heads-up. If the Mounties are stopping by, I'll need a couple of hours to tidy up. Think they'd like lox and cream cheese hors-d'oeuvres? I'd offer some white wine, but that might insult them since they're on duty. I'd hate to get on their bad side; they have Tasers. Now that I think about it, they have guns too. You bastard!


    by acanuck on Sat, 07/06/2013 - 1:31am

    There are options to try to maintain privacy on the web. DuckDuckgo is a search engine that does not track activity. There are free email services that promote no snooping policies like Hushmail. You can erase your IP address by connecting to a virtual private network for a fee. There is a free VPN, Tor Project, but it is slow. Detecting who attempted to track you is possible with browser add-ons like Do Not Track Plus. Entrepreneurs are probably at work to enhance the ability to be somewhat anonymous on the Internet in the wake of the PRISM story.

    I can't guarantee that anonymous browsing and email might make you a bigger target of interest, especially if you communicate outside of the country.


    by rmrd0000 on Sat, 07/06/2013 - 12:26am

    Via Arts & Letters Daily, came across this book review @ City Journal, basically talking about "learning to love the matrix" without any irony intended, a bunch of positive futurist counterarguments to the Orwellian agita:

    Every Breath You Take
    The age of all-seeing, all-knowing information analytics is nearly upon us.
    By Mark Mills, 2 July 2013

    [....] Soon big-data analytics will cross a Rubicon: we won’t have to guess or approximate what’s going on with many activities, we will know. Until now, given the scale and complexities of commerce, industry, society, and life, you couldn’t measure everything; you approximated by statistical sampling and estimation. That era is almost over. We won’t have to, for example, estimate how many cars are on a road, we will count each and every one in real time as well as hundreds of related facts about each car. Ditto soon for such things as your heartbeat or blood glucose, and much more.

    The fascinating thing about the scale of massive data sets is that, as Asimov predicted, they can reveal trends, even behaviors, that tell us what will happen without the need to know the “why.” (That was the trope in the movie Minority Report, based on a 1956 story by another great sci-fi writer, Philip K. Dick.) With robust correlations, you don’t need a theory to predict; you just know. This is where Mayer-Schonberger and Cukier quite properly devote much of their attention: “Big data may not spell the ‘end of theory,’ but it does fundamentally transform the way we make sense of the world.” [....]

    But big data is about more than mining existing information. The depth of the revolution has yet to unfold. Soon, data concerning trillions of objects and activities will be available; self-powered and self-networked sensors will track everything from our blood sugar to how many steps we take. [.....]


    by artappraiser on Sat, 07/06/2013 - 12:42am

    Great find - there certainly is some upside to technology predictions - the technology singularity branch is one that goes there. One of the best-known, Kurzweil, who invented the synthesizer among other things, was just hired by Google to work on his vision. Like many such collaborations, hiring a "Dream Team" may not work out, and Kurzweil's own predictions may be a bit optimistic, but the pace of technological improvement is speeding up on a number of converging levels, so we do get implications of a breakthrough point sometime in the next 50-80 years speaking conservatively. If we can hold on to our political structures and humanity during this period, we should end up much better off.

    It may be that Big Data itself is only useful for 5-10% of the problems we present it - but the potential for breakthrough performance in those minority cases seems quite high. Normally when we take a bizarre left-field approach to something we've failed with it's quite hit-and-miss - like the years Charles Goodyear spent trying tons of materials to find a solution like rubber for tires. Big Data has a repeatable mass analysis aspect that can allow pretty easy tweaking and re-conceptualizing until some better observation comes out - as the author notes not always with a theoretical underpinning but useful all the same. It's extremely early for Big Data, and the combination of its use in human behavior & motion, genetics & nanotech, modeling the atmosphere or ocean, and a variety of other areas is pretty breathtaking. It might take us 50 years to get the processing power to really use this well, but it will come. Given we haven't killed ourselves off through some biological mutation experiment gone bad.


    by PeraclesPlease on Sat, 07/06/2013 - 1:35am

    Having worked in data mining, I take great exception with that second paragraph. We can reveal trends that tell us what the expected behavior of an individual is, and for large enough groups (n ≃ 100) those predictions might even be shockingly accurate, but for individual behavior, we're never going to get Minority Report accuracy.


    by Verified Atheist on Sat, 07/06/2013 - 9:21am

    Your exception is interesting because it goes to the heart of what analysis provides (or not) to those who do it.

    If a traffic control system can regulate and monitor every intersection, it does not need to concern itself with what happens in the Cul de Sacs.

    From the point of view of information exchange, the individual is an event, not some kind of being that requires understanding or accuracy to regulate. The limit to the Orwellian idea of surveillance is the preposterous idea that anybody cares enough to watch you all the time. That is why his book 1984 is more about self selection than the apparatus of power.


    by moat on Sat, 07/06/2013 - 6:14pm

    (I'm not sure I completely understand what you're saying, so I might be misinterpreting it. The response below is to what I think you are saying.)

    First of all, no one has to watch me (or the average citizen) all of the time. That's the job of the intelligent agents, who look out for such subversive behavior as promoting peace, advocating against tax cuts for the rich, etc. I'm not actually concerned for my own well-being, but for the chilling effects that can be caused by this monitoring. I consider this to be as damaging as the GOP's new voter ID laws, if not more so.

    Secondly, another concern are the higher profile targets, which can have their behavior similarly altered, and could possibly even find themselves framed for crimes they didn't commit.


    by Verified Atheist on Sat, 07/06/2013 - 6:25pm

    I share your concerns. I agree with the viewpoint you have put forth that what might not hurt me (or you) could hurt others because they are more at risk because they do stuff or have made enemies with people who do stuff.

    But the main thrust of my response was meant to address the matter of scale you were talking about when you said there was a limit to what was 'possible' in terms of analysis of data. Put another way, I could ask: how much does one need to know to have learned what prompted them to inquire?

    One consumer of this information may only care about what shoes I buy. Another may want to know who I vote for. Who wants to know?

    One thing that bothers me a lot is that there is no natural counter point to this market analysis. What keeps it in its place? What is the countervailing force?

     


    by moat on Sat, 07/06/2013 - 6:56pm

    Interesting enough*, the scale limit of what is possible reflects the micropredictions, not the macropredictions. A useful analogy is quantum mechanics: in the double slit experiment, one cannot predict whether a single photon will pass through one slit or the other, but one can predict how a collection of photons will behave. I'm not suggesting that we are quantum particles, but the result is very similar.

    I can't predict with certainty whether you will buy a hamburger or a salad, but I can say in a meaningful way that there's an 85% chance you buy a hamburger, a 10% chance you'll buy a salad, a 5% chance you'll buy both, and a 10% chance you'll buy neither. When I analyze a large group of customers, I can have a fairly high degree of confidence about how many hamburgers and how many salads I should be preparing in anticipation of your order, with very little going to waste.

    Again, by analogy, if I flip a single coin, I don't know if it'll be heads or tails, but if I flip the coin 100 times, I can say with 95% confidence that it will come up heads 40-60 times. If I flip the coin 10,000 times, I can say with 95% confidence that it will come up heads 4,900-5,100 times. I can say with 99.7% confidence that it will come up heads 4,850-5,150 times. (The variance for a fair coin flip of 10,000 times is np(1-p) where n=10,000 and p=0.5, so that np(1-p) = 2,500. The standard deviation is the square root of the variance, and the square root of 2,500 is 50. The binomial distribution is approximately normal for large enough n.)

    *Well, at least I find it interesting. YMMV.


    by Verified Atheist on Sat, 07/06/2013 - 7:24pm

    They told me when I signed up at dagblog I wouldn't have to do any math. (glares)

    Actually I find it interesting too, thanks. I always like to get a bit more information in areas I haven't studied much.


    by ocean-kat on Sat, 07/06/2013 - 7:41pm

    Interesting, for sure.

    One difference between flipping coins and betting upon what people will buy is that people are only buying what has been made.  It would be a mistake to consider the element of chance that is obvious in the former case be presumed to be having the same role as what is happening in the latter.

    The emergence of statistical certainty in both cases does not make them similar things.

     


    by moat on Sat, 07/06/2013 - 8:34pm

    One thing that bothers me a lot is that there is no natural counter point to this market analysis. What keeps it in its place? What is the countervailing force?

    Some thinkers suggest souveillance by engaged citizens as a countervailing force. For example, while Google Glass or plain ordinary phones can be used by authorities to record and track individuals, they can also be turned against them to record abuses of authority.

    Another suggestion I read was to open database access online to what authorities and marketers are recording about us, private as well as public but each of us could access only our own private data not that of others.

    Some people like David Brin have been thinking about this for a very long time. Mid 90s he wrote an essay about it that he expanded into a book and a website and a following*, if you are interested:

     
     
     
    Oh, Brave New World!
     
    *I'd guess some of them are those scary-looking ones from the Wikipedia article. ;)
     

    by EmmaZahn on Sat, 07/06/2013 - 8:12pm

    WTF is wrong with this text editor?

    What I see when I write is not what dagblog gets!

     


    by EmmaZahn on Sat, 07/06/2013 - 8:14pm

    NSA called - it parses better in their chosen format. What you see now is what NSA gets. Where's your patriotism?


    by PeraclesPlease on Sun, 07/07/2013 - 3:55am

    Interesting points of view. Requires mulling. I will now mull and hope to back to this later.


    by moat on Sat, 07/06/2013 - 8:38pm

    Wolfgang Schmidt was seated in Berlin’s 1,200-foot-high TV tower, one of the few remaining landmarks left from the former East Germany. Peering out over the city that lived in fear when the communist party ruled it, he pondered the magnitude of domestic spying in the United States under the Obama administration. A smile spread across his face.

    “You know, for us, this would have been a dream come true,” he said, recalling the days when he was a lieutenant colonel in the defunct communist country’s secret police, the Stasi. http://www.mcclatchydc.com/2013/06/26/195045/memories-of-stasi-color-ger...

     


    by A Guy Called LULU on Sat, 07/06/2013 - 1:05pm

    As with all oppressive regimes, the primary failure of the East German government was the fact that the assholes didn't elect the American Democratic party to oversee their police state.

    Being placed in the care of a Democratic administration is sufficient to render any and all repressive policy into something progressive and good ... or at least justifiable.


    by Lazy KGB (not verified) on Sun, 07/07/2013 - 12:07pm

    While apparently the post office has been scanning envelopes a long time, the use of OCR means they have a huge searchable database of From, To and Date.

    Have any documentation for this? I can't find any discussion or confirmation of it at all.

    Destor's prohibition on American chattel daring to discus thoughts of constitutionality aside, isn't it possible for congress to pass a law and, upon challenge, it be determined that the law - or an executive implementation of it - is unconstitutional and therefore illegal?

    Avoiding court challenges and political backlash are stated purposes for program secrecy. That is not typically an approach taken by folks secure in the legal footing of their actions. You can trivially declare issues of constitionality won't be determined in comment boards, and that's fine. But until these programs and policies are subjected to - and sustain - challenge in an honest American adversarial legal process on their operational merits through to the Supreme Court, they sure as shit can't just be trivially declared as legal acts.

    All of this stuff is simply unvetted under America's constitutionally recognized system of jurisprudence. Sitcking your fingers in your ears and going "LALALALALALALA I can't HEAR you!" doesn't change that simple fact.


    by Anonymous (not verified) on Sun, 07/07/2013 - 12:00pm

    Oh ... hey ... it's me, BTW.


    by Lazy KGB (not verified) on Sun, 07/07/2013 - 12:01pm

    Hi KGB

    Re: Have any documentation for this? I can't find any discussion or confirmation of it at all.

    Are you serious? It's allover the internet since this NYT article by Ron Nixon was published:

    http://www.nytimes.com/2013/07/04/us/monitoring-of-snail-mail.html

    Search for The Mail Isolation Control and Tracking program (different from separate mail covers ordered by law enforcement of days gone by):

    The Mail Isolation Control and Tracking program was created after the anthrax attacks in late 2001 that killed five people, including two postal workers. Highly secret, it seeped into public view last month when the F.B.I. cited it in its investigation of ricin-laced letters sent to President Obama and Mayor Michael R. Bloomberg. It enables the Postal Service to retrace the path of mail at the request of law enforcement. No one disputes that it is sweeping.


    by artappraiser on Sun, 07/07/2013 - 12:35pm

    Can everyone here at Dag agree that ALL US government intelligence gathering, whatever the agency,  have open and ongoing public scrutiny so we the citizens know what exactly what these people are doing and how they are doing it?

    How else can we ensure our constitutional rights are being protected?


    by NCD on Sun, 07/07/2013 - 12:44pm

    I guess I should post a larger-size snippet for those who don't have NYT access, with the critical description of the program included:

    Mr. Pickering was targeted by a longtime surveillance system called mail covers, a forerunner of a vastly more expansive effort, the Mail Isolation Control and Tracking program, in which Postal Service computers photograph the exterior of every piece of paper mail that is processed in the United States — about 160 billion pieces last year. It is not known how long the government saves the images.

    Together, the two programs show that postal mail is subject to the same kind of scrutiny that the National Security Agency has given to telephone calls and e-mail.

    The mail covers program, used to monitor Mr. Pickering, is more than a century old but is still considered a powerful tool. At the request of law enforcement officials, postal workers record information from the outside of letters and parcels before they are delivered. (Opening the mail would require a warrant.) The information is sent to the law enforcement agency that asked for it. Tens of thousands of pieces of mail each year undergo this scrutiny.

    he Mail Isolation Control and Tracking program was created after the anthrax attacks in late 2001 that killed five people, including two postal workers. Highly secret, it seeped into public view last month when the F.B.I. cited it in its investigation of ricin-laced letters sent to President Obama and Mayor Michael R. Bloomberg. It enables the Postal Service to retrace the path of mail at the request of law enforcement. No one disputes that it is sweeping.

    I would like to add that the fact that Tens of thousands of pieces of mail each year undergo this scrutiny of specially-ordered old-fashioned mail covers suggests that the usefulness of the search of the bigger metadata program must be limited. Why else would they still order them if it were the case that they could just do a quick search of the database to get the same result?

    Then you have the keystone cops example of the mail cover for Mr. Pickering, where the carrier exposed the mail cover by putting the message meant for the carrier in Mr. Pickering's box....


    by artappraiser on Sun, 07/07/2013 - 1:03pm

    We have a Constitutional Scholar as President - 2 jobs in one - we don't need that time-wasting review stuff anymore. It's legal if he says it's legal. In fact I think we have precedent on that comment, so it shouldn't be news to you.


    by PeraclesPlease on Sun, 07/07/2013 - 12:39pm

    Determining the constitutional basis for intelligence gathering, how, why and on whom, cannot be safely decided by a secret court or by secret bureaucracies, true?

    What they are doing, and how they are doing it, must be examined in public because, frankly, those doing it cannot be trusted to operate in secret.....?


    by NCD on Sun, 07/07/2013 - 12:51pm

    How can you judge the constitutionality of something if no one's allowed to see what it is except for a rubber stamp judge that John Roberts appointed? (hint: Roberts doesn't seem to be the greatest advocate for personal rights on the court - even Scalia was to his left & a dissenter to his last decision)


    by PeraclesPlease on Sun, 07/07/2013 - 3:19pm

    National intelligence operation should be open, public and transparent, because we can't trust Chief Justice John Roberts, Congress, the NSA directors or the President to protect our rights. Right?

    The only problem is, how on earth can effective intelligence gathering be done if it's all subject to public scrutiny - which would include scrutiny by the very people we're trying to stop from harming the nation or breaking it's laws?
     


    by NCD on Sun, 07/07/2013 - 3:56pm

    I didn't vote for NSA directors, and is he supposed to scrutinize himself? Congress is being denied access to review - see Wyden & Udall, others. John Roberts is a single justice - not the whole Supreme Court - and couldn't even manage to get through his confirmation without lying. So it's back to the Unitary Executive theory - which for all purposes might as well be some king.

    No, I'm not saying there should be no secret info - but there should be checks and balances as via the Constitution, and not a new class of courts for tribunals and FISA and everything else, cutting out Congress in the process. Even the gang of 8 in Congress that's supposed to get the really confidential stuff isn't getting it. The executive branch just thumbs its nose, and nothing happens to it.


    by PeraclesPlease on Sun, 07/07/2013 - 5:32pm

    Yeah, it's up to Congress, and Congress would rather play politics with Benghazi, the debt ceiling, abortion or the IRS. Or they would rather cut banking regulations so they get more contributions from the fat cats.

    Congress does not want to do the hard job of setting standards and ensuring they are enforced at the NSA. It's hard work, and if an underwear bomber slips through the metadata net, Congress might find they are blamed, not the President, the easy political target. So they do nothing.

    They view their job as winning the next election. That takes money, getting suckers charged up to vote for them, usually over some hot button issue like abortion, gays or taxes, issues rarely resolved with finality. Until voters demand a responsible Congress that is focused on moving the nation forward, with responsible governance, compromise and oversight, not obstruction, there will be talk and little else.


    by NCD on Sun, 07/07/2013 - 7:14pm