Software increasingly manages the world around us, in subtle ways that are often hard to see. Software helps fly our airplanes (in some cases, particularly military fighter aircraft, software is the only thing keeping them in the air). Software manages our cars (fuel/air mixture, among other things). Software manages our electrical grid. And, closer to home for me, software runs our voting machines and manages our elections.

Sunday's NY Times Magazine has an extended piece about faulty radiation delivery for cancer treatment. The article details two particular fault modes: procedural screwups and software bugs.

The procedural screwups (e.g., treating a patient with stomach cancer with a radiation plan intended for somebody else's breast cancer) are heartbreaking because they're something that could be completely eliminated through fairly simple mechanisms. How about putting barcodes on patient armbands that are read by the radiation machine? "Oops, you're patient #103 and this radiation plan is loaded for patent #319."

The software bugs are another matter entirely. Supposedly, medical device manufacturers, and software correctness people, have all been thoroughly indoctrinated in the history of Therac-25, a radiation machine from the mid-80's whose poor software engineering (and user interface design) directly led to several deaths. This article seems to indicate that those lessons were never properly absorbed.

What's perhaps even more disturbing is that nobody seems to have been deeply bothered when the radiation planning software crashed on them! Did it save their work? Maybe you should double check? Ultimately, the radiation machine just does what it's told, and the software than plans out the precise dosing pattern is responsible for getting it right. Well, if that software is unreliable (which the article clearly indicates), you shouldn't use it again until it's fixed!

What I'd like to know more about, and which the article didn't discuss at all, is what engineering processes, third-party review processes, and certification processes were used. If there's anything we've learned about voting systems, it's that the federal and state certification processes were not up to the task of identifying security vulnerabilities, and that the vendors had demonstrably never intended their software to resist the sorts of the attacks that you would expect on an election system. Instead, we're told that we can rely on poll workers following procedures correctly. Which, of course, is exactly what the article indicates is standard practice for these medical devices. We're relying on the device operators to do the right thing, even when the software is crashing on them, and that's clearly inappropriate.

Writing "correct" software, and further ensuring that it's usable, is a daunting problem. In the voting case, we can at least come up with procedures based on auditing paper ballots, or using various cryptographic techniques, that allow us to detect and correct flaws in the software (although getting such procedures adopted is a daunting problem in its own right, but that's a story for another day). In the aviation case, which I admit to not knowing much about, I do know they put in sanity-checking software, that will detect when the the more detailed algorithms are asking for something insane and will override it. For medical devices like radiation machines, we clearly need a similar combination of mechanisms, both to ensure that operators don't make avoidable mistakes, and to ensure that the software they're using is engineered properly.

In today's New York Times, we read that Hollywood is working on a grand unified video DRM scheme intended to allow for video portability, such as, for example, when you visit a hotel room, you'd like to have your videos with you.

What's sad, of course, is that you can have all of this today with very little fuss. I use iTiVo to extract videos from my TiVo, transcoding them to an iPhone-compatible format. I similarly use Fairmount to rip DVDs to my hard drive, making them easy to play later without worrying about the physical media getting damaged or lost. But if I want to download video, I have no easy mechanism to download non-DRM content. BitTorrent gives access to many things, including my favorite Top Gear, which I cannot get through any other channel, but many things I'd like aren't available, and of course, there's the whole legality issue.

I recently bought a copy of Disney/Pixar's Up (Blu-ray), which includes a "Digital Copy" of some sort that's rippable, but the other ones are rippable as well (even the Bluray), so I haven't bothered to sort out how the "Digital Copy" works.

(UPDATE: the disc contains Windows and Mac executables which will ask the user for an "activation code" which is then sent to a Disney server which responds with some sort of decryption key. The resulting file is then installed in iTunes or Windows Media Player with their native DRM restrictions. The Disney server, of course, wants you to set up an account, and they're working up some sort of YouTube-ish streaming experiences for movies where you've entered an activation code.)

So what exactly are the Hollywood types cooking up? There are no technical details in the article, but the broad idea seems to be that you authenticate as yourself from any device, anywhere, and then the central server will let you at "your" content. It's unclear the extent to which they have an offline viewing story, such as you might want to do on your computer on an airplane. One would imagine they would download an encrypted file, perhaps customized for you, along with a dedicated video player that keeps the key material hidden away through easily broken, poorly conceived mechanisms.

It's not like we haven't been here before. I just wonder if we'll have a repeat of the ill-fated SDMI challenge.

This is a bit off from the usual Freedom to Tinker post, but with tomorrow being "Black Friday" and retailers offering some steep discounts on consumer electronics, many Tinker readers will be out there buying gear or will be offering buying advice to their friends.

Over the past several months, several friends of mine have mentioned that they were considering "moving up" to a D-SLR camera and asked me for advice. I've been what you might term a "serious amateur" photographer since high school, when I was the head photographer for the school yearbook and newspaper. (It was a non-trivial issue for me to decide whether to make my career in photography or in computers.)

To address this, I wrote a guide to upgrading your digital camera. I've written this for a non-technical audience. Pass it around and enjoy.

Sunday's New York Times had an article, Studios' Quest for Life After DVDs. To nobody's surprise, consumers want to have convenient access to "their" media, wherever they happen to be, without all the annoying restrictions that come into play when you add DRM to the picture. To many people's surprise, sales of DVDs (much less Blu-ray) are in trouble.

In the third quarter, studios’ home entertainment divisions generated about $4 billion, down 3.2 percent from a year ago, according to the Digital Entertainment Group, a trade consortium. But digital distribution contributed just $420 million, an increase of 18 percent.

Given that DVDs are really a luxury good (versus, say, food or electricity), the 3.2 percent drop seems like Hollywood is getting off easy. The growth in digital distribution is clearly getting attention, though. What's going on here? I imagine several things. People sometimes miss their shows. Maybe the cable went out. Maybe the TiVo crashed. Maybe they're on the road. Drop $2 at the iTunes Store and you're good to go. That's attractive and it's real money.

Still, the article goes on to talk about... yet more DRM.

Standing in the way are technology hurdles — how to let consumers play a video on various devices without letting them share it with 10,000 close friends on a pirate site — and the reluctance of studios to cooperate too closely with rivals for reasons of antitrust scrutiny and sheer competitiveness.
...
And piracy, at least conceptually, would be less of a worry. The technology [Disney's Keychest] rests on cloud computing, in which huge troves of data are stored on remote servers so users have access from anywhere. Movies would be streamed from the cloud and never downloaded, making them harder to pirate.

Of course, this is baloney. If it's going to work on my iPhone while I'm sitting in an airplane, the entire video needs to be stored there in advance. Furthermore, if the video is supposed to be "high definition," that's a bare minimum of 5 megabits/sec. (Broadcast HD is 20 megabits/sec and Blu-ray is 48 megabits/sec.) Most home DSL or cable modem connections either will never go that fast, or certainly cannot maintain those speeds without hiccups, particularly when sharing the line with other users. To do high quality video, you either have to have a real broadcast medium (cable, over-the-air, or satellite) or you have to download in advance and store on a hard drive.

And, of course, once you've stored the video, it's just not that hard to extract it. And it always will be. The challenge for Hollywood is to change the incentives of the game. Maybe sell me a flat-rate subscription. Maybe bundle it with my DSL provider. But make the experience compelling enough and cheap enough, and I'll do it. I regularly extract video from my TiVo and copy it to my iPhone via third-party software. It's practically painless and it happens to yield files that I could share with the world, but I don't. Why? Because there's real downside (I'd rather not get sued, thanks), and no particular upside.

So, dearest Hollywood executive, consider that selling your content for a reduced price, with no DRM, is not the same thing as "giving it away." If you allow third-parties to license your content and distribute it without DRM, you can still go after the "pirates", yet you'll allow normal people to enjoy your work without making them suffer for it. Yes, you may have kids copying content from one to the next, just like we used to do dubbing cassette tapes, but those incremental losses can and will be offset by the incremental gains of people enjoying your work and hitting the "buy" button.

I just got my invitation to Google Wave. The prototype that's now public doesn't have all of the amazing features in the original video demos. At this point, it's pretty much just a way of collecting IM-style conversations all in one place. But several of my friends are already there, and I've had a few conversations there already.

How am I supposed to know that there's something new going on at Wave? Right now, I need to keep a tab open in my browser and check in, every once in a while, to see what's up. Right now, my standard set of tabs includes my Gmail, calendar, RSS reader, New York Times homepage, Facebook page, and now Google Wave. Add in the occasional Twitter tab (or dedicated Twitter client, if I feel like running it) plus I'll occasionally have an IM window open. All of these things are competing for my attention when I'm supposed to be getting real work done.

A common way that people try to solve this problem is by building bridges between these services. If you use Twitter and Facebook, there are several ways to arrange for your tweets to show up at Facebook (bewildering Facebook users with all the #hashtags and @references) and there are also a handful of ways for getting data out of Facebook. I'd been using FriendFeed as a central hub for all this, but it would sometimes stop working for days at a time. Now that they've been bought out by Facebook, maybe this will shake itself out.

The bigger problem is that these various vendors and technologies have different data models for visibility and for how metadata is represented. In Twitter, everything is default-public, follow-up comments are first-class objects in the system, and there's effectively no metadata outside of the message, causing Twitter users to have adopted a variety of seemingly obscure conventions (e.g., "RT" to indicate a retweet of some other tweet). Contrast this with Facebook, where comments are a very different sort of message from the parent messages, where they have all sorts of security rules (that nobody really understands) about who can see what, and where there is actually structure to a message. If I link to a Youtube video, it gets magically embedded, versus the annoying URL shorteners that people have to use to shoehorn messages into Twitter.

Comments are a favorite area for people to complain. Twitter comments are often implicit with the @username tags. If I'm following a friend and a friend-of-my-friend comments on one of their tweets, I won't necessary see it. In Facebook, I have a better shot at seeing those comments. But what if I wrote a blog post here at Freedom to Tinker, which Facebook nicely picks it up and makes it look just like I posted a note on my Facebook page. Now we'll have comments on Freedom to Tinker and more comments inside Facebook which won't intermingle. Of course, thanks to FriendFeed, a tweet will (probably) be automatically generated when I post this, causing some small amount of Twitter commenting traffic, and there may be comments within FriendFeed itself as well as Google Reader commentary (which is also different from Google Reader's "share with note" commentary).

Given these disparate data models, there's no easy way to unify Twitter and Facebook, much less the commenting disaspora, even assuming you could sort out the security concerns and you could work around Facebook's tendency to want to restrict the flow of data out of its system. This is all the more frustrating because RSS completely solved the initial problem of distributing new blog posts in the blog universe. I used to keep a bunch of tabs open to various blog-like things that I followed, but that quickly proved unwieldy, whereas an RSS aggregator (Google Reader, for me) solved the problem nicely. Could there ever be a social network/microblogging aggregator?

There are no lack of standards-in-the-wings that would like to do this. (See, for example, OpenMicroBlogging, or our own work on BirdFeeder.) Something like Google Wave could subsume every one of these platforms, although I fear that integrating so many different data models would inevitably result in a deeply clunky UI.

In the end, I think the federation ideas behind Google Wave and BirdFeeder, and good old RSS blog feeds, will ultimately win out, with interoperability between the big vendors, just like they interoperate with email. Getting there, however, isn't going to happen easily.

I hate to sound like a broken record, complaining about professional mail distribution / spam-houses that are entirely unwilling to require their customers to follow a strict opt-in discipline. But I'm going to complain again and I'm going to name names.

Today, I got a spam touting a Citrix product ("Free virtualization training for you and your students!"). This message arrived in my mailbox with an unsubscribe link hosted by xmr3.com which bounced me back to a page at Citrix. The Citrix page then asks me for assorted personal information (name, email, country, employer). There was also a mailto link from xmr3 allowing me to opt-out.

At no time did I ever opt into any communication from Citrix. I've never done business with them. I don't know anybody who works there. I could care less about their product.

What's wrong here? A seemingly legitimate company is sending out spam to people who have never requested anything from them. They're not employing any of the tactics that are normally employed by spammers to hide themselves. They're not advertising drugs for sexual dysfunction or replicas of expensive watches. Maybe they got my email by surfing through faculty web pages. Maybe they got my email from some conference registration list. They've used a dubious third-party to distribute the spam who provides no method for indicating that their client is violating their terms of service (nor can their terms of service be found anywhere on their home page).

Based on this, it's easy to advocate technical countermeasures (e.g., black-hole treatment for xmr3.com and citrix.com) or improvements to laws (the message appears to be superficially compliant with the CAN-SPAM act, but a detailed analysis would take more time than it's worth). My hope is that we can maybe also apply some measure of shame. Citrix, as a company, should be embarrassed and ashamed to advertise itself this way. If it ever became culturally acceptable for companies to do this sort of thing, then the deluge of "legitimate" spam will be intolerable.

The Marines recently issued an order banning social network sites (Facebook, MySpace, Twitter, etc.). The Pentagon is reviewing this sort of thing across all services. This follows on the heels of a restrictive NFL policy along the same lines. Slashdot has a nice thread, where among other things, we learn that some military personnel will contract with off-base ISPs for private Internet connections.

There are really two separate security issues to be discussed here. First, there's the issue that military personnel might inadvertently leak information that could be used by their adversaries. This is what the NFL is worried about. The Marines order makes no mention of such leaks, and they would already be covered by rules and regulations, never mind continuing education (see, e.g., loose lips sink ships). Instead, our discussion will focus on the issue explicitly raised in the order: social networks as a vector for attackers to get at our military personnel.

For starters, there are other tools and techniques that can be used to protect people from visiting malicious web sites. There are black-list services, such as Google's Safe Browsing, built into any recent version of Firefox. There are also better browser architectures, like Google's Chrome, that isolate one part of the browser from another. The military could easily require the use of a specific web browser. The military could go one step further and provide sacrificial virtual machines, perhaps running on remote hosts and shared by something like VNC, to allow personnel to surf the public Internet. A solution like this seems infinitely preferable to forcing personnel to use third-party ISPs on personal computers, where vulnerable machines may well be compromised, yet go unnoticed by military sysadms. (Or worse, the ISP could itself be compromised, giving a huge amount of intel to the enemy; contrast this with the military, with its own networks and its own crypto, which presumably is designed to leak far less intel to a local eavesdropper.)

Even better, the virtual machine / remote display technique allows the military sysadm to keep all kinds of forensic data. Users' external network behavior creates a fantastic honeynet for capturing malicious payloads. If your personnel are being attacked, you want to have the evidence in hand to sort out who the attacker is and why you're being attacked. That helps you block future attacks and formulate any counter-measures you might take. You could do this just as well for email programs as web browsing. Might not work so well for games, but otherwise it's a pretty powerful technique. (And, oh by the way, we're talking about the military here, so personnel privacy isn't as big a concern as it might be in other settings.)

It's also important to consider the benefits of social networking. Military personnel are not machines. They're people with spouses, children, and friends back home. Facebook is a remarkably efficient way to keep in touch with large numbers of friends without investing large amounts of time -- ideal for the Marine, back from patrol, to get a nice chuckle when winding down before heading off to sleep.

In short, it's problematic to ban social networking on "official" machines, which only pushes personnel to use these things on "unofficial" machines with "unofficial" ISPs, where you're less likely to detect attacks and it's harder to respond to them. Bring them in-house, in a controlled way, where you can better manage security issues and have happier personnel.

You can run, but you can't hide. Here are a few of the latest things I've seen, in no particular order.

• On a PHPBB-style chat board which I sometimes frequent, there was a thread about do-it-yourself television repair, dormant for over a year. Recently, there was a seemingly robotic post, from a brand new user, that was still on-topic, giving general diagnosis advice and offering to sell parts for TV repair. The spam was actually somewhat germane to the main thread of the discussion. Is it still spam?
• In my email, I recently got a press release for a local fried chicken franchise celebrating their 40th anniversary. My blogging output generally doesn't extend to writing restaurant reviews (tempting as that might be), although I do sometimes link to foodie things from Google Reader which will also show up in my public FriendFeed. Spam or not spam?

The New York Times reports that China will start requiring censorship software on PCs. One interesting quote stands out:

Zhang Chenming, general manager of Jinhui Computer System Engineering, a company that helped create Green Dam, said worries that the software could be used to censor a broad range of content or monitor Internet use were overblown. He insisted that the software, which neutralizes programs designed to override China’s so-called Great Firewall, could simply be deleted or temporarily turned off by the user. “A parent can still use this computer to go to porn,” he said.

In this post, I'd like to consider the different capabilities that software like this could give to the Chinese authorities, without getting too much into their motives.

Firstly, and most obviously, this software allows the authorities to do filtering of web sites and network services that originate inside or outside of the Great Firewall. By operating directly on a client machine, this filter can be aware of the operations of Tor, VPNs, and other firewall-evading software, allowing connections to a given target machine to be blocked, regardless of how the client tries to get there. (You can't accomplish "surgical" Tor and VPN filtering if you're only operating inside the network. You need to be on the end host to see where the connection is ultimately going.)

Software like this can do far more, since it can presumably be updated remotely to support any feature desired by the government authorities. This could be the ultimate "Big Brother Inside" feature. Not only can the authorities observe behavior or scan files within one given computer, but every computer now because a launching point for investigating other machines reachable over a local area network. If one such machine were connected, for example, to a private home network, behind a security firewall, the government software could still scan every other computer on the same private network, log every packet, and so forth. Would you be willing to give your friends the password to log into your private wireless network, knowing their machine might be running this software?

Perhaps less ominously, software like this could also be used to force users to install security patches, to uninstall zombie/botnet systems, and perform other sorts of remote systems administration. I can't imagine the difficulty in trying to run the Central Government Bureau of National Systems Administration (would they have a phone number you could call to complain when your computer isn't working, and could they fix it remotely?), but the technological base is now there.

Of course, anybody who owns their own computer will be able to circumvent this software. If you control your machine, you can control what's running on it. Maybe you can pretend to be running the software, maybe not. That would turn into a technological arms race which the authorities would ultimately fail to win, though they might succeed in creating enough fear, uncertainty, and doubt to deter would-be circumventors.

This software will also have a notable impact in Internet cafes, schools, and other sorts of "public" computing resources, which are exactly the sorts of places that people might go when they want to hide their identity, and where the authorities could have physical audits to check for compliance.

Big Brother is watching.

On the 20th anniversary of the Tiananmen Square events (protests? uprising? insurrection? massacre?), the New York Times' Lens Blog put up a great piece about the four different photographers who photographed the iconic "Tank Man". Inevitably, half of the story concerns the technical details of being in the right place and having the right equipment configuration to capture the image (no small thing in the middle of a civil insurrection). The other half of the story, though, is about how the film got out of the camera and out to us. The story of Tank Man (NYT article, PBS Frontline piece) is quite amazing, by itself, but I want to focus on the photographers.

The most widely seen photo, by Jeff Widener, and all the other good coverage of Tank Man was all taken from one particular hotel, and the government security services were well aware of it. Our photographers had to get their images out. But how? Widener had a "long-haired college kid" assistant who smuggled several rolls of film in his underwear. Another photographer, Charlie Cole, wrote this:

After taking the picture of the showdown, I became concerned about the PSB’s surveillance of our activities on the balcony. I was down to three rolls of film, with two cameras. One roll held the tank encounter, while the other had other good pictures of crowd and PLA confrontations and of wounded civilians at a hospital.

I replaced the final unexposed roll into the one of the cameras, replacing the tank roll, and reluctantly left the other roll of the wounded in the other camera. I felt that if the PSB searched the room or caught me, they would look even harder if there was no film in the cameras.

I then placed the tank roll in a plastic film can and wrapped it in a plastic bag and attached it to the flush chain in the tank of the toilet. I hid my cameras as best I could in the room. Within an hour, the PSB forced their way in and started searching the room. After about five minutes, they discovered the cameras and ripped the film out of each, seemingly satisfied that they had neutralized the coverage. They then forced me to sign a confession that I had been photographing during martial law and confiscated my passport.

In both of these cases, the film was ultimately smuggled to the local bureau of the Associated Press who then processed, scanned, and transmitted the images. This leads me to wonder how this sort of thing would play out today, when photographers have digital cameras, where the bits are much easier to copy and transmit.

First, a few numbers. A "raw" image file from a modern Nikon D700 takes about 13MB and that already includes the (lossless) compression. Back in the film days, the biggest 35mm rolls could hold 36 images (maybe 38 if you were willing to push it on the edges), which tended to keep photographers' desire to press the button in check. Today, when giant memory cards cost virtually nothing, it's trivial for a photojournalist to generate tens of gigabytes of raw data in a day of work. So... how long does it take to transmit that much data? Let's say a hotel's Internet connection gives you a snappy 1.5 megabits of upstream bandwidth. That means it takes about 70 seconds to transmit one raw image.

If you fear the police will knock down your door at any moment, you don't have time to send everything. That means that you, the photographer, have got to crunch your pictures through your laptop in a big hurry. If you've got the fastest cards and card reader, you'll be able to copy the data to your hard drive at maybe three pictures per second. Got a thousand pictures on that memory card and you're waiting a nerve-wracking six minutes to complete the copy.

At the point where you're worried about somebody busting down the door, you're not in the frame of mind to tweak with your exposure, color balance, and so forth. Pretty much all you're thinking is "which one is the winner", so you're blasting through trying to select your favorites and then try to upload them.

Meanwhile, we need to consider the capabilities of the adversary. The PRC could well have prevented us from seeing Widener and Cole's photos, simply by locking down the AP's offices. (Two other photographers smuggled their raw film out of the country for external processing.) In the modern era, in a country like the PRC, they could just as well cut off the Internet altogether. (We already know that the PRC is cranking up the filtering of the Great Firewall to block Flickr, Twitter, and other services around the anniversary of the Tiananmen Square events, so it's easy to imagine far more draconian policies.) This places our hypothetical digital photographer in much the same problematic space as the film photographers of twenty years ago. Now we need to smuggle the bits out by hand.

Traveling with film is a huge pain. Higher-speed film, and particularly black & white film, is annoyingly sensitive to airport x-ray scanners. It's similarly sensitive to humidity and temperature. And, most important, you can't see it or copy it until you process it, which isn't really an option in a war zone. Instead, you've got the one roll with the one photo that you really want to get out. Alfred Hitchcock would call the film a MacGuffin and would spin a glorious tale around it.

Digital changes all that. Now, even if the Internet is down, the ability to copy bits is incredibly helpful to our photographer. An iPod, iPhone, or other such device will commonly have gigabytes of solid state storage within. That's not enough room for everything, but it's certainly enough room for the photographer to make copies of all the good stuff. Similarly, with memory cards getting so remarkably small (e.g., a Micro-SD card is 15mm x 11mm x 1mm), it's easy to imagine smuggling them in a variety of places. Advantage to the photographer? Certainly so, but also very dependent on how much time and preparation was available before the police busted down the door. The CompactFlash cards used by most D-SLRs (43mm x 36mm x 3.3mm) are much harder to hide (e.g., you can't just shove one into a crack in the floor).

There probably isn't much point in trying to encrypt or hide the data. If the police are busting down your door, they'll just take everything they can find and wipe everything before they give it back to you.