Plus gas. Comments here.
"Pourriez-vous s'il vous plaît répéter la question?" said Andre, head developer. His voice was garbled over a VoIP connection. "My English ... not great."
Yvonne sighed. She was the project manager of MetaPortal, the creatively named flagship product of MetaCortex. "Why is there no email validation for the signup form?" Yvonne repeated, speaking slowly and clearly.
"Email already required," Andre replied. "What more validation do you need?"
Yvonne lamented forgetting so much of her high school French that she couldn't say "regular expressions."
This call with Andre was the last in a dozen late-night conferences with the temperamental, but always polite, programmer, who lived in Normandy. In fact, his entire team was composed of four different nationalities and spoke three different languages. Yvonne, along with the executives of MetaCortex, worked in New York.
Yvonne tried again. "Regular expressions. Can you add a regular expression to the email validation? Right now someone can type 'merde' for their address and the form accepts it."
Andre chuckled. "Ah, make the email straight. No more 'merde' emails."
"That should do it for this sprint," Yvonne said, relieved.
Despite the linguistic difficulties, Andre and his team met their deadline, and the code for MetaPortal 1.9.11 was released the following Tuesday. Yvonne, exhausted from all those late-night conference calls, took a nap in her apartment that afternoon. She awoke the next morning to a banging at her door.
"Yvonne?" It was Charles, the head of Customer Service. "Are you alive?"
"Yes," she mumbled. She had slept close to 18 hours, she noticed. "Is something wrong?"
"We had to roll back 1.9.11. A bunch of users couldn't log into MetaPortal."
Yvonne washed her face and followed Charles back to the office. He showed her a list of thousands of angry emails from customers unable to log into their accounts. "They started the moment the code was pushed up to production."
"Merde," Yvonne said. "I'll get Andre on the horn."
"I do not understand, it works on my computer." Andre sounded confused and tired over the VoIP connection. "You say no one can sign up in France?"
"Some can, most can't. Some can't sign up in the US. Half can in Germany, most can't in Japan, most can in Australia. Almost everyone can in the UK. We're still trying to find commonalities."
"I do not think I can triage," Andre said. "I do not live outside of France."
"I've got some interns playing around with it," Charles said. "Maybe it's just a matter of enough monkeys banging enough keyboards at this point."
Yvonne got coffee from the vending machine downstairs while Charles and his team looked at the issue. It's not regional, not exactly. It should be uniform if it were, say, an issue with a national firewall. So why is the UK the one country where it's not a problem? What's very common in England, a bit less common in the US and Australia, and not common at all elsewhere?
Merde! She headed back for Charles' desk.
"It only works in English." Charles pointed to his computer as she found him. "Something goes wrong when a user selects another language."
"I think I know why," Yvonne said. "May I?" She pointed to his computer.
He nodded. Yvonne opened the MetaPortal login page in a new browser window. "We use an automated translation service to localize much of our content. When the session is in English, nothing is translated ... but when I switch it to something else, like our friend Andre's native French, much of the output gets processed in the translation API. Did your monkeys try switching languages before?"
"Not before now," Charles said.
"Okay, watch this." Yvonne typed her account info into the login screen and hit submit. The browser showed an error icon. Yvonne opened the Inspector window. "'pour
is not a valid keyword.'"
"I'll be..." Charles said. He took the mouse from Yvonne and viewed the page source. Where they expected to see:
for (var i = 1; i < a.length; i++) {
Was instead:
pour (var i = 1; i < a.length; i++) {
"I'll tell Andre to add an exception to the translation API. It should always skip embedded JavaScript code. How can I say that in French so he understands perfectly?"
"Developers and management already speak different languages," Charles said, "even if they both speak English."
E. T. wrote to tell us of a support tale from days long gone by at a company long since acquired by a much larger behemoth. A customer had called in, entered all of his information, but hung up before a human got on the line. The support system generated a nameless ticket which got randomly assigned to one of the support folks. Then the customer called back, entered all of his information again, and got E. T.
The customer wanted to delete slice 0 on his system. For those of you not familiar with this, in *nix, slice 0 is the root of the file system and basically points to where everything on the disk is located. Deleting slice 0 is the equivalent of deleting everything on the entire hard disk. While there are the occasional disk corruptions that require this action, they are exceedingly rare, and once done, you are forced to reinitialize and re-install the operating system.
E. T. told him that he didn't want to do that because it would wipe out the entire file system. The customer persisted, insisting that he wanted to free up the precious GB of disk space that this unnecessary file system was consuming. He insisted that he did not install whatever was on this file system, and that it didn't belong there, so he was going to delete it, and needed instructions on how to get it done.
Since E. T. was obligated to help the customer, he finally went over to the software guys, and asked one: Do you know how to delete slice 0? Naturally, the software guy replied: You don't want to do that... E. T. continued: I know, but this guy is insisting that he didn't install anything on that partition, and he wants to reclaim the space. I know it's stupid, but "The customer is always right" and all that; we're obligated to support him and answer his question! The software guy told E. T. to Talk to Bob.
E. T. hunted down Bob and asked: Hey, Bob, do you know how to delete slice 0? Of course, Bob replied: You don't want to do that...
This went on with several people, until finally E. T. got someone who told him how to do it.
As E. T. was headed back to his desk, the guy next to him asked: Hey is that F.P. Dingbat at XYZ company?
Yeah why?
Because when he originally called and hung up, his case was assigned to me; if you have him I'll just close it out.
E. T. decided there was a better way to handle this situation: No don't. He is about to do something above and beyond stupid (little did E. T. know the true scale of Epic Stupidity™ that the customer was about to commit). So let's give him 1/2 an hour to hang himself, then you call back so we can see what happened.
E. T. got back on the phone and told the guy how to delete the file system, but again, advised him that what he was about to do was evil, bad, would make his life Hell-on-Earth, and that he should absolutely, positively, not do it. Of course, the customer got all snooty at being told he was making a mistake, and barked back: See, you CAN do it!, and hung up.
A half hour later, the other support guy calls the customer back to see if he could help him with his original issue.
Did the guy delete slice 0? Yes but in a far more destructive way than one might imagine. It turned out that he had issued a command to SU on every machine in his company and execute the command sequence to remove slice 0. Then he executed the commands on his own box. He took the whole place down!
The customer asked the support guy: I only entered the one command; what the fsck did I just do?
Update: This is a controversial claim. I have some sources I cannot name. Also: I don't have the exact details as to what "many" means: 1%? 10% 30%??
Update: If we assume the NSA controls 1% of Tor nodes, that comes out to one-in-a-million chance the NSA will unmask somebody on any random connection. If a million connections are created per day, that means the NSA unmasks one person per day.
Update: This is partly mitigated by the "guard" ingress node concept. You crease only a single connection to the guard node, then fan out paths from there. But, mitigated doesn't mean the same thing as "fixed".
Update: Also, some of your egress traffic is poorly encrypted, such as the 1024-bit keys without forward security that Facebook uses.
Update: @addelindh points out that things like SSLstrip often works because people aren't paying attention and websites don't support things like HSTS, and thus, even when you want SSL, it'll sometimes fail for you in the face of a hostile attacker. Somebody needs to setup an exit node, then SSLstrip it to figure out how often that works.
Update: We know the NSA can crack 1024-bit keys, because would cost only a few million dollars. What we don't know how many such keys it can crack per day. The number could be less than one such key per day.
Major Update: Because of Tor's "perfect forward secrecy", the NSA wouldn't be cracking the RSA key when eavesdropping. Instead, they would need to crack the "ephemeral" keys. A lot of older servers use 1024-bit DH ephemeral keys, which are about as easy to break as 1024-bit RSA keys. Newer servers use 256-bit ECDH keys which are a lot stronger, and likely not crackable by the NSA (estimates say NSA can crack up to 160-bit ECDH keys). Thus, for older servers, the ability of the NSA to passively eavesdrop and crack keys is a big threat, but for newer servers, it's likely not a threat. (I'm using http://www.keylength.com/ and round numbers here for key lengths). (I'm using http://torstatus.blutmagie.de/ and my own pcaps to confirm a lot of 1024-bit DH is still out in the Tor nodes).
@ErrataRob so "NSA runs nodes" and "NSA DoSes nodes" both would show up in the network. Empirically we don't see evidence for those.
— @eqe (Andy Isaacson) (@eqe) August 7, 2013
@ErrataRob Tor's 'guard node' design feature affects, but does not completely render invalid, your "Tor creates many new paths" point.
— Nick Mathewson (@nickm_tor) August 7, 2013
@ErrataRob Recent Tors use ECDH for forward secrecy, so breaking RSA keys would only be an (expensive) way to impersonate a node.
— Nick Mathewson (@nickm_tor) August 7, 2013
@ErrataRob @thegrugq There are a number of factual inaccuracies in your blog post, I wish you had asked me to review it before publishing.
— Runa A. Sandvik (@runasand) August 7, 2013
@ErrataRob @stevelord @donicer Isn't that because people have unrealistic expectations on Tor? Not broken, just not a silver bullet.
— Andreas Lindh (@addelindh) August 7, 2013