Your Security Needs to Work Together

January 2nd, 2017

Here follows a short analogy regarding the importance of getting all your security to work together.

Let us say that a bank is under attack by a gang of thieves that intend to break into its vault and walk away with what’s inside it. Not as sophisticated as getting one of their members to be promoted to a C-level position and then engage in massive securities fraud, but it’ll do for our analogy.

Let us say also that the bank has a security force to deal with each aspect of physical security, but they do not cooperate with each other. The thieves attack in broad daylight, after they’ve had a nice lunch and a nap, so they’re absolutely fresh.

At 1500, the guard watching the video surveillance system notes in his log that ten masked men with weapons entered the bank. He knows this will be something for his team to discuss in their weekly meeting in two days.

At 1501, the guards in charge of the bank lobby note the ten armed men heading towards the vault door. As they are not robbing any tellers in the lobby, the lobby security men do not interact with the men in masks.

At 1503, the guards in charge of the vault door respond to an alert that the door has been blown off its hinges and is laying on the floor of the bank vault. With crack speed, they remove the old door and replace it with a new one. A new guy on the team asks what to do about the ten guys with masks that are empyting out the deposit boxes, but the team lead tells him not to worry, the deposit box guys will handle that: they just need to focus on the door.

At 1505, the guards in charge of the deposit boxes arrive at the vault door, but they cannot enter, as they do not have access.

At 1534, the guards in charge of the deposit boxes note that they can now enter the vault area, as the door has been blown off its hinges. They pass by ten men in masks, each with a weapon and a large bag that seems to be full to capacity. As the men are leaving the vault area, that is not their security concern.

At 1535, the guards in charge of the vault door respond to another alert that a door has been blown off its hinges. It’s a good thing that they always keep two spare vault doors in stock! The team lead prepares the two blown-up doors for an RMA to their manufacturer.

At 1536, the guards in charge of the lobby note the ten men in masks, en route out of the bank front door. Again, they pose no immediate threat to the tellers, so there is no call to engage with them.

At 1538, the guard in charge of the security cameras notes in his log that ten suspicious-looking men in masks with large, full bags, were making their way to their cars in the parking garage. This will be another interesting thing to discuss at the security camera guard team meeting, exactly the sort of thing they should be noting in their logs.

At 1545, the guards in charge of the vault door are wondering what to do with a request for access out of the vault from the security deposit box team. How did they manage to get into the vault without anyone having authorized their entry? The vault door team lead plans to write up the security deposit team for an access violation, as soon as he’s finished with the vault door RMA paperwork. 


September 19th, 2016

I had a very sad friend. His company bought all kinds of really cool stuff for security monitoring, detection, and response and told him to point it all at the firm’s offices in the Russian Federation. Because Russia is loaded with hackers, right? That’s where they are, right?

Well, he’d been running the pilot for a week and had nothing to show for it. He knows that the tools have a value, and that his firm would benefit greatly from their widespread deployment, but he’s worried that, because he didn’t find no hackers nowhere in the Hackerland Federation, his executives are going to think that these tools are useless and they won’t purchase them.

So I asked him, “Do you have any guidance from above on what to look for?”

“Hackers. They want me to look for hackers.”

“Right. But did they give you a software whitelist, so that if a process was running that wasn’t on the list, you could report on it?”

“No. No whitelist.”

“What about a blacklist? Forbidden software? It won’t have everything on it, but it’s at least a start.”

“Yes, I have a blacklist.”

“Great! What’s on it?”

“Hacker tools.”

“OK, and what are listed as hacker tools?”

My friend sighed the sigh of a thousand years of angst. “That’s all it says. Hacker tools. I asked for clarification and they said I was the security guy, make a list.”

“Well, what’s on your list?”

“I went to Wikipedia and found some names of programs there. So I put them on the list.”

“And did you find any?”

“Some guys are running the Opera browser, which has a native torrenting client. I figured that was hacker enough.”

Well, security fans, that’s something. We got us a proof of concept: we can find active processes. I described this to my friend, and hoped that he could see the sun peeking around the clouds. But it was of no help.

“They’re not going to spend millions on products that will tell them we’re running Opera on a handful of boxes!”

He had a point, there. Who cares about Opera? That’s not a hacker tool as featured on the hit teevee show with hackers on it. And, to be honest, the Russian offices were pretty much sales staff and a minor production site. The big stashes of intellectual property and major production sites were in the home office, in Metropolis, USA.

So I asked, “Any chance you could point all that stuff at the head office?”

“What do you mean?”

“Well, it’s the Willie Sutton principle.”

“Who was Willie Sutton?”

I smiled. “Willie Sutton was a famous bank robber. His principle was to always rob banks, because that’s where the money was. Still is, for the most part. Russia in your firm is kind of like an ATM at a convenience store. There’s some cash in it, but the big haul is at the main office. Point your gear where the money is – or intellectual property – and see if you don’t get a lot more flashing lights.”

My friend liked that. He also liked the idea of getting a software whitelist so he’d know what was good and be able to flag the rest as suspect. He liked the idea of asking the execs if they had any guidance on what information was most valuable, so that he could really take a hard look at how that was accessed – and who was accessing it.

And maybe there were tons of hackers in Russia, but they weren’t hacking anything actually in Russia. And maybe said hackers weren’t doing anything that was hacking-as-seen-on-television. Maybe they were copying files that they had legitimate access to… just logging on, opening spreadsheets, and then doing “Save As…” to a USB drive. Or sending it to a gmail account. Or loading it to a cloud share…

The moral of the story is: If your security policy is driven by the popular media, you don’t have a security policy.

Election Fraud Alert – August 2016

August 30th, 2016

This made me chuckle:

Oh, so NOW elections can be hacked? Sheesh… where were these guys when Diebold began putting out voting machines that had no paper trail to corroborate votes with? That was back in the 90s and I remember howling about those. There was the Mexican national election of 1988, in which Cuahutemoc Cardenas was in the lead, strongly, when “the system crashed”… after being restored, Carlos Salinas de Gortari was out in front. All the ballots were burned three years later so that nobody could check to see if the voting database had been restored properly after crashing. President de la Madrid later admitted that the voting that year had been rigged.,_1988

We’ve had in the US media outlets print election results before voting happened (most recently in Florida), voter registration screening software match white felons’ names to innocent black citizens, voter registration software deny the existence of residences (and hence said residents’ right to vote), and other miscarriages of election justice with specifically digital components to them. The idea that only just now are election machines vulnerable to hackers (no doubt mustache-twirling Russians!) is outrageous.

Thing is, those Diebold boxes are incredibly vulnerable to local tampering. There’s no need to run an attack on voting databases over a network. Physical access to the Diebolds means they can be made to report whatever the guy with access wants them to report. No paper trail, remember? So a story like this sounds a lot to me like part of a propaganda campaign, to extend a narrative that needs to be spun for possible future use.

We’ve already seen a narrative connect Russians to Donald Trump. I wonder aloud if we won’t see Trump victories blamed on voter fraud, connected back to “Russians” without any credible evidence offered up, and then see Trump’s cronies face criminal voter fraud charges. Such a move would be a “just in case” maneuver, should the election need a little stealing to get it back on track with how certain people believe it should come out. Interesting how the states mentioned were Illinois, a Democrat-leaning state, and Arizona, a “battleground” state. Illinois is a red herring: Arizona is where fun can be had. That state, plus 10 more electoral votes from either Missouri or Nevada plus New Hampshire, would be all that Clinton needs to get 270 electoral votes and win the election. This is why the story runs… just in case Clinton needs another 11 to win, she’s sown the seeds in Arizona to cast doubt on that election’s outcome, should it go towards the “Russian” favorite son, Donald Trump.

IT Counterintelligence

August 22nd, 2016

I really don’t like blaming all IT-related intrusions on “hackers”. The elaborate ones aren’t just some kid that’s figured out how to do a ransomware scam. They’re run by criminals and spies, or a mix of both. Being attacked by just hackers has a bit of comfort implied in that wording – hackers only use computers, right? They’re people with poor social skills that never break out of that stereotypical mold, right? But to say that criminals or spies were involved means that the attackers have a wide range of tools at hand to get what they want. They’ll engage in blackmail, murder, deception, all that criminal and spy stuff along with the IT-related intrusions as part of a comprehensive program to gather information.

These guys have zero intention of compromising the target beyond what is needed to gather information. They prefer that there be no ransomware or worm taking things down. They want five nines of uptime as much as the CEO does. They penetrate deep into a network and they require more than just an IPS or DLP in place to keep them from taking secrets out. They need some really paranoid thinking on the part of the target’s employees to minimize what they’re able to gather.

To that end, I’m going to recommend two videos from YouTube about counterintelligence.

Take a look at It’s a little harsh, I agree, but it’s also for combat. All the same, it sets up a mindset I’d like users in general to have in regards to security. Don’t write down passwords. Don’t send information over unencrypted, unsecured channels. Don’t think for a minute that LinkedIn and Facebook aren’t unencrypted, unsecured channels… stuff like that…

Another video at about tracking down and gathering information in a counterintelligence operation. It’s not just a matter of stopping the spies: it’s a matter of making sure that they don’t continue to penetrate one’s organization. If a firewall stops an attacker from getting in one way, it’s not necessarily a success. It just means an attacker will try to find another way in.

Real security is never thinking that the bad guys have given up and walked away. Some will, sure. But the rest? They’re always there, watching and probing. Real security means always watching and maintaining OPSEC‍ .

The Internet of Things with Pre-Installed Backdoors

August 12th, 2016


The SEO-friendly URL says it all. The Rockwell PLCs in question have a RW SNMP community common to a range of their devices, undocumented, but if you can find it, you can light up every one of them.

Correction: there’s another SNMP string that allows even more access, also undocumented. That’s what’s makes this newsworthy. Not one backdoor, that’s old news. Two pre-installed backdoors, now we got us a story!

If you work with PLCs, read the article above, check to see if you’re using any of them, and then contact the manufacturer. You need to get all over this like a donkey on a waffle.

The Internet of Things Still in Development

August 11th, 2016

So, I just read an article on some solar panels with a wifi connection that had a default admin name and password. That’s not news. What is news is that the solar panels were still in development and were mistakenly shipped to customers.

As we plunge deeper into the world of everything having an IP address, this incident highlights a new concern: what impact does improper labeling have on security? What if these were autonomous vehicles (our favorite bugbear) shipped to a dealer? More terrifying, what if these were controls for a LNG terminal or a nuclear reactor? It’s bad enough we have default credentials on production devices, but now we have to consider a mis-shipment of even less secure development devices.

Or, we can start to say “no”. The promises of cost savings and higher productivity need to be placed against a realistic risk assessment. Is saving a few bucks per IP-enabled lightbulb worth the possibility of a major PCI breach? OK, maybe I’m engaging in hyperbole, as well, but it’s no worse than the hyperbole of IoT marketers that aren’t telling the full story of how human fallibility is always a constant, even when we use computers to speed our poor decision-making processes.

We’ve had product recalls before, and we’ll have them again. But IoT ubiquity means a window of opportunity between the zero-day and the day of repair to wreak havoc, mayhem, and unintended accidents.

I’ll raise another concern: what about device interoperability? I know that if I have medication A, I may have to abstain from substance B if I don’t want a horrendous drug interaction. When will we be able to look at IoT devices working with each other and possibly breaking code as a result of such interoperation?

We need to have a Serious Discussion of Things before we have an Internet of Things.

A Night at the Outsourcer

August 5th, 2016


Driftwood: All right. It says the, uh, “The first part of the party of the first part shall be known in this contract as the first part of the party of the first part shall be known in this contract” – look, why should we quarrel about a thing like this? We’ll take it right out, eh?
Fiorello: Yeah, it’s a too long, anyhow. (They both tear off the tops of their contracts.) Now, what do we got left?
Driftwood: Well, I got about a foot and a half.

After talking with people from companies whose experiences with their outsourcing‍ contracts can be best described as “disappointing”, I wonder if they didn’t have the equivalent of the‍ Marx Brothers‍ representing them in their contract negotiations. I’m not saying that the corporate lawyers were idiots‍ , just that they may have been outclassed by the outsourcers’ lawyers. This is a specialized situation, after all.

Like the company doing the outsourcing, the outsourcer wants to maximize profits. Outsourcers are not charitable organizations, offering up low-cost business services to help the hapless firm with IT‍ needs. They want to get paid, Jack! Some may want a long-term, quality relationship with a client, but there are plenty out there that want to sign a contract that, on the surface, looks like it will reduce costs, but it contains hidden standard business practices‍ that will rake the clients over the coals.

One of the biggest gotchas in an outsourcing contract is the fact that the relationship between a company and its IT is no longer one of company to employee, but company to contractually provided service. That means the “one more thing” that managers like to ask for from their employees isn’t an automatic wish that will be granted. Did the contract authorize that one more thing? No? Well, that will cost extra, possibly a lot extra.

Another loss is the ability to say, “I know that’s what I wrote, but what I meant was…” as a preface to correcting a requested change. In-house staff can be more flexible and adapt to the refinement of the request. Outsourced staff? Well, it seems as though the staff were engaged to make a specific change, so there’s a charge for that, even though you decided to cancel the change in the middle of it. Now, the change you requested needs to be defined, submitted, and approved in order for us to arrange staff for the next change window…

There’s also the limit on the time-honored technique of troubleshooting the failed change and then making the troubleshooting part of the change. Consider a firewall change and then discovering that the vendor documentation left out a port needed for the application to work. In-house staff have no problem with adding that port and making things work. Outsourcers? If that change isn’t in writing, forget about it until it is. And, then, it may be a matter of rolling back the change and trying again, come the next change window.

Speaking of firewalls, that brings me to the “per line of code” charge. If the contract pays by the line of code, prepare for some bulky code if the contract does not explicitly state that lines of code must be consolidated whenever possible in order to be considered valid and, therefore, billable. Let me illustrate with an example.

My daughter is 14 and has zero experience with firewall rules. I asked her recently how many rules would be needed for two sources to speak to two destinations over five ports. She said five rules would be needed. I then gave a hint that the firewall help file said that ports could be grouped. Then, she proudly said, “one!”

While that’s the right answer for in-house IT staff, it’s the wrong answer for an outsourcer being paid by the line. 20 is the right answer in that case. It blew her mind when I told her how many different firms I’ve heard about that had 20 rules where one would do. As a teenager with a well-developed sense of justice, she was outraged. So long as contracts are signed that don’t specify when, how, and what to consolidate, she will continue to be outraged.

I didn’t have the heart to tell her about how some outsourcers contract to provide services like email, but the contract did not outline all the things we take for granted as part of email but which, technically, are not email. Shared calendars? Not email. Permissions for an admin assistant to open a boss’ Inbox? Not email. Spam filtering? Not email. Email is the mail server sending/receiving to other mail servers and allowing clients to access their own inboxes. Everything else is not email, according to the outsourcers’ interpretation of the contract. Email is just one example, and all the other assumptions made about all the other services add up with the above to create a situation in which the outsourcing costs significantly more than keeping the work in-house.

This can have significant impact on security. Is the outsourcer obligated to upgrade devices for security patching? Is the outsourcer obligated to tune security devices to run optimally? Is the outsourcer required to not use code libraries with security vulnerabilities? If the contract does not specify, then there is zero obligation. Worse, if the contract is a NoOps‍ affair in which the customer has zero visibility into devices or code, then the customer may never know which things need what vulnerabilities mitigated. There may be a hurried, post-signing negotiation of a new section about getting read rights on the firm’s own devices and code… and that’s going to come at a cost.

Another security angle: who owns the intellectual property in the outsourcing arrangement? Don’t make an assumption, read that contract! If the outsourcer owns the architecture and design, your firm may be in for a rough ride should it ever desire to terminate the contract or let it expire without renewing it.

I’m not even considering the quality of work done by the outsourcer or the potential for insider threat – those can be equal concerns for some in-house staff. The key here is that the contract is harsh, literal, and legally binding. That means vague instructions can have disastrous results. Tell an outsourcer to “make a peanut butter and jelly sandwich,” do not be surprised if the outsourcer rips open a bag of bread, smashes open the jars of peanut butter and jelly, mashes the masses of PB & J together, shoves the bread into that mass, and then pulls out the bread slices with a glob of peanut butter, jelly, glass, and plastic between them. He gave you what you specified: it’s not his fault that the instructions were vague.

There can be a place for oursourcing, particularly as a staffing solution for entry-level positions with high turnover. But every time I talk with someone from a place that either is currently in or is recovering from an outsourcing contract that went too far, I hear the horror stories. The outsourcers’ lawyers know what they’re doing and the firm’s lawyers fail to realize how specific they have to be with the contract language to keep from looking like they may as well have been the Marx Brothers‍.

Driftwood (offering his pen to sign the contract): Now just, uh, just you put your name right down there and then the deal is, uh, legal.
Fiorello: I forgot to tell you. I can’t write.
Driftwood: Well, that’s all right, there’s no ink in the pen anyhow. But listen, it’s a contract, isn’t it?
Fiorello: Oh sure.
Driftwood: We got a contract…
Fiorello: You bet.

Hell Hath No Fury Like an Admin Scorned

July 29th, 2016

Take a good look at this guy, because he may be potentially more devastating you your company than a major natural disaster. He is an admin, and he’s not happy about going to work every day.


A network admin from Citibank was recently sentenced to 21 months in prison and $77,000 in fines for trashing his company’s core routers, taking down 90% of their network. Why did he do it? His manager got after him for poor performance.

I don’t know how the manager delivered his news, but it was enough to cause that admin to think he was about to be fired and that he wanted to take the whole company down to hell with him. Thing is, he could have done much worse.

What if he had decided to sell information about the network? What if he had started to exfiltrate data? What if he had set up a cron job to trash even more network devices after his two-week notice was over? And there could be worse scenarios than those… what can companies do about such threats?

It’s not like watching the admin will keep the admin from going berserk. This guy didn’t care about being watched. He admitted to it and frankly stated that he was getting them before they got him. His manager only reprimanded him – who knew the guy was going to do all that just for a reprimand? But, then, would the company have endured less damage if it had wrongfully terminated the admin, cut him a check for a settlement, and then walked him on out? So what about the other admins still there? Once they find out how things work, they could frown their way into a massive bonus and we’re heading towards one of those extremes I mentioned.

So what does a manager do with a poorly-performing employee that’s about to get bad news? Or an amazingly good employee that nobody (including him) is about 10 minutes away from an experience that will make him flip out? Maybe arranging a lateral transfer for the first guy while everyone changes admin passwords during the meeting… but the second guy… there was no warning. He just snapped.

Turns out, good managers don’t need warnings. Stephen Covey wrote about the emotional bank account, and IT talent needs a lot of deposits because the demands of the job result in a lot of withdraws. A good manager is alongside her direct reports, and they know she’s fighting battles for them. That means a great deal to an employee. I know it’s meant a great deal to me. My manager doesn’t have to be my buddy, but if my manager stands up for me, I remember that.

Higher up the ladder, there needs to be a realization in the company that it needs to pay the talent what it is worth. I’ve known people that earned their CCIE, expected a significant bump in pay, and got told that company policy does not allow a pay increase of greater than 3% in a year. They leave the company, get paid 20% more to work somewhere else for a year or two, and then their former employer hires them back for 20% more than that. By that time, though, they’re now used to following money and not growing roots to get benefits over time. By contrast, maybe a 20% bump – or even a 15% bump, maybe – could have kept the employee there.

What are the savings? Not just the pay. The firm doesn’t have to go through the costs of training someone to do the job of the person who’s left. The firm retains the talent, the talent is there longer and now has a reason to try to hold on to those benefits, and there’s a sense of loyalty that has a chance to develop.

If an employee has a sense of loyalty, feels like compensation is commensurate with skills, and has a manager that fights real battles, that employee is better able to ride out the storms of the job and not snap without warning. If that manager has to encourage an employee to do better, maybe then he’ll try harder instead of trashing all the routers.

There may be no way to completely prevent these damaging outbursts from happening, but the best solutions for people’s problems aren’t technological. They’re other people, doing what’s right.

Republican Party: Ur DOIN IT WRONG

July 27th, 2016

Well done, Rince Priebus. The chairman of the Republican National Committee (RNC) said, “Maybe our folks are better at securing our e-mail and our cloud and our data than the DNC. I don’t know what the answer to that is, Andrea, but at this point, we haven’t been hacked… but, I can assure if someone hacked my e-mails, they wouldn’t find me calculating against particular candidates and it’s not something that I would do.”

He could have saved some effort by simply saying, “We are pleased to announce a hackathon, starting now, directed against the RNC servers. While I doubt I have anything embarrassing on them, please feel free to share whatever you find with public sources of shared information including, but not limited to WikiLeaks, Pastebin, and an open Dropbox folder.” Or, for the even shorter translation that would produce the same invitation as in the above two comments: “ALL UR h4x R WEAKSAUSE!!!!@!~~~!!~~~!!!!111!!!eleven!!!”

Organizations need to have some defined style codes and talking points when representatives are speaking about security issues. Foremost among such codes and points should be an admonition to not tempt fate by declaring invulnerability to attacks or by saying there’s nothing worth finding on one’s network. Now, the people already attacking the RNC network are about to be joined by other, previously unmotivated individuals who now, out of a sense of curiosity or self-righteousness, are going to see if the RNC’s servers are indeed better secured and/or have nothing of value on them. Such information would then be shared, most likely on one of the Internets or maybe even a website, because Internets and websites are things hackers will use in their cyber.

I wrote that last sentence in jest, but it pains me to think that there are people in the RNC as well as the DNC, and a number of other organizations, that would have taken notes on that sentence if I presented it to them in a PowerPoint. I would then be asked follow up questions to clarify what is meant by “Internets”, “websites”, and “cyber.” Internet security is so much more than just looking both ways before crossing the street. It also involves not standing defiantly in the intersection while yelling “COME AT ME, BRO!” to approaching drivers.

Come to think of it, that would make another really cool slide. If you’re in the RNC or DNC, call me. I got a slide deck that will open your eyes!

This News Just in…

July 22nd, 2016

If you’re an infamous person that the government is out to get, don’t use iTunes. Or Facebook. The guy supposedly in charge of Kickass Torrents used both, his IP address for iTunes matched the IP address for editing his website’s Facebook page, and it was all over for him. He would seriously have benefited from adversary-resistant computing and adversary-resistant networking.