Skip to content

Give your adversary every opportunity to make a mistake.

This is my first maxim of Information Security. It is my keystone. We hear variations on this. An adversary only needs to be right once to get in, but then only needs to be wrong once to be discovered. APT 1 had behaviors that led FireEye to track them to Shanghai and a building tied to China's People's Third Army. Crowdstrike reviewed the DNC hack and was able to discern that two separate Russian intelligence bureaus hacked into the system, and didn't realize the other also had. Guccifer 2.0 forgot to turn on his VPN just once before going onto twitter and his location was tagged in a building in Moscow tied to an intelligence directorate. Stuxnet was traced back to the NSA, Duqu to the Israelis. The best of the best make mistakes. This leads to a corollary to my first maxim: on a long enough timeline, everyone makes a mistake.

Here's a story that was shared with me by a good friend in the industry. It is missing relevant details out of respect for my friend. Some details have been changed. The processes, trail, and TTPs are accurate. Apologies to Dick Wolf.

An adversary (henceforth identified as Beetroot) was intent to commit fraud. Beetroot would accomplish this by pretending to be an American company that would help foreign businesses get loans that would allow it to establish a presence inside the United States. The presence would help them register with the IRS and get an Employee Identification Number. Beetroot would claim to be able to facilitate the paperwork, the line of credit with an American bank, and set up contacts in the United States for the foreign business allowing them access to the lucrative American markets, for a moderate to large fee with a revenue sharing percentage over some amount of time. Beetroot claimed to be able to do this because he was a university professor with access to Masters and PhD candidates to do the work for research and credits. Beetroot would reach out to targets by utilizing Search Engine Optimization (SEO) on popular foreign search engines (Yandex or Baidu, for example).

Beetroot had been running this scam for a long time. As he didn't target American citizens or businesses, no one domestically took any notice. His fee amounts were small enough foreign governments wouldn't go through the hassle of dealing with the US State Department to attempt to apprehend Beetroot or retrieve the money. Beetroot was safe.

Beetroot would do some brand impersonation on a website. One of the brands he impersonated found out and had his site taken down. Beetroot spun up another site and impersonated someone else.

Later on Beetroot spun up another site, with a domain name very similar to the one that had previously been used against my friend. Once again, my friend's educational institution (a collegiate business school in the greater midwest) found the site, and worked to take it down. My friend came to me and asked me to take a look at what he had. We worked at different shops but were both contracting through the same firm so NDAs were easy to handle.

### I reread that NDA 4 times before hitting publish. This births a new maxim. Do not mess with an NDA.

Beetroot used servers in Eastern Europe. Beetroot used privacy guard. Beetroot used publicly available information from any search engine to do the impersonation. Beetroot had no digital footprint of any kind in the US. There wasn't much to go on. Except Beetroot went back to the well and impersonated the same school twice (mistake #1).

This time Beetroot's tradecraft was nearly flawless. But, since the attack was virtually identical in every way (what he did, how he did it, who he targeted, where the targets lived) one could say with moderate confidence it was the same adversary. So the focus of the investigation was the original impersonation website.

Both websites were a variation on the school's URL acronym, but at .com instead of .edu (many schools, even business schools, don't register the .com - poor brand defense). But, on the original one, Beetroot made one hiccup. At one point he switched registrars. Maybe it was due to being cheap, maybe he had a deal, maybe he liked the local geolocation better. But the day he switched, he forgot to check the box for whois privacy. (mistake #2). And for one day, the full whois record was listed, and passive DNS captured it in perpetuity. There was no name, but there was an e-mail address, and a street address. Tied to the registration date, we had behaviors tied to an indicator - we had pivot points. The e-mail address turned up three more websites that were impersonating Australian and New Zealand schools that had business and law departments specializing in South Pacific maritime law, offering to (for a fee) set up businesses in regional countries to deal with shipping laws. Same scam, different business model (mistake #3).

The street address was diamond studded 24 carat platinum plated solid gold. Over 40 websites with 15 different e-mail addresses tied to that address. All 40 sites were hosted on one of three different Middle Eastern bulletproof hosts. At each host, all the sites lived on a /30 subnet. Every single site used the same web server. The web server differences were tied to versions, and the versions tracked to when the sites were spun up. There were more sites on those subnets, and they led to a few more e-mail addresses, which led to a few more sites (mistakes 4 -1329542). These took the timeline to a point, when Beetroot figured to privacy guard everything. There were tons of pivot points to investigate, spoofing tons of other schools in English speaking countries.

That wasn't all. Looking at the original site that spawned the original investigation, there was one line of text that stood out. It looks like a sentence was run through Google Translate into another language and back into English. The original line wasn't hard to guess, and when run through translate into Russian and back into English it produced the distinctive sentence. We ran a Google search on that sentence. We got three hits. One website didn't exist anymore. The other two did. And they were near carbon copies of the original website my friend originally investigated. Those two were privacy guarded. And they had the same web server, same web structure, and operated on a subnet that tied to an early DNS record for the original imposter site (mistake X). But the defunct website was a diamond the size of a softball.

The original site was <university acronym>.<general university-biz word dash LLP>.com. It contained multiple subdomains for all the business types Beetroot would spoof. Whois wasn't private, the address nearly lines up (one digit was off), and the registrant had a phone number, and it had the area code and local prefix of the city and state in the whois. Later in the whois history, Beetroot switched phone numbers to a Google Voice number, which used geolocation to give him a number with the same area code and prefix. The registration date puts this as the first site spun up. A web archive view of the site showed a very rough draft of some of the impersonating sites.

The cherry on top - Google Earth. The addresses should be tied to a lat // long scale. Beetroot's address was in the middle of nowhere. Google Earth showed an empty field of tall grass.  We went down the road in both directions, and found that the addresses on the few mailboxes didn't line up with Google Earth. So we clicked down the road to a small house in surrounded by fields for hundreds of yards. The address marker had the address of the original discovered address from whois. The small house had multiple satellite dishes (like one would have for Dish or Direct TV), which would make sense for middle of nowhere internet. And the smile on the Mona Lisa? We spun the Google Earth around, and someone had paid the money to put an internet junction box like you see in suburbs right across the street from this house in the middle of nowhere. There were still signs of a fresh trench dig and fill in from there to the direction of the highway. And a fresh strip of asphalt from it across the street to (what I assessed with High Confidence based on everything together) was Beetroot's house.

From a Threat Intel standpoint, this was unbelievable. It was the Deathly Hollows, the Lost Ark, even the alien from Area 51. We had tradecraft. We had a full timeline from start to current. We had targets. We had consistent TTPs stretching over years. And we had Beetroot's home.

We imagined that's what it felt like when the Mandiant researcher stood outside the office building in Shanghai and took that picture.

Beetroot represented something that gets zero discussion in most online Infosec circles - the Persistent Threat. We hear about Advanced Persistent Threats all the time. And we hear about script kiddies who wreak havoc with a tool. Beetroot fell in the middle. Beetroot probably started out as one person, and then worked with others to make his scam work. Beetroot's skills improved with time. But Beetroot never wiped his slate clean. As his tradecraft got better, he didn't clean up his previous footprints.

Persistent threats have greater initial technical debt, and much more limited resources. They need to build on previous successes with very limited budgets. Their advantage is it's harder to defend than attack, and Beetroot wasn't attacking anyone who had the means to fight back. But the work wasn't lucrative enough to throw away his old infrastructure, and then he likely forgot about it. He diversified, but not enough. He (like most adversaries) had consistent TTPs across his fraud. Lone indicators were a starting point, but the TTPs were so obvious from one to the next.

We think of the near impossibility of finding APTs without multiple dedicated staff assigned to each Infosec function. And how would one train to challenge such an adversary? Lots of businesses will fall into the targeting reticule of one of the many APTs. But for each of the APTs, there are dozens of persistent threats coming after your networks, with tradecraft not as good. You can use these to show successes to leadership. You can use these to sharpen your skills. And you can use the learning experience to better position yourself to catch the advanced threats, who will also make mistakes.

Give your adversary every opportunity to make a mistake. They will. And you will catch them.


Who are you?

That one question defines so much of you. Thinking about the question defines you. Specifically, how you think about that question. In Infosec you have to be analytical. Whether you work or desire to work at a strategic (leadership), operational (cooperative), or tactical (technical) level, the ability to ask the right questions, and analyze questions asked is part of the job. What are you trying to find out? What will that information get you? Why is getting that information important? What does the person asking the question want to know? What do they need to know? Are they asking for what they need? What questions will the answer you give prompt? A proper analytic question is the start of a series of multi-order effects birthed by the series of questions that spawn from the first one.

By virtue of reading this blog, I'd bet money you have created a profile on at least one social media site, even if it was for a short time. If you haven't, you've at least read one profile on social media. The odds that neither are true are smaller then a rounding error to significant digits. Think of any profile you have read. There is a character limit. They are designed to be small blurbs, succinct, and by their very nature incomplete. And that is the problem - especially in Infosec.

A moment in time can change a life. A person's most outrageous experience in life comes down to one single moment. Every social media post, upload, and interaction is at best one moment in time. Sometimes the ones we want to show the world. Often it is one's weakness, rage, or hate, vile and unfiltered. And very disturbing, this is prevalent in Infosec. Even worse, those in Infosec are willing to judge based on one moment. What makes that an egregious sin is Infosec is supposed to be so analytical. A moment in time is an indicator. And an indicator without adversarial TTPs only shows what happened right at that moment. If that. Investigators who claim to be purely analytical when dealing with a digital indicator will then judge someone worthy of damnation (or termination from whatever job they have) based on an indicator. And based on a truly perverted sense of absolutist justice.

One of the great moments in the movie High Fidelity is when John Cusack explains why Joan Cusack came into his shop and referred to him in a very unkind fashion. He then explains four pieces of information his ex-girlfriend most likely shared with Joan that painted him in a very unflattering light. He then explains to the audience that each of these four horrible things was absolutely true. He then goes on to rationalize (minimize) these behaviors. Knowing full well that the audience is judging his character, he looks into the camera and gives the audience a pop quiz. Think of the top five all time worst things you've done to your mate that they don't know about. There is a pause, giving the audience time to think. Then he gives the line of the movie: now who's the fucking asshole.

Infosec rationalizes it's bad behavior under the justification that people don't understand the fight we had to get where we are. There is no easy in to this part of technology. We see evil intent and behavior as part of our job, so in comparison our snap judgements, our condemnations, our willingness to hurt (trying to take someone's job away so they can't eat, have shelter, have transportation is a most cruel hurt) shouldn't be held against us - we fight the bad guys. We see a moment in time, and depending on who the perceived slight would hurt judgement is hurled. Ends (vanquishing evil) justifies the means (inflicting harm).

Except we're looking at one point in time. Infosec people would make a very bad juror. Think back to a judgement, whether hurled in a tweet, said behind someone's back, or used to cause harm. Think of the worst, or the most recent. To quote Cusack, now who's the fucking asshole?

I am fortunate. Whether it's my path, age, having lived life ever on the outside, or likely a combination of the above, I focus on my bias more and more often. I focus on the source of that bias. I focus on how it affects my life. I focus on how it will be viewed by others. My most reoccurring maxim is Words Matter, and that is continually apropos, moment by moment.  My words reflect my bias.

I was taught by individuals, by collective groups, and by my state government that, on the basis of my demographic, I was disposable, and that the world was justified in disposing me based on actions of others long dead, or with more resources and power than I will ever have. Therefore, those who cling to victimhood, as if they were special, or that the history of their identity group should grant them favor or recompense, I identify as weak and untrustworthy. Bias.

I have always been on the outside of whatever large groups I wished to belong to. I have seen and experienced the injustice of the mob. I have experienced those in power applying different rules to me than the group because I wasn't part of the group. I see larger groups that won't police themselves as corrupt and incapable of being a voice to justice. People don't ask forgiveness because they are sorry, they ask forgiveness to avoid punishment. Bias.

Like Colm Meaney's character Gene in Layer Cake, I'm too loyal for my own good. Very often I've held up my end of a deal based on a promise - real or strongly implied - that the other side never had any real basis to honor. A former boss told me that in ten years of reference checks, my former managers gave the exact same weakness, when asked about mine. When he's part of a project or a team and people aren't holding up their end, he won't let it fail. He puts on boots and a cape and saves the day, every single time. That makes him reliable, and difficult to work with. People will abuse my ethic. People will find a way to betray. On a long enough timeline, people will show they can't be trusted. I discard people who betray my trust with great ease. Bias.

Depending on how you read that, your bias shows. Do you see someone who has overcome adversity, understands his responsibility in life to himself and others, and works to keep the team from failing and to preserve earned trust? Or, do you see an angry man who never fit in and won't give people a chance? That's your bias. No matter which you choose, judgement based on three paragraphs shows bias. And if you say you didn't, you're either Detective Columbo or a liar. And Columbo is dead.

And that's the point. Bias seeps into everything. It colors your judgement. I have taken seemingly extreme actions in some factions of life lately. They weren't based on a single indicator, but people's TTPs (pattern of behavior). I've paid a price for it. That price will collect a reoccurring fee of opportunities and allies lost for a long time. Those choices were made for the right reasons, even if the outcomes attempt to reinforce my biases.

So who are you? You are far more than a profile or post. You need to understand you. Understand as much of you as you can define, as you can put into thought. Once you can do that, you can start to view that from the other side of the looking glass with Alice. Analyze. Like a good investigator. Like a good communicator. Like a good researcher. Once you've identified your bias, you can work to overcome it. Like a good human being.

Both an infinite collection of moments in time, and their sum total. That's who you are.


My old boss had one iron clad rule when reporting on an alert or incident. Don't think, know. What he meant by this is the need in any investigation to be sure. He ran security at a very large financial organization before joining the institution where he and I met. He had to face breach notices, legal summons, and visits from at least one three letter agency. And in all these dealings, he understood the difference between 0% and 99% understanding was minuscule compared to the difference between 99% and 100% sure.

100% sure is obvious. There is proof. There is evidence. There are logs. All of these combine to paint a complete picture. They leave no doubt, much less reasonable doubt.

99% sure is where the problems occur. Your odds are so overwhelming that you have virtual certainty you are correct. 1% is a rounding error, or a margin of error.

The truth is that 1% is an error. Employees being termed, adversaries being arrested, even APTs live and die off that one tiny percent. Believe me, when the lawyers get involved, that 1% can save someone from legal action or keep them out of jail. A majority of the time that 99% will bury far more than 99% of your adversaries. The ones who can navigate that 1% are the ones you should really be worried about.

Enter 'Don't Think, Know.'

We see a system beaconing out to an IP listed in a threat intel report as being part of APT 29's infrastructure, ergo the Russians hacked us. What process spawned the call? What spawned the process? Is the IP a compromised public server the APT used to piggyback as a watering hole attack, and the system is making a normal call to the box? Was an engineer playing with a sample and triggered the call? Has the alert been verified with the source? How recent is the intel? Did EDR flag on anything? Did EPP block the rest of the process? Did the firewall stop the dropper's download? Sweeping declaratory statements are made at the end of an investigative process, not the beginning. In threat intel, an indicator by itself is a starting point at best. The behavior and the chain of events that spawn from that indicator's investigation determine fact. The desire to be right, to fight the good fight and take down the bad guys can cloud the search for fact. One can think they are right. If one isn't 100% sure, they may not clearly see that difference between 1% and 99%.

Some times it's easy. Someone leaves a digital footprint that only they could leave. Someone makes a blatantly sexist or racist remark in a print medium. Don't assume this is common. And, most important of all, do not project your bias onto it. This leads one to disregard evidence that can contradict their thesis.

Accusations have a human cost. People so easily point fingers. This is due to our thirst for answers, and the need for closure to an event. And our desire for retribution. Just look at any twitter mob. If you follow a large enough chunk of Infosec twitter you will see these far too often and they will include people who are incident responders and investigators who should know better.

An accusation is an indicator. Investigators need to take every accusation seriously. But an accusation isn't fact, it's a starting point. When an accusation is leveled that someone has committed fraud, embezzlement, theft, or worse, that accusation needs to be taken seriously. The voice making the accusation can lend a great degree of credibility to it, but by itself is not indisputable proof of wrongdoing.

Less common in Infosec (I hope) but prevalent in the real world (too often) is the ending of an incomplete investigation with a declaratory statement claiming nothing was wrong. No malfeasance happened. At some point in an investigation, it will get hard. An investigator will have to dig in deep and wade through logs. This isn't a quick process. It shouldn't be rushed. Conclusions shouldn't be rushed. Behavior needs to be analyzed. The blank spaces have to be filled in.

When you don't know, look for a way to find out. If it is impossible to find out (e.g. logs rotate) an investigator needs to state where the holes in that part of the investigation are. The investigator needs to find a way to corroborate the behavior, not let assumption become fact and move on. When that is not possible, take a queue from Colin Powell.

What do you know?

What don't you know?

What do you think?


These questions answered as honestly and completely as possible, are what it takes to shrink down that 1% to as small a number as possible.

Any ethical investigator needs to be mindful of the human cost of their work. To do that, they need to be as thorough as possible. Their behavior comes down to one simple credo.

Don't think, know.