Just in time for my previous post...

...how your network can be compromised by your software supplier: https://www.bleepingcomputer.com/news/security/ccleaner-compromised-to-distribute-malware-for-almost-a-month/

Hot war, cold war, cyber war

"I know not with what weapons World War III

will be fought, but World War IV will

be fought with sticks and stones."

Albert Einstein

With the relations between Russia and US having deteriorated to the point where both countries are evicting each others diplomatic missions, one might ask, what's next?

The hot war between the two nuclear powers is probably - hopefully - impossible.

Too much risk for conventional weapons exchange degenerating into nuclear exchange.

The cold war is something that US can pursue - but not Russia, which has very few, if any, allies.

There is another option left to Russia - a cyberwar.

We already had a preview of what's possible during Presidential elections of 2016, where just news manipulation on social networks helped bring Trump to power, dealing an enormous damage to US government.

Internet is incredibly important part of the fabric of the modern society, and abusing it can have a devastating effect.

For example, tampering with the stock market data can lead to market collapse, and if done carefully, can result in a long term negative economic impact by turning a bull market into a bear one.

Even something as innocuous as traffic data can create chaos around major cities and non-trivial losses of productivity, not to mention shutting down major infrastructural pieces such as Google Search or Exchange Online.

And we did not even begin discussing banking, credit cards, social security numbers. If you think, like I do, that the breach of Equifax was bad enough, imagine simultaneous compromise of all credit rating agencies, banks, and credit card companies.

Society can cope with breaches of security in a single organization, but what if all or most credit cards become dysfunctional at once? The economy will grind to a halt in hours, not days, and it will take weeks to repair the damage - by which the impact on earnings, stock prices, and commerce will be very material.

What if all large credit ratings start spewing random numbers in place of real data? The lending would seize, and we know exactly what this means for the economy.

If Russia were to embark on a cyberwar against the US, it would have options that are unavailable to a regular hacker, options to which our cyberdefenses are ill prepared.

Scale

With many billions of dollars available, a national government has time and resources necessary to execute a cyberattack on a devastatingly large scale.

It can destroy many infrastructural systems - Google, Microsoft, Facebook, banks, credit rating agencies, market makers - at once, laying low until a necessary coverage of penetrated organizations is achieved.

It takes a long time to recover from a cyberattack, but it will take a nonlinearly large amount of time to do so in case of multiple of these covering the entire industries, because of availability of qualified personnel and the adverse network effects and interdependencies between multiple different installations.

On the other hand, the attackers enjoy the positive network effect when they compromise more and more organizations.

For example, the entire supply chain can be attacked - even if your own organization has implemented all the security controls and is "secure", what about the other code that you run - the OS, the patches, the device drivers, other software?

Are you as confident in the release process at the Chinese company that produced the chipset driver for your servers as you are in your own release process? Exploiting them may open the door to exploiting you.

Coercion

A national government has - compared to individual hackers and criminal organizations - an enhanced power to coerce employees into subversive action on its behalf.

It can appeal to sense of patriotism, but it can also threaten an action against one's family ("put this USB keychain in a server, or your grandmother dies"). Obviously, extended families of people from former Soviet Union countries are the easiest to threaten, but this does not necessarily apply to only Russians - executing a hit inside the United States is probably well within the capabilities of Putin's secret services, to say nothing of China, India, and other countries.

With more than half of developers having international roots in many teams, the susceptibility to coercion via blackmail is a real threat.

With FISMA/FedRAMP stipulation of no malicious activity in production without collusion, organizations that are compliant with these standards are somewhat - but not necessarily fully - protected.

For example, if this requirement is "solved" by "separation of duties", meaning that a developer cannot access the production, and the operator does not know how to compromise the product, the organization is engaging in security theater, not real security. The developer can slip the exploit into the source code, and an operator can slip the exploit directly into production.

But even if this requirement is implemented really well, what about collusion? Who is to say that Russian government cannot compromise multiple people in the organization?

Deniability

Prosecution of computer crimes relies on cooperation of international law enforcement authorities. A government player can stymie investigation by denying access to people investigating the breach.

In case of a major military power, this can be further enhanced by application of said military power - for example, an apartment complex in Syria or Ukraine used in perpetrating a cyberattack can be "accidentally" destroyed by bombing.

Can US retaliate?

Of course, and it's reasonable to expect that it will. However, the reliance on the Internet is much weaker in Russian society than it is in the Western world, and reliance on the resources beyond Russian borders is much weaker. So the impact on the Russian society will not be nearly as strong as the impact of Russian attack on the US.

So where do we go from here? I don't know if there is a good path, but the organization with major online assets should first and foremost plan for an attack from a national government rather than individual hackers.

US government would also do well to take another look at FISMA and FedRAMP frameworks and strengthen the requirements for cyberdefenses. And it should mandate the compliance to a broader set of companies - those whose infrastructure is essential for functioning of American society, rather than the governments and government contractors.

Private organizations should be asking themselves questions about their susceptibility to human and supply chain compromise.

Are you confident in all software that runs on your desktops and in the datacenters?

Where did this build of Python/Perl/Node.JS come from? What about the OS? Compilers? Device drivers? Tools and utilities used by the devs? (FAR, anyone?) Who compiled them? Was there a security review? If there was, were people qualified and thorough, and able to defeat really complex exploits that may have originated from the cyberwar industry's best minds?

Is your source code as well secured as your operations? Are code reviews mandatory? How many people need to sign off on a code review before the change is committed? Is there a possibility of collusion? Do you add a random set of people who have to review the code in addition to the normal owners of the code? Are these people, again, qualified to spot non-trivial exploits slipped in by a sophisticated enemy?

What about the source control itself? Who has access? Can people with admin access make unreviewed check-ins? The system that tracks code reviews - does it have a group of developers or operators who can make untraced check-ins?

What about the build lab? Are there people with administrative access to binaries who can slip in a virus? The release share - who has admin access to these servers?

In operations, do you use an authentication scheme vulnerable to pass the hash attacks using tools like Mimikatz, or are user names/passwords generated per access? Can people log on to a server in production without a witness? Is only one witness required, and can they collude?

Is access to your production system based on a trust provider? Who is in charge of this trust provider? Is there a developer or ops team which can enable access for themselves in an untraceable way?

Do you have a break-glass scenario for access to production? Can it be used to obtain untraced access?

How about the hardware ecosystems? Who has access to the network? Can they make changes in an untraceable way? Who has the access to core routers/PDUs secured? Do vendors have access? Are passwords rotated from the factory defaults? Are you confident in the hardware infrastructure vendors process for releasing the firmware and tools that are used to diagnose the hardware?

What tracing/logging/security monitoring are you using? Is it on- or off-box? How hard is it to interfere with trace collection/security monitoring? Can a person who designed it fool the system? What about the team who owns security monitoring - can they access production without a trace?

And once the attack is ongoing - does your organization have a mechanism to stop it? Is it possible to clear the assets held by the attacker? Or is reimaging the production - and thus losing your customer's data (and most likely then, the customers) - the only option?

This is just a small subset of questions that a competent security review should examine. Do you have answers for these in your organization? The answer is almost certainly "no", and though this may be OK when confronting a small-budget hacker or a criminal organization, the system is almost certainly fall when attacked by adversary with a multibillion dollar cyberwar budget.

We live in interesting times...

Would you like to work on one of the world's biggest distributed storage platforms?

“What are the important problems of your field? What important problems are you working on? If what you are doing is not important, and if you don't think it is going to lead to something important, why are you working on it?”

   - Richard Hamming

We are a team in Azure Compute responsible for utilizing exabytes of storage in Microsoft datacenters. Our software aggregates unused disk space and makes it available to customers both as block storage - we have created Petabyte-sized disks in the past - as well as relational storage available through a SQL interface.

Our technology is unique because it works at every level of software stack - from the deepest level of kernel storage drivers, to massively replicated, PAXOS-based system that coordinates storage allocation.

Our team subscribes to all modern paradigms of software development - we roll out to production every week, we code-review every change, we design high-scale, loosely coupled systems with fail-safe mechanisms built in, we do copious amount of production debugging, testing and monitoring, and we collect the data before writing code. And we have been doing it for the last five years.

We are looking for senior and principal engineers who can help us take the system to the next level: going from single exabytes to tens and eventually hundreds of exabytes of storage. You need to have a non-trivial, multi-release, deep experience (at least five years) with either massively distributed systems, or operating systems kernel and driver development. Experience with the Windows Storage stack is preferred, but not required. You should be a master coder in C++ (on a scale from one to ten where Stroustrup is eight, we expect you to be no less than five). You should be able to think on your feet, produce simple solutions, implement them quickly and efficiently, and guide them to success in production deployments.

Why should you work on our team?

Our technology is one of the top two or three most advanced systems in its field on the planet. If you love storage and distributed systems, this is the system to work on. If you have expertise in operating systems development, and would like to expand into distributed systems, or vice versa, this is a great opportunity to capitalize on your existing expertise while learning the other universe.

We have the resources of a huge, powerful company behind it, but none of the bureaucratic overhead that is often associated with it. We are at the tip of tens of billions of dollars the company is investing in software services in general, and Azure in particular.

You will work with brilliant people on a project that directly impacts thousands of developers, and indirectly impacts hundreds of millions of customers. You will learn new things, and share your knowledge with us.

Interested? Send your resume to sergeyso@microsoft.com and we will be happy to talk more!