Learn how to architect, design, and build distributed systems for the real world.

Become a Distributed Systems Professional

In this module, we will examine ways to architect, design, and build software—particularly distributed systems.. This module is designed to prepare us for unpredictable users who might be a potential threat to our softwares. Our software will be under attack from the moment it is released. It needs to stand up to flash mobs or DDoS attacks by poorly secured IoT toaster ovens. We’ll examine  a software that failed the test and find ways to ensure our software survives contact with the real world.

Distributed Systems: Building Software for the Real World

# Sensitive passwords

There’s obviously no way that somebody can interactively key in passwords every time an application server starts up. Therefore, database passwords and credentials needed to authenticate to other systems must be configured in persistent files somewhere.
As soon as a password is in a text file, it is vulnerable. Any password that grants access to a database with customer information is worth thousands of dollars to an attacker and could cost the company thousands in bad publicity or extortion.
 
These passwords must be protected with the highest level of security achievable.
At the absolute minimum, passwords to production databases should be kept separate from any other configuration files. They should especially be kept out of the installation directory for the software. For example, I’ve seen operations zip up the entire installation folder and ship it back to development for analysis during a support incident.

## Owner access to passwords

Files containing passwords should be made readable only to the owner, which should be the application user. If the application is written in a language that can execute privilege separation, then it’s reasonable to have the application read the password files before downgrading its privileges. In that case, the password files can be owned by root.
Password vaulting keeps passwords in encrypted files, which reduces the security problem to that of securing the single encryption key rather than securing multiple text files. This can assist in securing the passwords, but it is not, by itself, a complete solution. Because it’s easy to inadvertently change or overwrite file permissions, intrusion detection software such as Tripwire should be employed to monitor permissions on those vital files [$^{22}$](https://www.tripwire.com).

# AWS key management service
**AWS Key Management Service (KMS)** is useful here. With KMS, applications use API calls to acquire decryption keys. That way the encrypted data (the database passwords) don’t sit in the same storage as the decryption keys! If we use Vault, then it holds the database credentials directly in the vault.

In every case, it’s important to expunge the key from memory as soon as possible. If the application keeps the keys or passwords in memory, then memory dumps will also contain them. For UNIX systems, core files are just memory dumps of the application. An attacker that can provoke a core dump can get the passwords. It’s best to disable core dumps on production applications. For Windows systems, the infamous blue screen indicates a kernel error, with an accompanying memory dump. This dump file can be analyzed with Microsoft kernel debugging tools Depending on the configuration of the server, it can contain a copy of the entire physical memory of the machine, passwords and all.

# Security as an ongoing process

Frameworks can’t protect us from the top 10. Neither can a one-time review by our company’s AppSec team. Security is an ongoing activity. It must be part of our system’s architecture: crucial decisions about encrypted communication, encryption at rest, authentication, and authorization are all cross-cutting concerns that affect our entire system.

New attacks emerge all the time. We must have a process to discover attacks, hopefully before they are used, and remediate our system quickly. This is doubly true when we deploy technology that hasn’t been battle-hardened. New technology with new APIs will have vulnerabilities. That doesn’t mean giving up the advantages it offers. It does mean that we need to be vigilant about patching it. Make sure we can redeploy our servers on a moment’s notice.

# Wrapping up

Application security affects life and livelihood. It’s another area where we need to consider both the component-level behavior and the behavior of the system as a whole. Two secure components don’t necessarily mix to make a secure system. The most common target of value is user data, especially credit card information. Even if we don’t handle credit cards, we might not be off the hook. Industrial espionage is real and it can sometimes look harmless.

Beware of the pie crust defense. Internal APIs need to be protected with good authentication and authorization. It’s also vital to encrypt data on the wire, even inside an organization. There’s no such thing as a secure perimeter today. Bitter experience shows that breaches can be present for a long time before detection, more than enough for an attacker to devise recipes to get at that sweet user data.
Full treatment of application security is way beyond the scope of this book. The topics covered in this chapter earned their place by sitting in the intersection of software architecture, operations, and security. Consider this a starting point in a journey. Follow the trail from here into the rich and scary world of CVEs [$^{23}$](http://cve.mitre.org), CWEs [$^{24}$](https://cwe.mitre.org/index.html), and CERTs [$^{25}$](https://www.cert.org).
This finishes our slow zoom out from the physical substrate, copper, silicon, and iron oxide, all the way to systemic considerations. In the next part, we will look at the moment of truth: deployment!

Learn about sensitive passwords, protecting passwords in files, AWS key management service, and security as an ongoing process.

Configured Passwords

Sensitive passwords