This is a continuation of my previous post. See the introduction at the top of the previous post to get more information on what this series of posts is about.
This post focuses on devops practices I adopted to secure my web server.
If you have a server exposed to the Internet, use a firewall (or security group in the case of AWS) to lock access to the server as much as possible, limiting the ports and IP address ranges that can be used to access it. This drastically reduces the risk surface area. Even if you do nothing else, do this.
Separate, separate, separate. Even if things get compromised up to a point, that shouldn't compromise everything. Keep in mind the concept of defense in depth as you think of things.
Related to the above: Give only as much permission to each thing as it actually needs; this idea is called the principle of least privilege and you may also hear of it as "right-sizing permissions" in some contexts.
Security isn't just about preventing or limiting attacks, it's also about being able to recover more quickly and get back to a clean slate. Backups with streamlined restore procedures, that I covered extensively in part 1, help a bunch with this.
Make sure you have good security (strong password, two-factor authentication, etc.) on the online accounts that you use to manage your servers. A secure server managed through an insecure online account is like a locked safe that's wide open from above.
Some time after I started drafting this post, I discovered a great podcast called Darknet Diaries that covers a lot of security-related themes. Many of the concepts I talk about in this post show up organically in the episodes of this podcast, and I recommend this podcast for people interested in security.
Some aspects of security are about preventing information from leaking out. Other forms of security are about preventing others from being able to modify the behavior of the server or impersonate the server to modify the expected behavior for others (e.g., serving their own website in place of mine).
For a web server, the latter kind of security is more relevant, but the former also matters since in at least some cases, the source code of a public website is not public (and may contain information that shouldn't be public).
I conceptualize security as being about layers of defense against attackers, both intentional attackers and accidental intruders. To my mind, security is about being able to make strong statements of the form "Even if A is broken/misconfigured/compromised, B will still not be compromised" where A and B are "near" each other in some sense. So, security isn't just about having a strong outer defense, but also about inner decoupling to the extent possible.
I conceptualized this before I learned the formal jargon of defense in depth, but upon learning of that jargon I see that my concept is closely related. Specifically, defense in depth is the idea of having multiple enclosures or layers of defense, so that even if outer layers are breached, inner layers are still intact. My concept of "security conditional to partial compromise" is a local version of this general idea.
When a security issue occurs, it is best if the issue is self-containing, easy to diagnose, and easy to remediate with minimal disruption. A key aspect of this is separation; even functionally similar things should be as separate as possible so that the responsible party for an issue can be quickly identified, the issue remains limited in scope, and it's easy to fix the one thing causing issues without affecting other things. A few ways this manifests itself:
For different servers (even if they're running similar code), I always put different credentials for access to third-party services. That way, if one server gets compromised, the credentials on that server can be disabled without affecting the functionality of other servers.
Within the same server, I use different username/password combinations for access to different websites. This is the case even though the same web user runs the code to access all these websites, so that user has access to all the username/password combinations in principle. However, the portion of code that gets compromised could still be more local and specific to a particular site, so it's good if the damage that such a compromise could achieve is limited.
The purpose of this section is to lay out a general framework that helps motivate the coming few sections where I talk about specific aspects of security.
Before getting into the details, here's an effort to step back and use the general philosophy outlined above to figure out what the next few sections should be.
The outermost starting point seems to be the ports through which users can connect to the server, so control over ports (aka firewalls / security groups) is the outermost security layer to begin with. This motivates the practices I describe in the "Firewall / security group" section below.
With the firewall set up, the only way to access the server is through the front door i.e., the web processes. So the next layer of defense is to limit the scope of the web process. This is two-fold: make sure the web process is running on a user different from the main user, and limit the file and database permissions that the web user can access. This motivates the practices I describe in the "Separate users for web processes, and limited permissions for these users" section below.
With the web process constrained, there is now a limited range of damage possible, but there is still some damage possible. We're now in the stage where code is being run, so the next step is:
Secure the code I've written as much as possible; this motivates the practices I describe in the "Robust codebase for websites that is harder to attack" section below.
Update software that I use directly for web serving (MediaWiki and WordPress); this motivates the practices I describe in the "Management of updates for third-party software" section below.
I started by saying that the outermost starting point is the ports through which the user hits the web server. But even before that, there is a process by which the name of the website resolves to the IP address. This process has two layers:
Securing these basically means securing the online logins through which these are controlled. In particular, I have strong passwords for both cases and also have 2-factor authentication for both. The online accounts are also important as they control access to the servers themselves, and in particular may be powerful enough to destroy or reset settings on the servers.
Servers often have a bunch of supporting functions that may not directly be part of the web request flow, but are still important. These includes:
The operating system and software that are necessary to keep things running. Vulnerabilities in the operating system or the core software could compromise the server, even vulnerabilities that seem completely unrelated to the code or content of the websites you're serving. Some high-profile examples of major bugs: Heartbleed and Shellshock.
This makes it particularly important that the operating system and core software is updated regularly. The update process should ideally be automated; however, at minimum it should be highly streamlined and checked on periodically. This motivates the practices I describe in the "Operating system updates" subsection below.
In some cases, private credentials may be needed on the server to connect to external services. For instance, my web server needs to connect with Amazon S3 to copy backup files to S3. It also need to connect with GitHub to clone repositories and fetch updates. These aren't part of a web request flow; these happen either periodically or as needed by me. Nonetheless, having these credentials in an open, readable location, or having overly generous permissions for these credentials, can increase the damage from a security breach. This motivates the practices I describe in the "Unique credentials for external services with limited permissions" section below.
This is relevant both for security against leakage and for security against manipulation.
Here, in the language of "Even if A is broken/misconfigured/compromised, B will still not be compromised", we have:
A firewall is an outer layer of defense on a server that gates incoming and/or outgoing traffic based on conditions. The conditions are generally specified in terms of the port on the server that is being communicated with, and the IP range of outside machines that are allowed or not allowed to communicate.
Amazon Web Services (AWS) uses the term "security group" for the same idea, so if you mainly use AWS you should be on the lookout for security groups.
A port is a virtual endpoint on a server, assigned a numeric label, at which it sends and receives messages from or to other servers. The numeric labels of ports range from 0 to 65,535. A few specific numeric values of ports have particular significance: for instance, 22 is used for SSH, 80 for plain HTTP, and 443 for HTTPS. The bulk of the numbers are not assigned any meaning, though there are conventions governing typical usage pattern (for instance, 9000 is often used for local applications such as PHP, 3306 is often used for MySQL or similar applications). Here's Wikipedia on ports, and here's CloudFlare.
A port-based firewall rule specifies, for a given port, what IP addresses are allowed and not allowed to access that port. There's a language to express this concisely (called CIDR blocks). This language supports both specifying an individual IP address and specifying "all IP addresses" at once, as well as a bunch of stuff in between.
Although not super-easy, IP address spoofing is possible, i.e., an attacker may be able to, in some cases, pretend to be coming from an IP address that's different from the attacker's actual IP address. Conversely, it's also possible that the IP address of a legitimate entity changes. So, when thinking about IP ranges, keep in mind:
Use the IP address only as an additional/secondary form of verification and not as a primary method of authentication.
Be prepared to have to log in and edit the firewall when IP ranges change.
The main way I expect people to use my server is for web serving. So outsiders should only need to access the ports needed to access webpages, which is 80 for HTTP and 443 for HTTPS. So to a first approximation, I want to allow access for everybody to ports 80 and 443, and allow access for nobody to other ports.
This is the case even though there are other services running on other ports! For instance, MySQL may be running on port 3306. But outsiders don't need to directly connect to MySQL! And they should not. The MySQL port is only intended for internal serving. The firewall is only for traffic coming from outside, not local traffic within the server, so it shouldn't disrupt the server's ability to run other services locally on other ports.
I want to be able to SSH into my web server. SSH is over port 22, so I want to leave port 22 open. But ideally, only for IP addresses I expect to be using (SSH is also gated by secret credentials, not just IP address, but having IP address as an additional mechanism helps filter out password hacking attempts). So, I add a firewall rule allowing access to port 22 from a hand-crafted list of IP addresses that are locations I use regularly. I may need to edit this list occasionally.
I like to be able to ping my server. Ping actually doesn't go through a numbered port, but instead uses a more primitive protocol called ICMP. I have a firewall rule that basically allows for free ping access.
Any traffic that doesn't match any of the criteria above gets blocked.
Although this isn't directly a security issue, I also use the firewall on occasion to block very clear IP-based patterns of spam traffic on the ports 80 (for HTTP) and 443 (for HTTPS).
It allows access for all IP addresses (except a few IP addresses known for spam traffic) to ports 80 (the HTTP port) and 443 (the HTTPS port). My servers run Nginx that listens to these ports.
It allows access to port 22 (SSH) only from a few IP addresses that I use (such as my home and office IP address). This means that if a hacker figured out my SSH username and password, that person still couldn't get on the machine.
It allows ping access from arbitrary locations.
If you're used to Amazon Web Services (AWS) you would have seen the same idea encapsulated in AWS's concept of security groups.
There are bots constantly running, checking for combinations of server and port that are open to traffic from the Internet. Many of these bots are malicious: once they find a server with an open port, they'll check if what's running behind the port is some known software (such as Redis, or PHP, or MySQL) and whether it is possible to send it instructions, or hack into it, to achieve the bot's own goals. This process is called "port scanning"; see here for instance.
Some bots just want compute to run their own crypto mining (this is harmless except to the extent that it competes with your server's resources), some bots will try to access files on your server to see if they can read secret credentials that can open up access to other stuff, and some bots will try to take over the site and serve their own spam.
Now, in principle, when starting a service on a port, you can explicitly restrict it to listen only to localhost on that port, and therefore to not respond to outsiders who send messages on that port. For instance, you can start MySQL on port 3306 to only listen to localhost connections on port 3306, or start PHP-FPM on port 9000 to only listen to localhost connections on port 9000. That way, even without a firewall, there are no unintended open ports for outsiders to use.
But making sure of this requires care with every service, and it's possible to slip up at some point (or to not have the ability to configure some software). A firewall, as an outer layer of defense, prevents any such misconfiguration from having a negative impact.
When playing around with configurations for a dev server, I accidentally started PHP-FPM on port 9000 open to the world (rather than just to localhost, as I intended). Nothing went wrong immediately, but within a few hours to a few days, nearly all the CPU on the server was being used by the Kinsing malware for some crypto mining purposes. Fortunately, this was a dev server, not identified with me, and the bot did nothing else to the server, so the impact on me was limited. But this was a cautionary tale and I immediately worked on setting up the firewall -- something I should have done to begin with!
I've heard of a couple of cases where MySQL installations that had a weak or empty root password set by default as well as having the port open to the world were predictably hacked into. In some cases this was right around the time the server was being set up (so hadn't had time to switch away from default credentials) whereas in others the server had been left idle with default credentials.
I am not alone; see this excerpt from Darknet Diaries podcast episode #88 (Victor):
JACK: Here’s the basic process; databases run on a certain port, so MySQL, for example, runs on port 3306, and so he could just scan the internet or use a website like Shodan to first look for any IPs that have port 3306 open. Then once he finds that, he’ll try default usernames and passwords or maybe a handful of very weak passwords like the word ‘password’ to see if that’s it. Yeah, from this alone, he was tripping over tons of open databases. Some didn’t even have passwords at all. Then they’d tell all the database owners that their stuff is insecure. As their efforts grew, so did the number of people pitching in to help GDI. [MUSIC] In the beginning, it was just Victor and his friends but almost fifty different volunteers have joined on [00:10:00] in the last five years. In that time, the foundation has filed coordinated vulnerability disclosures on over a million security issues out there on the internet.
Using a firewall to lock down most ports dramatically reduces the risk surface area to a few specific ports that are being served by the software you are focused on (in my case, web-serving software Nginx). You don't have to worry as much about accidentally exposing other ports to strangers.
Further, for ports that need to be accessible only to specific people (in my case, the SSH port that only needs to be accessible to me), using firewall rules to restrict the IP ranges that can access those ports, further reduces the risk surface area.
In the language of "Even if A is broken/misconfigured/compromised, B will still not be compromised", we have:
The concept of limited permissions for web users is a special case of the more general idea of the principle of least privilege.
In the previous section, we talked about using firewalls to lock down everything except the ports 80 and 443 that serve web traffic. A web server does need to have those ports open, and the service running on those ports, that serves the traffic, is a point of vulnerability. In the case of my web server, I run Nginx, proxying to other backends (PHP-FPM in most cases). An error in the configuration of Nginx, or of PHP-FPM, or of the code running any of the websites, can potentially create an opportunity for a clever end user to be able to execute arbitrary commands as the users running these processes (cf. cross-site scripting or the more general concept of code injection).
So the next line of defense is to make sure these users, even if compromised, have limited ability. Unfortunately, they do need some level of access to be able to do a good job serving the websites. The goal here is to make sure that they don't have any more access than they need. In particular, they should not have the ability to modify or write things they just need to read, and they also shouldn't need to read things that aren't directly relevant for web serving (for instance, a credential file or a Makefile). In the next few subsections I go into a little more detail on what this means.
I run the Nginx and PHP processes using users that are separate from each other and also separate from my main user.
I've already gone over why I don't run these as my main user. The reason to separate the Nginx and PHP users from each other is partly security, but also partly more ease of diagnosis of what's going on (any activity reported to a user can be traced to the respective process).
A well-known direction of attack used by real-world attackers as well as penetration testers is a directory traversal attack where the web process is tricked into accessing files and folders that are outside of the web root folder. One layer of security to defend against this kind of attack is to limit the permissions to files and folders that the user running the web process has. So even if this user gets compromised, the user can only do damage within a limited sphere.
All the automatic scripts that I've written to set up new sites include steps to make sure that the files readable by others are just the ones that are needed for the web processes. And in almost no cases do the users running the web processes have file write permissions; all writes are done through scripts run directly by me, or scheduled through cron to be run automatically -- but not invokable via the web. The exception is when there is a need to save uploaded or programmatically generated images; in such cases, the write permissions of the user running the web process are limited to the folder(s) containing images.
In addition, I've set umask settings so that any files I create
through the command line on the server will get the default of not
being readable, writable, or executable by the users running the Nginx
or PHP processes. Note that this means that if I manually fetch git
changes on the command line, the files may end up getting overly
restrictive permissions and I would need to manually fix the
permissions for the site to continue working.
The web user has access to the configuration file containing the MySQL username and password that can be used to access the databases needed to serve various sites. However, unless specifically needed for the site, this user only has read permissions for the database. Operationally, this translates to privileges to select, lock tables, show view, and create temporary tables. The user does not have the permission to insert, update, or delete. Moreover, the username/password is different for each site, so any bug affecting a specific site is limited to the database for that site.
For some sites, the end user's actions can lead to data updates in MySQL, and in these case, we do need to give at least some level of write access.
While as much as possible, it is good to limit the permissions for the web processes, there are some cases where the web processes do need write permissions (particularly on the database side) to execute legitimate tasks. We want to prevent end users of websites from being able to send the web process any instructions or commands outside of the very specific things we want to constrain them to. For instance, we don't want them to be able to run arbitrary SQL queries, or arbitrary shell commands.
One common form of attack is a SQL injection; this generally applies to SQL queries that include some variables inside them that are populated based on inputs that can be influenced through the web. For instance, a SQL query that is constructed based on url parameters or based on data entered into a form. A SQL injection attack may use some syntax tricks to smuggle in another query into the url parameter or form data, where this other query allows for manipulation of the data in ways different than desired.
Limiting database permissions as discussed in an earlier section, to only allow read permissions unless otherwise needed, can limit the damage here: if the user running the SQL commands only has read permissions, the user can't do much damage even if it gets SQL-injected. However, some kinds of sites require write permissions to be available to the SQL user. So having a way to query that doesn't allow for SQL injection would be nice.
Fortunately, MySQL has a concept of prepared statements whereby it does the query interpolation itself. As much as possible, I've tried to switch all my homegrown PHP sites to use prepared statements. Regarding the off-the-shelf software I use that uses MySQL database, I hope and expect that the developers of the software are constantly vigilant about the issue and stick to prepared statements.
The idea is that, for external services, such as Amazon Web Services (AWS) and GitHub, that the servers need to connect with, each web server (production or development) has its own unique credentials. Moreover, to the extent possible, the credentials are scoped to have only the power that the servers need. However, the point about uniqueness applies even across different identically powered servers: they should not share credentials.
In the language of "Even if A is broken/misconfigured/compromised, B will still not be compromised", we have:
Specifically, as soon as the credential leakage for the server is detected, that credential can be removed or changed, without needing to make updates to other servers. Therefore, the change of credentials can be executed seamlessly and with minimal disruption to other servers.
Moreover, to the extent that access logs are available, having different credentials on different servers allows us to get information on which server's credential leaked. This might allow us to more quickly get to the root of the problem.
I use third-party software such as MediaWiki and WordPress for serving websites; I use other software on the server to support this work. The codebase for these softwares is large, and often, vulnerabilities are detected in the softwares (or in extensions or plugins to the softwares). Staying up-to-date with security patches is important.
Also, a streamlined setup process makes it easier to update the underlying OS and stack more frequently, allowing the server to benefit from the improved security of newer versions.
Having out-of-date software is a liability even if it has no known security holes, because the vendor may not be publishing patches for future security holes once they are found. Out-of-date software may also lack compatibility with up-to-date versions of other software or even operating systems.
Vulnerabilities in the operating system can put the server at risk. Two major OS vulnerabilities in recent years were Heartbleed and Shellshock.
Fortunately, operating systems are generally quick to patch discovered vulnerabilities; however, if hackers discover an exploit before it's found by security researchers, then it is particularly dangerous (such exploits are called zero-days). We can't do much about zero-days, beyond having good other layers of defense such as those described in preceding sections: firewalls, limited permissions, separate credentials. However, for bugs that have already been discovered and patched, it's important to benefit from the patches by keeping the operating system up-to-date.
Different operating systems have slightly different approaches to security patches. See for instance this page describing the options around automatic updates.
One of the risks of automating all updates is that major version updates could cause compatibility issues with existing web-serving code, and we don't want to break web serving! My general philosophy around major version updates is to start a fresh server and migrate the content to that server. This is very much possible with a good set of backup/restore procedures.
WordPress is an interesting case. The standard/default setup for WordPress updates relies heavily on the web user. WordPress uses a concept called wp-cron, where any pageview of a page on a WordPress site can trigger the wp-cron, which in turn can trigger background tasks to update the core software, themes, or plugins. However. I wasn't comfortable with this for two reasons:
As mentioned, I've already limited the web user's permissions, including file permissions, so the web user is often unable to take the steps needed to update the core software, plugins, and themes.
I prefer to have the updates done at a specific time of day when I expect to be around, and to include additional validity checks.
I found that the WordPress CLI does a good job of addressing these concerns. It allows for command-line execution of checks for updates.
To make use of this, I wrote a wrapper script about the WordPress CLI. The script does something like this:
This runs once a day, at a time that I'm awake and usually near my computer. I also take daily backups of my WordPress sites which allows me to check the condition of the site before the update. The WordPress CLI offers a bunch of functionality around managing updates, but it's good to know that I have backups in case things don't work as expected.
The previous section hinted multiple times at the importance of backups and established recovery procedures. Let's now talk about these explicitly.
Backups with established recovery procedures are helpful for security, because they give us the option to discard any compromised resource and recreate it from scratch using the backups. For instance, if a server gets compromised today, I can restore from yesterday's backup.
Backups can also serve as checkpoints that help diagnose when exactly a system became compromised.
With that said, backups can also be a source of security issues if the backups themselves are not secured. See for instance, the 2022 LastPass incident where hackers got a hold of their backups. Luckily for them, the important data is fully encrypted to a zero knowledge standard, but you may not be so lucky. The importance of backup security is something I include as a consideration when looking at various aspects of backups in my backups post.
It's important to have tight security (with redundancy) for the hosting service of the web server, the service used for DNS records, the domain name registration service, and any services used for backups. If others are able to hack into the hosting service's web portal, they may be able to make changes to the web server's configuration. If others are able to hack into the domain name registration service, they may be able to steal the domain.
Such tight security might include the following:
Gaps in bill payment can lead to the loss of control over some resources. In particular, failing to renew a domain registration can be catastrophic if somebody else registers it in the meantime. While online services are generally good about sending reminders, it is also important to check these proactively.
Here are a few best practices I follow:
I pay particular attention to all email messages from the services I use for web servers, DNS records, domain name registrations, and backups. I also check my spam email at least once a day, so if any email from them reaches spam I can catch it.
Every month, I review the bills from these services, either based on what's emailed to me or by logging in.
I've set up automatic payments for all the services. Also, where possible, I try to maintain a slight positive balance in the accounts for these services. The positive balance makes sure that even if there is a glitch with the automatic payments, I am protected for a while. In addition, I review and top up the balances for the services monthly, which means that in practice the automatic payment ends up never being needed.