Empire :: Production Best Practices - WORK IN PROGRESS
This document is meant to explain what we believe are some best practices & guidelines around running Empire in a production or production-like environment. We hope to provide more instruction for this as well as tools to make this easier in the future.
- Separate Minion & Controller hosts
- Private VPC for all hosts
- Securing the API
- Access to the Empire API via VPN/specific IP
- Separate, secure database for Empire
- Router application
In general it is recommended that you have two pools of servers. Neither pool should be able to communicate with the other directly (no ssh, etc).
These are the servers where you will run the Empire daemon. We keep this separate from where you run actual applications (the minions, see below) so that if there is an issue with an application you deploy, it doesn't affect the control plane. They are also kept separate in order to provide additional security - if someone manages to break out of one of your applications, they should not be able to reach the Controllers, which at least stops them from spinning up more resources via Empire.
These hosts should sit on a private subnet in your VPC, and only expose the API via an elastic load balancer.
The minion hosts are where applications deployed via Empire will run. Again, these should have no access to the controllers.
They should sit on a private subnet, and shouldn't be exposed to the internet. When you expose an app via a domain in Empire, it will create a load balancer for that app that will handle the exposure for you. These load balancers are all part of the same security group, and have the same security group rules.
All hosts should run in a private VPC and should not be directly exposed to the internet. If you need ssh access to the hosts, we suggest either using a VPN or a set of tightly controlled & secured bastion hosts.
If you go the bastion host route, we suggest that you limit the access to those hosts - via security group rules (only allowing your office IP to connect, for example) and any other means available.
Also, it might be worth looking into something like ssh-ca for managing ssh keys, allowing you to grant access to hosts on a case by case basis without putting any single person's ssh keys on the host.
Securing the API
It's very important that you keep the Empire API secure. There are a few best practices for doing so:
Since you will be passing authentication information to the Empire API, you should ensure that the load balancer that it uses is setup to only accept connections via HTTPS
Access Token Secret
You should ensure that you set the
EMPIRE_TOKEN_SECRET environment variable to a long random string. This value is used to sign access tokens generated by the API.
It's also recommended that you set
EMPIRE_SERVER_SESSION_EXPIRATION to a reasonable value, to ensure the API tokens are periodically rotated by users by requiring them to re-authenticate. We recommend something like
Github Authentication or SAML Authentication
For GitHub, refer to the docs on configuring the GitHub authentication backend.
For SAML, refer to the docs on configuring the SAML authentication backend.
Limiting API Access by IP
A final option that can be useful is to only allow access to the Empire API loadbalancer via VPN, or from specific IP addresses (such as your office IP).
Separate, secure database for Empire API
Empire requires a postgres database in order to function. It is suggested that you use a database host specifically for this purpose, rather than sharing with other applications. This allows you to lock access down to only the Empire API.
If you plan to expose services, rather than having Empire expose them (via adding a domain) you can instead deploy a 'router' application, and expose that. Remind uses a single router app that is exposed to the internet via Empire (through an ELB) that has rules to route to other services based on the hostname.