Most companies have external facing web interfaces – a web site that people can access via the Internet. This article is meant to explain the change management process of a simple external facing web server infrastructure for a small-to-mid-size business environment. The process is scalable up or down as the
need may be.
The design involves a highly available web server architecture as seen below. There are three environments in this design: Development, Test and Production. There is often overlap in terminology between the Dev and Test environments and this can cause confusion. The Test environment is meant to test the entire deployment process. To that end, software developers do not work on code directly in the Test environment.
Note: The Test environment is NOT a development environment.
If a deployment doesn’t work in the Test environment, it is rolled back just as it would be in the Production environment and the software developers must determine the cause for the failed push before deploying again. The Test environment is as close a replica as possible to the Production environment. There is obviously more flexibility in the Test environment than in the Production environment, but it should be understood that the Test environment is a critical and sensitive part of the change management process.
This is a step-by-step breakdown of the change management process for a highly available external facing web server infrastructure. It can be adapted and scaled up or down as needed.
1. Developers develop code
a. Developers may need Windows Server on their workstation if they do not have a Development server environment.
2. Developers check code into the Change Management System (CMS)
3. Developers push code to the Test environment
a. This tests the push process and allows developers to fully document the necessary deployment procedures for a push to the Production environment
4. Once code is tested and approved, a Change Request (CR) is created and submitted
a. The CR contains detailed deployment and rollback instruction
b. The CR contains the primary developer point-of-contact (PoC) for the deployment
c. The developer PoC is responsible for testing the change deployed to the Production environment
d. The CR is reviewed and approved by the Change Control Board (CCB)
i. Most mid-to-large-size businesses have a CCB or similar entity for this process
5. Change is deployed
a. Admins remove one server from cluster
b. Admins deploy the change per instructions in the CR
c. The developer PoC tests the change and approves or rejects it
i. If the change is approved, add server back to cluster
1. Repeat for second web server
ii. If the change is rejected, the necessity of a rollback must be determined
1. If a change must be rolled back, a post mortum and root cause analysis must be performed
This process requires removal of a web server from cluster for testing. This protects the customer from a bad web experience in the case of a failed deployment. It also protects the company from possible down time due to a failed deployment. Not all deployments will go as planned, and not all rollbacks will go as planned. In the event that a deployment fails AND a rollback process fails, both the customer and the business are protected from the potential down time and fallout it would cause. At least one web server will be fully operational during this entire process with the second server to be restored to full operation as soon as possible.
Since a web server is removed from the cluster for testing purposes, a couple considerations will have to be addressed.
- The web site on the server will have to be tested using the server’s dedicated IP address, not the cluster address. That means IIS will have to be configured to listen on the server’s dedicated address as well as its NLB address if you are using Microsoft Network Load Balancing (NLB). If a separate device is being used for load balancing the web traffic, IIS should already be configured to listen on the server’s dedicated IP address; make sure network traffic can get to the server’s dedicated IP address through any necessary network devices.
- The developer PoC may have to modify their hosts file on their local machine to get to the web site on the server’s dedicated IP address. A script can be used for this purpose and is perhaps the simplest option. It is not recommended to modify DNS for this purpose, although that is also an option. When using the server’s DNS name to reach the web site on the server’s dedicated IP address, IIS will have to be configured to listen for the server’s DNS name as a host header entry. This doesn’t work well for servers hosting several web sites on the same IP address.
- Removing each server from cluster precludes the use of shared storage for web site code. This means a DFS share should not be used for hosting the web site code. In the event that a deployment fails, it will fail on both servers since the code is in a shared repository. This could mean bad customer experience, down time, loss of money, etc. The process is designed to specifically avoid this scenario, so a shared repository for storing web code is out of the question.
- Server permissions should be configured appropriately in each environment. I will link to an article on server environments here when I get it posted.
The change management process is a critical part of any business practice. The process involves communication and buy-in from the necessary departments and personnel. At a minimum, this means the Infrastructure and Software Development departments must communicate regarding deployments. This relationship is critical to the shared goals of providing an excellent customer experience and of business continuity and growth.