TechNote (Archived)

Clustering enterprise application servers from Macromedia with Alteon's ACEdirector Switch

Alteon's ACEdirector switch is rich with capabilities. This article sorts through the vast variety of possible switch configurations and defines setup parameters best suited to the needs of most Macromedia enterprise server customers. It describes a basic procedure resulting in a server farm running ColdFusion or JRun behind an ACEdirector (AD3).

Frank DeRienzo, MBA, MCSE
Principal Technical Support Engineer
Macromedia, Inc.

This article first appeared in the ColdFusion Developer Center. It has been condensed for this TechNote to include only the necessary technical information. You may find the full version here.

The AD3 will manage session traffic through a virtual server. The AD3 virtual server will associate with each Web/application server in the farm; AD3 will monitor the health of each Web/application server in the farm and distribute session load accordingly. ClusterCATS (CC) will provide probes that proactively monitor the health of any web server: NES, IIS, or Apache. It will also monitor JRun or ColdFusion. If either the Web server or application server hangs, CC will kick the server back up and send out status alerts via E-mail. The combination of CC and AD3 presents a paradigm for Web site resiliency.

Note: If you are running ColdFusion 5, instead of clustering, you may wish to use the Server Monitoring option with AD3. ColdFusion 5 Server Monitoring provides all the server level capabilities of CC without the overhead of a cluster.

Note also: If you wish to set up your server farm in distributed mode (DM) by separating your Web servers and your application servers onto different systems, go to the Macromedia Support Center and search on the key word distributed. The articles which appear will provide procedures and recommendations for configuring a distributed mode server farm. You will also find distributed mode referenced under the Designer and Developer link on the Macromedia home page.

Configuring AD3 with CF/JR & CC

To enhance clarity and simplicity, this article is example-based. Consider the following example of a two-server cluster. Picture two server-class systems - tiger-one.macromedia.com and tiger-two.macromedia.com - configured in a passive-mode cluster using private IP addresses 192.168.64.11& 12. A third IP address, 192.168.64.1, will be used on the AD3 to provide a default gateway for the server farm. The fourth address 10.64.20.93 is for the virtual server on the AD3; this address corresponds in DNS to the name that your browsing customers will place in the URL line. It represents a public address. The fifth, 10.64.20.96, will be used for administration of the AD3.

I use private, internal addresses for the Web/application servers, because only the Alteon VIP address needs to be public. Public IP addresses tend to be rare these days.

The example list of names and IP addresses that follows is an illustration of the hosts file that you will need to place on each server in the farm. On an NT server, the hosts file resides in \winnt\system32\drivers\etc. On Solaris or Linux, it resides in /etc. Before going on to the next step, use your favorite text editor to create and place the hosts files on the appropriate servers in the farm: tiger-one and tiger-two. Do not forget to configure the default gateway on each server.

 192.168.64.11  tiger-one.macromedia.com        tiger-one 192.168.64.12  tiger-two.macromedia.com        tiger-two 192.168.64.1   webserv-gateway.macromedia.com  webserv-gateway 10.64.20.93    www.macromedia.com              www 10.64.20.96    admin-ad3.macromedia.com        admin-ad3

Note: This example is based on using Macromedia enterprise application servers ColdFusion or JRun with AD3 software version 8.0.60.7. If you need to upgrade your AD3, contact Alteon technical support. I found Alteon's binary upgrade procedure to be very simple and fast.

A.) Place the two CF/JR Web servers into a passive-mode CC cluster behind the AD3. You must use static Web site IP addresses; CC failover (high-availability) must be turned off. During CF/JR installation, select no server failover. If you are running NT 4.0, do not set up your Web servers with dynamic IP addresses. AD3 will be providing failover services. Dynamic IP addressing is only used with CC's implementation of failover on NT. If you are adding AD3 to a ColdFusion and CC cluster that is already set up with dynamic Web site IP addresses on NT 4.0, you must switch to static Web site IP addresses and disable CC failover.

To switch to static addresses, you must:
1. Right-Click on Network Neighborhood and go to Properties - Protocols - TCP/IP - Advanced. Make the Web site IP addresses static by adding to the primary NIC the addresses that were the dynamic. Do this to each clustered server.
2. Reboot the servers simultaneously.
To disable CC failover on NT or Win2k servers, you should re-install ClusterCATS choosing load balancing, but not failover. This is the most robust way to disable failover because it ensures that you are running the latest CC bits (that are compatable with the version of your specific application server - be careful not to mismatch versions) while completely disabling CC failover services. Simply install this into the cfusion\brighttiger directory (or wherever you installed CC previously), choose load balancing (not failover), and simply overwrite any older CC version. Another way to disable failover follows:
1. Stop the brighttiger and the ipcheck services: Start > Settings > Control Panel > Services > Bright Tiger Service> Stop, Bright Tiger Ipcheck > Stop.
2. Go to the brighttiger/program directory.
3. Rename ipaliasd.exe to wsm.exe.
4. Restart the brighttiger service: Start > Settings > Control Panel > Services > Bright Tiger Service > Start.

B.) Let's begin to configure the AD3, and then we will go back to the clustered server farm. This example-based article is designed as a CF/JR supplement to your AD3 documentation. Use this procedure along with your Alteon documentation incorporating the following supplemental modifications illustrated using the Web servers from our example and using an AD3 virtual server address (VIP) of 10.64.20.93:

You may log into your AD3 using Hyperterminal from a serial port by connecting to the AD3 using a DB9 male to DB9 female straight through cable (if you need to upgrade your AD3 software, the recommended binary method uses a serial connection). The initial AD3 password out of the box is admin. I begin this procedure by resetting (clearing) the AD3 in order to ensure that I am working with a clean configuration:

The Hyperterminal settings to connect to the AD3 are:
```
 Bits per second 9600 Data bits 8 Parity none Stop bits 1 Flow control none 
```
To reset your AD3, choose the boot option from the main menu.

Main# boot

Choose the conf and reset options from the boot menu.
```
 Boot Options# conf Boot Options# reset Confirm reset [y/n]: y] 
```
After the switch reboots from the reset, choose to manually configure the AD3 (remember the scope of this article is only to configure the AD3 with a basic configuration in front of a two server farm):
```
 Will you be configuring VLANs? [y/n] n Would you like to run from top again? [y/n] n 
```
The first thing to configure on the AD3 is an external administrative IP address - the address that you will use to administer the AD3 from outside of the server farm either through telnet or through a browser. Do not be confused; this is not the Web site address. For this example, I chose 10.64.20.96, registered it in DNS with the name admin-ad3.macromedia.com, then configured it on the AD3. Choose the configuration option from the main menu, the IP option from the configuration menu, and interface #1 from the IP menu as follows:
```
 Main# cfg Configuration# ip IP# if 1 
```
From the interface #1 menu, be sure to enter your subnet mask prior to entering your IP address and disable bootp as needed.
```
 IP Interface 1# mask Current subnet mask: 0.0.0.0 Enter new subnet mask: 255.255.255.0 IP Interface 1# addr Current IP address: 0.0.0.0 Enter new IP address: 10.64.20.96 Pending new broadcast address: 10.64.20.255 Switch is set to use BOOTP for IP address assignment. Do you want to DISABLE the use of BOOTP? [y/n] y Use of BOOTP disabled. 
```
Enable and apply the interface.
```
 IP Interface 1# ena Current status: disabled New status: enabled IP Interface 1# apply 
```

Next, you must configure a second interface to act as a default gateway for the server farm by backing up one directory to the IP menu and creating the private interface using address 192.168.64.1.

 IP Interface 1# .. IP# if 2 IP Interface 2# mask Current subnet mask: 0.0.0.0 Enter new subnet mask: 255.255.255.0 IP Interface 2# addr Current IP address: 0.0.0.0 Enter new IP address: 192.168.64.1 Pending new broadcast address: 192.168.64.255 IP Interface 2# ena IP Interface 2# apply

Now it is time to tell AD3 about the server farm. The CF/JR servers in the farm are referred to as real servers; configure the first real server (192.168.64.11tiger-one.macromedia.com) on the AD3.

 IP Interface 2# .. IP# /cfg/slb/real 1 Real server 1 # rip Current real server IP address: 0.0.0.0 Enter new real server IP address: 192.168.64.11 Real server 1 # ena Real server 1 # apply

Configure the second real server (192.168.64.12 tiger-two.macromedia.com).

 Real server 1 # ../real 2 Real server 2 # rip Current real server IP address: 0.0.0.0 Enter new real server IP address: 192.168.64.12 Real server 2 # ena Real server 2 # apply

Tiger-one and tiger-two are now identified as real servers one and two respectively on the AD3. If your real servers are actually plugged into the AD3 and online you can now ping them. For this example, I have tiger-one's network cable plugged into port three on the AD3, and tiger-two's cable plugged into port four. Port one is connected to my external network hub by a crossover cable. Test your configuration by pinging the real servers from the AD3.

 Real server 2 # ping 192.168.64.11 [host 192.168.70.2, max tries 5, delay 1000 msec] 192.168.64.11: #1 ok, RTT 1 msec. 192.168.64.11: #2 ok, RTT 1 msec. 192.168.64.11: #3 ok, RTT 1 msec. 192.168.64.11: #4 ok, RTT 1 msec. 192.168.64.11: #5 ok, RTT 1 msec. Ping finished. Real server 2 # ping 192.168.64.12 [host 192.168.70.4, max tries 5, delay 1000 msec] 192.168.64.12: #1 ok, RTT 0 msec. 192.168.64.12: #2 ok, RTT 1 msec. 192.168.64.12: #3 ok, RTT 1 msec. 192.168.64.12: #4 ok, RTT 1 msec. 192.168.64.12: #5 ok, RTT 1 msec. Ping finished.

Place the two real servers into a group.

 Real server 2 # ../cfg/slb/group 1 Real server group 1# add 1 Real server 1 added to real server group 1. Real server group 1# add 2 Real server 2 added to real server group 1. Real server group 1# apply

Create a virtual server on the AD3 and configure it to use port 80. This is the address that browsers will use to access your site (10.64.20.93 www.macromedia.com).

 Real server group 1# /cfg/slb/virt 1 Virtual Server 1# vip Current virtual server IP address: 0.0.0.0 Enter new virtual server IP address: 10.64.20.93 Virtual Server 1# ena Virtual Server 1# apply Virtual Server 1# service 80 Virtual Server 1 http Service# group 1 Current real server group: 1 New pending real server group: 1 Virtual Server 1 http Service# apply

If you are following this example closely, you will need to enable client processing on port 1. As I mentioned before, for this example I have connected port one on the AD3 with a hub on my external network using a crossover cable. In order to hit the Web site on your real servers from a browser on your external network, enable client processing through port 1.
```
 Virtual Server 1# /cfg/slb Layer 4# port 1 SLB port 1# client Current client processing: disabled Enter new client processing [d/e]: e SLB port 1# apply 
```

Continuing this example, enable server processing on AD3 ports three and four into which the real servers are connected using straight-through RJ45 cables running directly from the NIC of each real server in the farm.

 SLB port 1# .. Layer 4# port 3 SLB port 3# server Current server processing: disabled Enter new server processing [d/e]: e SLB port 3# .. Layer 4# port 4 SLB port 4# server Current server processing: disabled Enter new server processing [d/e]: e SLB port 4# apply

Enable server load balancing on the AD3.

 SLB port 4# /inf/slb/dump Server Load Balancing is globally turned OFF. Server Load Balancing Information# /cfg/slb/on Current status: OFF New status: ON Layer 4# apply Layer 4#  Mar 15 12:17:00 NOTICE  slb: real service 192.168.64.11:80 operational Mar 15 12:17:05 NOTICE  slb: real service 192.168.64.12:80 operational

In step three, we configured an internal interface to act as a default gateway for the server farm using address 192.168.64.1 to allow the real servers to communicate from their private net through the AD3 out to the external network of 10.64.20.X. Next, we will add an upstream default gateway (the gateway of that external net). This step is particularly important if you are staging your Web site in a lab and testing it under load - as you should always do - prior to going live.

Adding the upstream gateway to your AD3 will allow you to reach out to your desktop where you are developing those killer web applications and drag those all-important bits into that internal lab subnet where your proactive IT department has your testbed segregated. For this example I will use 10.64.20.254 as the upstream default gateway. Make sure you apply and save the new configuration.
```
 Layer 4# /cfg/ip IP# gw Enter default gateway number: (1-4)  Default gateway 1# addr Current IP address: 0.0.0.0 Enter new IP address: 10.64.20.254 Default gateway 1# apply Default gateway 1# ena Current status: disabled New status: enabled Default gateway 1# apply Default gateway 1# save Confirm saving to FLASH [y/n]: y Do you want to change that to the active config block? [y/n] y 
```
By default, the spanning tree option is turned on. You may want to use spanning tree in production, but probably not in your testing lab. For this simple configuration, I will turn it off. For a complete explanation of Alteon's spanning tree feature, see your documentation for ACEdirector Software Version 8.0. As always, be sure to apply and save your changes.
```
 Main# cfg/ Configuration# stp Spanning Tree# off Current Spanning Tree setting: ON New Spanning Tree setting: OFF Spanning Tree# save Confirm saving to FLASH [y/n]: y New config successfully saved to FLASH. 
```

C.) Now it is time to turn back to the clustered real server farm. Open the CC Explorer on tiger-one and select your cluster: Start > Programs > ClusterCATS > ClusterCATS Explorer > Right-click on a cluster and select > Properties > Load-Balance > Load-Balancing Product > Other.

Enter the name of the AD3 virtual server in the Web site Alias field. This is a fully qualified host name (FQHN) with forward and reverse DNS entries. This FQHN is the visible (to browsers, possibly PING, etc.) name of the AD3 VIP. It may correspond to what was previously the round robin name prior to implementing the AD3. This is the name that will be visible to all browsing clients; it is the name placed in the client/customer browser's URL line to get to your Web site. For our example, the VIP is www.macromedia.com.
Click OK to apply your changes.
Right-click on a server in CC explorer. Choose > Configure> State > Passive Member. The server will turn from green to white. Do this to all servers in the cluster. You may now use the CC features, such as alarms and reporting, the Web server probe and the application server probe both set to restart a stalled ColdFusion server.
1. To set up a CFProbe > right-click on a server in CC explorer and choose > New Monitor > Choose OK > Click on the blue arrow over the label probe-type > Click in the startup parameters field and arrow to the right without deleting any text> Change the word Norestart to Restart > Click on Register> Close > Close. You now have a monitor (CFProbe or JRunProbe) that is looking at a ColdFusion Web page looking for the word "Hello." If the page becomes unavailable, the probe will restart the CF/JR server.
2. To set up alarms > Right-click on the cluster in CC explorer and choose > Configure > Alarm Notification > Put in your mail server address in the SMTP host field > Type in the E-mail addresses of anyone you want alerted of the various events.
3. To setup support mail > right-click on the cluster in CC explorer and choose > configure > support > put in your mail server address in the SMTP gateway field > type in the e-mail addresses of anyone you want to have receive daily reports.

D.) If you are running ColdFusion, configure the cfprobe.cfm to record application awareness. Edit cfprobe.cfm in the cfusion\brighttiger\btauxdir using a simple text editor such as Notepad in such a way that ColdFusion is used to provide output. When properly configured using cfprobe.cfm as the target page, AD3 will detect failure of an application server. In all tests, the AD3 keepalive uri was pointed at cfprobe.cfm or jrunprobe.jsp along with the CC probe.

Note: If you are running JRun, there is no need to edit the jrunprobe.jsp page; it is already application aware.

 <CFSET AITCH = "H"><CFOUTPUT> #AITCH#ello world</CFOUTPUT>

E.) Now that your cluster is up and running, you are ready to configure the AD3 for CF/JR content and application awareness health checking. This will enable the AD3 to properly recognize and react to any disruption of service in the real server farm. This is a critical step especially if you are running your server farm in distributed mode with the Web servers set up on separate platforms from your application servers. It is not hard to picture your AD3 pouring sessions onto a Web server that has a stalled backend application server. What would tell the AD3 that the application server behind the Web server was down?

Health checking is the answer. Health check scripts are probably the most complex aspect of this procedure (if not the most complex thing to set up on the AD3). I will set up the health script to look at the cfprobe.cfm or jrunprobe.jsp and look for the return string of "Hello" (which, after the previous step is now generated by the application server rather than simply plain text). If an application server is down, the AD3 will not get back the "Hello" message and will mark the appropriate server(s) down while redirecting request to the server(s) that remain(s) alive.

Note: These strings are case sensitive and this first example checks the health of a JRun application server.

 Main# /cfg/slb/adv Layer 4 Advanced# script Enter health script number: (1-8) 1 Health Script 1# open 80 Health Script 1# send "GET /btauxdir/jrunprobe.jsp HTTP/1.1\\r\\nHOST:10.64.20.93\\r\\n\\r\\n" Health Script 1# expect "Hello" Health Script 1# close Health Script 1# apply Main# /cfg/slb/group 1 Real server group 1# health script1 Current health check type: tcp Enter health check type: script1 New pending health check type: script1 Real server group 1# apply Real server group 1# save Confirm saving to FLASH [y/n]: y

The most complicated piece of the health check script is the send line. Depending on how you view this article, it may appear that the send line is broken into two commands but it is not. Be careful when printing this article, some printers will cut off the end of the send line:

For JRun use:

 send "GET /btauxdir/jrunprobe.jsp HTTP/1.1\\r\\nHOST:10.64.20.93\\r\\n\\r\\n"

For ColdFusion use:

 send "GET /btauxdir/cfprobe.cfm HTTP/1.1\\r\\nHOST:10.64.20.93\\r\\n\\r\\n"

Health check scripts on the AD3 are as complex as they are important. In order for the script to work against ColdFusion code, you must make sure that the hosts file on each real server in the farm is complete to include the AD3 virtual server (in our example 10.64.20.93). The default gateway on the private side of the AD3 (in our example 192.168.64.1) must also be configured on each real server in the farm.

Another tricky AD3 entry is the health check typescript1, you will notice that when you change the health check type of the group from tcp to the script there is not a space between the word script and the number 1. This is inconsistent with the rest of the AD3 commands, but the script check will not supercede the default tcp check if you put in a space.

Test your health checking script. If it is correctly configured, it should appear as follows:

 Main# /inf/slb/real 1   1: 192.168.64.11, 00:60:97:69:cc:13, vlan 1, port 3, health     4, up script 1, up, current Server Load Balancing Information# /inf/slb/real 2   2: 192.168.64.12, 00:60:08:19:81:86, vlan 1, port 4, health     4, up  script 1, up, current

Continue testing your health checking script by simulating a stalled application server. To simulate a stalled application server, simply open jrunprobe.jsp on tiger-two using a simple text editor such as Notepad. Delete the H in the Hello message which is also the expect string in the health checking script on the AD3. Save the modified jrunprobe.jsp. The AD3 will mark tiger-two down and redirect all requests to tiger-one as illustrated below.

Note: For ColdFusion, use cfprobe.cfm.
```
 Server Load Balancing Information#  Apr  6 11:05:11 ALERT   slb: cannot run script 1 on real server 192.168.64.12/inf/slb/real 1   1: 192.168.64.12, 00:60:97:69:cc:13, vlan 1, port 3, health 4, FAILED   script 1, DOWN, current   send GET /btauxdir/jrunprobe.jsp HTTP/1.1\r\nHOST:10.64.20.93\r\n\r\n   expect Hello 
```

Finish testing your health checking script by editing the jrunprobe.jsp page back to its original form - add the H back into the jrunprobe.jsp and the AD3 will dynamically mark tiger-two back up.

 Server Load Balancing Information#  Apr  6 11:05:24 NOTICE  slb: real server 192.168.64.12 operational Server Load Balancing Information# /inf/slb/real 1   1: 192.168.64.12, 00:60:97:69:cc:13, vlan 1, port 3, health 4, up   script 1, up, current Server Load Balancing Information#

This concludes the article. The result of following this workaround procedure is a ColdFusion or JRun cluster running behind an AD3 with CC monitoring CF/JR and IIS (or NES or Apache) availability. This provides Web server recovery in the event of a hang or crash, automatic E-mail alerts for Web server problems, and daily server status reports. There is much more you can accomplish by combining ACEdirector with Macromedia's enterprise application servers. For example, you may choose to set up numerous geographically distributed CF/JR Web sites using ACEdirector's Global Server Load Balancing feature. ACEdirector is a high-end switch; when combined with a cluster of Macromedia's enterprise application servers, the end result is a world-class Web site.