Building NLB Exchange 2010 RTM CAS / HT Servers (Hyper-V)

Building NLB Exchange 2010 RTM CAS / HT Servers (Hyper-V) – Part 1

UPDATE: 07/12/2011

Since this has been a topic of debate, I’ve added an additional network diagram (not one generated in my lab) so that it may become a little clearer as to how NLB works. The understanding for this network configuration comes directly from Michael Platts’ blog post: Balancing Act: Dual-NIC Configuration with Windows Server 2008 NLB Clusters.

I know I’ve been pretty offline regarding this blog but I fully intend to jump start it soon.

This post was created using Joe Hoegler’s post as a guide. All Exchange 2010 client access to mailboxes and other resources go through the CAS server. Clients no longer connect to the Exchange server directly for anything. With this said, it makes sense to build CAS servers in an HA configuration. This article was created to detail the steps required to create a load balanced CAS / HT server using Windows Network Load Balancing.

Before deploying the NLB configuration, the following should be carefully considered:

“You achieve load balancing for Hub Transport servers when you install more than one Hub Transport server in the same Active Directory site. By default, connections to Hub Transport servers are automatically load balanced if more than one Hub Transport server is deployed in an Active Directory site. If one Hub Transport server is unavailable, the operational Hub Transport servers continue to accept connections. If all Hub Transport servers in an Active Directory site are unavailable, messages are queued until a Hub Transport server becomes available or the messages expire.

Load balancing of outbound connections to remote domains is achieved by specifying more than one Hub Transport server in the same Active Directory site as a source server for the corresponding Send connector. Load balancing doesn’t occur when the source servers for a Send connector are located in different Active Directory sites.

Note:
If the Hub Transport server is installed on the same hardware as the Mailbox server role, load balancing may not occur. When the Hub Transport server role is on the same hardware as the Mailbox server role, the local server is preferred for all messages that are sent by users who have mailboxes on that server. Therefore, in this scenario, true load balancing does not occur.” Taken from: http://technet.microsoft.com/en-us/library/bb125239.aspx

Requirements

Two or more servers running Windows 2008 R2 (earlier version do have NLB built in but for this blog entry, I will be using Windows 2008 R2). Ideally each server will be configured with two NICs. One for client access and one for the cluster heartbeat. NLB requires all IPs to be on the same network.

Notes on Hyper-V

If you’re deploying an NLB on a Hyper-V host, the NICs that are used for NLB communication, i.e. the Private network NICs must be configured with a static MAC address and be enabled for MAC address spoofing. More about this at the end of the post.

Planning:

I will be using the following configuration in the lab for the NLB implementation:

NOTE: This design was created for my lab. In a live environment it is highly recommended that the IP addresses used are either consecutive or have some sort of meaningful numbers that make it easy to administer.

UPDATED GRAPHIC

Client Traffic Flow

Implementation

Once the planning for the cluster has been completed, we are now ready to start the deployment process.

Create a DNS A record entry for the virtual IP address. (MCA1CASArray.morecoffeeany1.com)

Name the NICs on each server to reflect its function.

After the NICs have been renamed, ensure that the client facing NIC is the first NIC in the binding order. Open the Network Connections applet and hold down the Alt key and press F to get the menu to display. Once this is done, select Advanced from the dropdown menu and then Advanced Settings…

The Network Load Balancing services are not installed by default. They can be installed via a Command Prompt by running the following command:

“ServerManagerCmd -i NLB”

Launch Server Manager (if you’re more comfortable with the GUI):

Start->Administrative Tools->Server Manager. Click on Add Features. Select/Install the Network Load Balancing feature.

Launch the Network load Balancing Manager to continue the process. Start->Administrative Tools->Network Load Balancing Manager.

Right click the Network Load Balancing Cluster Node and select New Cluster.

Enter the name of the first cluster node in the New Cluster Wizard page and click Connect.

Click Connect. Both NICs available on the server will be displayed. Choose the NIC slated for cluster only communications and click Next. In our case this NIC is the “Cluster Only LAN”

On the Host Parameters applet, leave everything default (Do not change the Priority) and click Next.

On the Cluster IP Address screen, click Add and enter the IP address of the Virtual IP. This is the IP address of the DNS record created earlier.

On the Cluster Parameters screen, enter the FQDN of the VIP. In our case, this will be “MCA1CASArray.morecoffeeany1.com”. Click Next. Note the preferred Operation Mode is Unicast.

The Port Rules screen can be left as is and it will allow users to connect but in the interests of added security as this will be the interface (in most cases) that the incoming connection from the Internet (via an edge device) will connect to, thus only configuring the required ports is highly recommended. The following ports are configured:

443 HTTPs
80 HTTP
143 IMAP
110 POP
135 RPC End Point Mapper
1024-65535 dynamic port range for Outlook RPC access

NOTE: IF this CAS server will be INTERNET FACING ONLY, I noticed that some people suggest securing access by removing the RPC ports.

In the lab, we will require 80, 443, 135 and 1024 – 65535 only.

For more on Client Affinity go to: http://technet.microsoft.com/en-us/library/bb687542.aspx

Once all the ports are mapped for the installation, click Finish

Now we’re ready to add a second host to the cluster. Right click the cluster name and slect “Add Host to Cluster”. Remember that the host that is to be added needs to have the cluster installed first (SeverManagerCMD –i NLB) before adding the host to the cluster.

On the Connect screen, type the name of the scond host your wish to add to the NLB and click Connect. This will display the network adapters available for the NLB cluster. Select the network adapter to be used for NLB communication only and click Next

Leave the Host Parameters screen default and click next.

Note that the Port Rules for the second host will appear as configured earlier. Leave these as is and click Finish

I built my NLB cluster in a Hyper-V lab and did NOT make the changes I listed in the beginning of this post, I encountered these two errors.

If you do encounter these errors, simply change the to a static MAC and enable MAC spoofing. Once you’ve done this, go back to the NLB Manager and right click the first host and select Host Properties and click OK. The host with them attempt to revalidate the cluster. This is all I had to do to get it fixed. Now we’re ready to validate the cluster.

Building NLB Exchange 2010 RTM CAS / HT Servers (Hyper-V) – Part 2

Building NLB Exchange 2010 RTM CAS / HT Servers (Hyper-V) – Part 3

Like this:

Be the first to like this post.

March 19, 2010 - Posted by ivorb | Exchange 2010 RTM, NLB | NLB Windows 2008 R2

10 Comments »

Great post! Very relevant with the Hyper-V details. Thanks for the reference.

Comment by Joe Hoegler | March 24, 2010 | Reply
- Thanks Joe. Just deployed an NLB setup on VMWare at a customer site and found that we had to configure static MACs in VMWare before the NLB cluster would converge.
  
  Comment by ivorb | March 24, 2010 | Reply
When you set the static MAC address on the Heartbeat virtual NIC in Hyper-V for the second VM, did you set it to the same MAC address as for the first VM, or to a different value?

Comment by Mark | April 16, 2010 | Reply
I have to ask point out theres that appears to be fundamentally wrong with your config here.

your diagram shows your NLB cluster with public and private NICs. They are on different IP subnets.

It looks like you want your private nics (10.0.0.x) to be heartbeat only, with the public NICs (192.168.0.x) to be used for application IO.

But in your NLB cluster you are binding the cluster IP (192.168.0.113) to the private NIC’s (10.0.0.x)

If this is a real world configuration then i can only assume that you are not using VLANs. For this to work as you have it, both your clients and all the NLBS NICs would have to be on the same broadcase domain.

Otherwise, traffic to the FQDN of your cluster (192.168.0.113) is never going to be reach the NIC where the Cluster IP is bound (10.0.0.x).

Put simply – if theres a router in there, traffic to 192.168.0.x will not get routed to the 10.0.0.x subnet.

Are you sure this works anywhere except in the Lab?

Also – apologies if i misunderstood your setup. but it really looks that way to me.

Comment by boxing surfer | December 5, 2010 | Reply
- Boxing Surfer, you are correct. This is a lab configuration. Fortunately, I’ve managed to steer clear of NLB’s in production. NLB is a failover technology and NOT a true load balancing technology. If a client has a small budget, I usually recommend something like Kemp Technologies load balancers to get them going.
  
  There are no VLANS in this lab configuration and I’ve been super lucky that almost all the clients I’ve worked with so far with Exchange 2010 either had hardware load balancers in place or purchased them for the Exchange project.
  
  Comment by ivorb | December 6, 2010 | Reply
Is it the case, then, that we should configure our systems like this:

Guest Machine 1
LAN NIC: 192.168.1.100
NLB NIC: 10.0.0.1

Guest Machine 2
LAN NIC: 192.168.1.101
NLB NIC: 10.0.0.2

NLB VIP 10.0.0.3

I have done this and it is the only way I can get the vip to be pingable. If I make the VIP 192.168.1.113, I cannot ping it.

Comment by David Parent | December 28, 2010 | Reply
- Correct. I’ve sliced and diced this configuration and this seems to be the only way it works. Why, that I can’t answer. If anyone else can shed light on that, I would like to know.
  
  Comment by ivorb | December 28, 2010 | Reply
  - Thanks much for your reply. Your article is very helpful.
    
    Comment by Dave Parent | December 29, 2010
This a good tutorial. It seems to me, however, that we need to make just a few changes to get it to work. We should name the “Cluster Only LAN” to “Heartbeat NIC.” Then rename the “Client LAN” to “Cluster NIC.” Next, we need to add the two 192…..x VM hosts to the cluster and point them to their 192….x VIP. In this way the cluster nodes should converge.

Does anyone disagree with this? If so, help us understand. Also, can someone comment on how the physical hosts should be wired to the switch? For example Can you place a crossover cable between the physical heartbear NICs and then connect the physical Client NICs to the switch?

Comment by Wilem Donziger | March 22, 2011 | Reply
- Hi Wilhelm
  
  i made the same mistake in my interpretation. When i did my first NLB in production I learned some things. Its worth considering the following points:
  
  First – and most importantly – there is not really an idea of a heartbeat NIC in NLB the way you are thinking of it. What you have in your head is something that exists in ‘Windows Failover Clustering’ not NLB, and its in a Failover Clustering setup where we would typically create a seperate hearbeat network.
  
  I think that its an easy mistake to make. when we see NLB hosts configured with multiple NICs we naturally assume that one of them is like a heartbeat NIC. But actually thats not the case. The reason we have multiple NICs is not for any ‘heartbeat’ reason
  
  Here’s the rub:
  
  In an NLB host with 2 NIC’s we usually have the NLB Service bound to only 1 NIC. Its only this NIC that the NLB driver is bound. Its only this NIC that participates in NLB. All of the NLB related function happens on this NIC. No NLB related communication happens on the other NIC to which the NLB driver is NOT bound
  
  The NLB NIC is also the NIC where the ‘cluster’ Virtual IP is bound. So this is the NIC that receives our application traffic. This is the NIC we chose when we first create an NLB cluster in the NLB Manager.
  
  Any other NIC’s in the machine are not participating in NLB at all. They are there just like normal NIC’s
  A NIC that doesnt have the NLB service bound to it is not aware of NLB,
  A NIC that doesnt have the NLB service bound to it doesnt ‘converge’ with any other NIC, and does not send any ‘heartbeat’ traffic
  
  This is confirmed in MS documentation
  
  So – you ask why do we have a second NIC?
  
  The reason why we have 2 NIC’s is usually to seperate ‘application’ traffic (traffic we want to load balance) from ‘host’ or ‘administration’ traffic or other traffice that we know we dont want to load balance. This includes traffic where the 2 hosts need to talk to each other,or talk to anything else on the LAN – like regular individual windows hosts.
  
  You see, there is a big processing overhead associated with receiving data on an NLB NIC Any packet received on an NLB enabled NIC has to be processed by the NLB driver. So if we receive anything on an NLB NIC the driver will have to process it and figure out if its something that NLB interested in or not.
  
  Typically we dont want anything other than the application we are trying to load balance to come in on an NLB NIC. We certainly dont want ‘AD Domain’ traffic or maybe ‘SQL’ traffic or ‘RDP’ or anything like that going on to an NLB address. We dont want any of the regular windows chit chat coming in through this NIC either. This bad for 2 reasons
  
  1) becuase – depending on if we are running unicast of multicast mode – the hosts might not be distinguishable from each other at Layer 2 (MAC address) on the NLB NIC – so we may not be able to know which host we are reaching. and….
  2) because every single one of those packets will have to be processed by the NLB driver on either or both of the hosts.
  
  In my environment we only use NLB for INTRANET applications. Both NICs are connected to the the same switch, but NLB and non NLB NICS are on different VLANs. There are so many difrerent ways you can set this up. There is not 1 correct way. It depends what you are trying to do.
  
  I do this:
  
  -> NLB NICS are connected to switchports that are set on a dedicated VLAN (example. 10.0.20.0 /28)
  -> non NLB NICS are connected to switchports that are set on a seperate VLAN (example 10.0.50.0 /25)
  -> Cluster IP address setup on the NLB NICs in the NLB IP subnet and NOT the regular ‘subnet’
  (ivorb’s great guide above is very misleading in this step (see my comment above)
  -> NLB NIC’s are NOT configured with any default gateway or DNS servers
  -> non NLB NIC’s have the Default Gateway and DNS (example – core router on 10.0.50.1)
  -> The DNS name of the application i want load balanced is manually registered in DNS with the NLB cluster IP address only.
  -> DNS dynamic registration is turned off on the NLB NICS
  -> enable IP forwarding from the NLB NIC’s to the non NLB NIC with ‘netsh interface ipv4 set int “” forwarding=enabled’ so that replies to NLB traffic can be sent out on a different NIC to that which it was received on.
  
  With the above setup, my NLB NICS are effectively ‘receive only’ NIC’s and they only receive application traffic that i want to load balance. Any and all traffic thats sent from any NLB host is sent via the other non NLB NIC. Likewise the hostnames are registered on the non NLB NICs, so anything thats intended for the host name (rather than the application) comes in to a refular NIC.
  
  This works pretty well for me.
  
  I quote from this blog post which explains it quite well:
  
  http://blogs.technet.com/b/clint_huffman/archive/2007/10/08/an-optimal-network-load-balancing-nlb-configuration.aspx
  
  “[Windows Failover Clustering].. uses a backend “heartbeat” network used to determine if their partner server is alive or down. NLB also has a “heartbeat”, but it’s heartbeat is really just broadcast TCP/IP packets on the NLB enabled network… let me repeat… on the NLB enabled network…..”
  
  Comment by boxing_surfer | July 10, 2011 | Reply

More Coffee Anyone?

Notes from the lab and the field