Thursday 1 December 2016

Custom Load Monitor on Netscaler



Recently worked on Load Monitors and it was quite tricky as there is no proper explanation for same. Sharing my knowledge based on that

Custom Load Monitors:-
When you have a scenario where you need to do load balancing based on the Load then we go for the CUSTOMLOAD Method and Load monitors.
Load monitors use SNMP polled OIDs to calculate load. The load monitor uses the IP address of the service to which it is bound (the destination IP address) for polling. It sends an SNMP query to the service, specifying the OID for a metric. The metrics can be CPU, memory, or number of server connections. The server responds to the query with a metric value. The metric value in the response is compared with the threshold value. The NetScaler appliance considers the service for load balancing only if the metric is less than the threshold value. The service with the lowest load value is considered first.

Use Case Scenario:- GSLB load balancing method doesn’t understand the Actual load on the Server but the load on the loadbalancer. So if you need to do a GSLB loadbalancing based on the Load on the Server then you go for this method.

Another Scenario would be when you want to load balance based on cpu of the server or Memory load on the server or traffic load. i.e any value which can be fetched via SNMP. 
 Note:-
The backend server should be SNMP compatible so that Netscaler can fetch the value using SNMP

Lab Method:-
LB on the Netscaler
Service 1 is the NS1
Service 2 is the NS2
Method is based on the number of client connection on the NS1 and NS2

Configuration:-

First need to create a Metric Table for the SNMP, Here I am using the Inbuilt Netscaler Metric Table


Then Create A Load Monitor and give the destination ip as the IP of the Server


Under the Special Parameters select the Metric Table and provide the type ( Here I am using the client connection) and Give the Threshold ( I have given 50 here).
SNMP Community name should be based on the name you are using for SNMP. Here I am using public so my server SNMP community name should also be “public”


Similarly Create the other Load Monitor for Second Server




Now We need to create the Services for same
Create the first service as NS1 and bind the Load Monitor 1

And for Second Service NS2 bind the Load Monitor 2

Now on the LB Vserver bind the 2 services

And select the Method as CUSTOMLOAD


This all scenario works based on the SNMP Value received. This should be an Integer Value. So make sure the Backend server are giving proper response for the SNMP Query.


FLOW and Design :-

Initially when you will configure this method the present method you will notice will be Round Robin

> sh lb vserver lb
        lb (10.104.23.72:80) - HTTP     Type: ADDRESS
        State: UP
        Last state change was at Sat Apr  4 10:36:35 2015
        Time since last state change: 0 days, 00:14:25.870
        Effective State: UP
        Client Idle Timeout: 180 sec
        Down state flush: ENABLED
        Disable Primary Vserver On Down : DISABLED
        Appflow logging: ENABLED
        Port Rewrite : DISABLED
        No. of Bound Services :  2 (Total)       2 (Active)
        Configured Method: CUSTOMLOAD
        Current Method: Round Robin, Reason: A new service is bound
        Mode: IP
        Persistence: NONE
        Vserver IP and Port insertion: OFF
        Push: DISABLED  Push VServer:
        Push Multi Clients: NO
        Push Label Rule: none
        L2Conn: OFF
        Skip Persistency: None
        IcmpResponse: PASSIVE
        New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
        TD: 0
        Mac mode Retain Vlan: DISABLED
        DBS_LB: DISABLED
        DNS64 Synth: DISABLED   Bypass AAAA: NO

1) NS1 (10.104.23.131: 80) - HTTP State: UP     Weight: 1
2) NS2 (10.104.40.43: 80) - HTTP State: UP      Weight: 1

In stat lb connection you can notice the number of Hits to the site and connection


> stat lb vserver lb

Virtual Server Summary
                      vsvrIP  port     Protocol        State   Health  actSvcs
lb              10.104.23.72    80         HTTP           UP      100        2

           inactSvcs
lb                 0

Virtual Server Statistics
                                          Rate (/s)                Total
Vserver hits                                       0                    0
Requests                                           0                    0
Responses                                          0                    0
Request bytes                                      0                    0
Response bytes                                     0                    0
Total Packets rcvd                                 0                    0
Total Packets sent                                 0                    0
Current client connections                        --                   25
Current Client Est connections                    --                   25
Current server connections                        --                    0
Requests in surge queue                           --                    0
Requests in vserver's surgeQ                      --                    0
Requests in service's surgeQs                     --                    0
Spill Over Threshold                              --                    0
Spill Over Hits                                   --                    0
Labeled Connection                                --                    0
Push Labeled Connection                           --                    0
Deferred Request                                   0                    0
Invalid Request/Response                          --                    0
Invalid Request/Response Dropped                  --                    0

Bound Service(s) Summary
                          IP  port         Type        State     Hits   Hits/s
NS1            10.104.23.131    80         HTTP           UP        0      0/s
NS2             10.104.40.43    80         HTTP           UP        0      0/s

                 Req    Req/s      Rsp    Rsp/s Throughp ClntConn   SurgeQ
NS1                0      0/s        0      0/s        0        0        0
NS2                0      0/s        0      0/s        0        0        0

             SvrConn   ReuseP  MaxConn ActvTran  SvrTTFB     Load
NS1                9        0        0        0        0       42
NS2                9        0        0        0        0       82

In the Wireshark You can see the number of Connections based on SNMP to the Service


Initial Method will show Round Robin unless the Server receives enough number of Request. The CustomLoad method will start when the number of Hits exceeds the Threshold value ( 50) for both the value.

I do see that in my lab the number of Hits Reached 102/103 but still the Method was Round Robin, So I suspect it takes some number of request which should be greater then threshold when it triggers the Custom Load.

> stat lb vserver lb

Bound Service(s) Summary
                          IP  port         Type        State     Hits   Hits/s
NS1            10.104.23.131    80         HTTP           UP      138      5/s
NS2             10.104.40.43    80         HTTP           UP      137      5/s
Done

> sh lb vserver lb
        lb (10.104.23.72:80) - HTTP     Type: ADDRESS
        State: UP
        Last state change was at Sat Apr  4 10:36:35 2015
        Time since last state change: 0 days, 00:15:17.800
        Effective State: UP
        Client Idle Timeout: 180 sec
        Down state flush: ENABLED
        Disable Primary Vserver On Down : DISABLED
        Appflow logging: ENABLED
        Port Rewrite : DISABLED
        No. of Bound Services :  2 (Total)       2 (Active)
        Configured Method: CUSTOMLOAD
        Current Method: Round Robin, Reason: A new service is bound
        Mode: IP
        Persistence: NONE

I do see when the Request reached enough Numbers like 265 then the CUSTOM Load was Triggered
> stat lb vserver lb

Virtual Server Summary
                      vsvrIP  port     Protocol        State   Health  actSvcs
lb              10.104.23.72    80         HTTP           UP      100        2

           inactSvcs
lb                 0

Virtual Server Statistics
                                          Rate (/s)                Total

Bound Service(s) Summary
                          IP  port         Type        State     Hits   Hits/s
NS1            10.104.23.131    80         HTTP           UP      265     10/s
NS2             10.104.40.43    80         HTTP           UP      150      0/s

> sh lb vserver lb
        lb (10.104.23.72:80) - HTTP     Type: ADDRESS
        State: UP
        Last state change was at Sat Apr  4 10:36:35 2015
        Time since last state change: 0 days, 00:15:39.240
        Effective State: UP
        Client Idle Timeout: 180 sec
        Down state flush: ENABLED
        Disable Primary Vserver On Down : DISABLED
        Appflow logging: ENABLED
        Port Rewrite : DISABLED
        No. of Bound Services :  2 (Total)       2 (Active)
        Configured Method: CUSTOMLOAD
        Mode: IP
        Persistence: NONE
        Vserver IP and Port insertion: OFF
        Push: DISABLED  Push VServer:
        Push Multi Clients: NO
        Push Label Rule: none
        L2Conn: OFF
        Skip Persistency: None
        IcmpResponse: PASSIVE
        New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
        TD: 0
        Mac mode Retain Vlan: DISABLED
        DBS_LB: DISABLED
        DNS64 Synth: DISABLED   Bypass AAAA: NO

1) NS1 (10.104.23.131: 80) - HTTP State: UP     Weight: 1
2) NS2 (10.104.40.43: 80) - HTTP State: UP      Weight: 1

Now the Netscaler Will See the SNMP Response Returned by both the Service and the load will be shared based on the low value


As we notice that NS1 service is giving a response as 30 and NS2 is giving a response as 23. So More load will be given to NS2. We can confirm that in stat lb

> stat lb vserver lb

Virtual Server Summary
                      vsvrIP  port     Protocol        State   Health  actSvcs
lb              10.104.23.72    80         HTTP           UP      100        2

           inactSvcs
lb                 0

Virtual Server Statistics
                                          Rate (/s)                Total
Vserver hits                                      52                 1050
Requests                                          52                 1050
Responses                                         52                 1048
Request bytes                                   5689               115060
Response bytes                                897924             18392716
Total Packets rcvd                               485                 9840
Total Packets sent                               798                16264
Current client connections                        --                    7
Current Client Est connections                    --                    7
Current server connections                        --                    2
Requests in surge queue                           --                    0
Requests in vserver's surgeQ                      --                    0
Requests in service's surgeQs                     --                    0
Spill Over Threshold                              --                    0
Spill Over Hits                                   --                    0
Labeled Connection                                --                    0
Push Labeled Connection                           --                    0
Deferred Request                                   0                    0
Invalid Request/Response                          --                    0
Invalid Request/Response Dropped                  --                    0

Bound Service(s) Summary
                          IP  port         Type                   State     Hits   Hits/s
NS1            10.104.23.131    80         HTTP           UP      850     45/s
NS2             10.104.40.43    80         HTTP           UP      200      7/s

                 Req    Req/s      Rsp    Rsp/s Throughp ClntConn   SurgeQ
NS1              850     45/s      849     45/s        6        1        0
NS2              200      7/s      199      7/s        1        1        0

             SvrConn   ReuseP  MaxConn ActvTran  SvrTTFB     Load
NS1                4        3        0        1       18       54
NS2                8        1        0        1       59      102

Now this will continue untill we are not reaching the Threshold of 50 provided in the LB Monitor

Now if any of the Backend Server reaches the Threshold of 50 and reply with a Value of more than 50 then LB will Skip that service.

As we see if there are more connection on the server then we receive following Response from the Backend Servers on SNMP which is exceeding the Threshold we have configured.

Then the Netscaler will skip the service

> sh lb vserver lb
        lb (10.104.23.72:80) - HTTP     Type: ADDRESS
        State: UP
        Last state change was at Sat Apr  4 10:36:35 2015
        Time since last state change: 0 days, 00:17:25.590
        Effective State: UP
        Client Idle Timeout: 180 sec
        Down state flush: ENABLED
        Disable Primary Vserver On Down : DISABLED
        Appflow logging: ENABLED
        Port Rewrite : DISABLED
        No. of Bound Services :  2 (Total)       2 (Active)
        Configured Method: CUSTOMLOAD
        Mode: IP
        Persistence: NONE
        Vserver IP and Port insertion: OFF
        Push: DISABLED  Push VServer:
        Push Multi Clients: NO
        Push Label Rule: none
        L2Conn: OFF
        Skip Persistency: None
        IcmpResponse: PASSIVE
        New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0
        TD: 0
        Mac mode Retain Vlan: DISABLED
        DBS_LB: DISABLED
        DNS64 Synth: DISABLED   Bypass AAAA: NO

1) NS1 (10.104.23.131: 80) - HTTP State: UP     Weight: 1
                Warning: Service will be skipped in LB as it has reached threshold
2) NS2 (10.104.40.43: 80) - HTTP State: UP      Weight: 1
                Warning: Service will be skipped in LB as it has reached threshold

1)      Policy : appfw Priority:100     GotoPriority Expression: END

Done