In-depth Troubleshooting on NetScaler using Command Line Tools

Andrew Sandford

Senior Readiness Specialist, Worldwide Support Readiness EMEA

Citrix Support Secrets Webinar SeriesIn-depth Troubleshooting on NetScaler using Command Line Tools

27 March 2014

© 2014 Citrix | Confidential – Do Not Distribute

Agenda

NetScaler CLI recap

What’s my NetScaler?

How is my NetScaler configured?

What’s my NetScaler doing?

Time for Technical Support?

Q&A

NetScaler CLIRecap

© 2014 Citrix | Confidential – Do Not Distribute© 2014 Citrix | Confidential – Do Not Distribute

NetScaler Native CLI

> show

>add

>remove

>set

>enable

>disable

>force

>bind

>unbind


NetScaler Native CLI

>help <command>

>man <command>

>set cli prompt %u@%h-%T

>set cli mode -color ON


NetScaler BSD Shell

#


# tar

# head

# less

# more

# cat

# zcat

# ls

# find




> show version

> show ns hostname

> show hardware

> show interface –summary

>stat ssl

#sysctl –a netscaler | more


show version/show ns hostname

> show version

NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27

> show ns hostname

Hostname: nstipster


show hardware

> show hardware

Platform: NSMPX-10500 8*CPU+2*E1K+8*E1K+2*IX+8*CVM 1620 760100

Manufactured on: 10/7/2010

CPU: 2832MHZ

Host Id: 1234567890

Serial no: M123456789

Encoded serial no: M123456789


show interface -summary

> show interface -summary

--------------------------------------------------------------------------------

Interface MTU MAC Suffix

--------------------------------------------------------------------------------

1 0/1 1500 00:25:90:12:eb:5a Gig Ethernet 10/100/1000 MBits

2 0/2 1500 00:25:90:12:eb:5b Gig Ethernet 10/100/1000 MBits

3 1/1 1500 00:e0:ed:1a:24:97 Gig Ethernet, copper SFP

4 1/2 1500 00:e0:ed:1a:24:96 Gig Ethernet, copper SFP

5 1/3 1500 00:e0:ed:1a:24:95 Gig Ethernet, no SFP found

6 1/4 1500 00:e0:ed:1a:24:94 Gig Ethernet, no SFP found

7 1/5 1500 00:e0:ed:1a:24:a3 Gig Ethernet, copper SFP

8 1/6 1500 00:e0:ed:1a:24:a2 Gig Ethernet, no SFP found



11 10/1 1500 00:1b:21:77:c0:35 10G Ethernet,...SFP+/SFP found

12 10/2 1500 00:1b:21:77:c0:34 10G Ethernet,...SFP+/SFP found

13 LO/1 1500 00:25:90:12:eb:5a NetScaler Loopback interface


stat ssl

> stat ssl

SSL Summary

# SSL cards present 8

# SSL cards UP 8

SSL engine status 1

SSL sessions (Rate) 0

System

Transactions Rate (/s) Total

SSL transactions 0 301

SSLv2 transactions 0 0

SSLv3 transactions 0 0

TLSv1 transactions 0 301


sysctl

# sysctl -a netscaler

netscaler.developer: 0

netscaler.recovery: 0

netscaler.sysid: 450000

netscaler.serial: 98310000cb254307ee78

netscaler.descr: NetScaler Virtual Appliance 3G

netscaler.num_pe_running: 1

netscaler.version: NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27

netscaler.model: 3000

netscaler.vmpe_max_cpus: 2

netscaler.Classic: 0

netscaler.nCore: 1

netscaler.descr: NetScaler Virtual Appliance 3Gnetscaler.num_pe_running: 1netscaler.version: NetScaler NS10.1: Build 124.13.nc, Date: Feb 20 2014, 18:53:27netscaler.model: 3000netscaler.vmpe_max_cpus: 2netscaler.nCore: 1




> show ip

> show feature

> show ns mode

> show info

> show license

> show run | more


show ip

> show ip

Ipaddress TD Type Mode Arp Icmp Vserver State

--------- -- ---- ---- --- ---- ------- ------

1) 192.168.196.45 0 NetScaler IP Active Enabled Enabled NA Enabled

2) 192.168.196.146 0 SNIP Active Enabled Enabled NA Enabled

3) 192.168.196.147 0 VIP Active Enabled Enabled Enabled Enabled






show feature

> show feature

Feature Acronym Status

------- ------- ------

1) Web Logging WL ON

2) Surge Protection SP OFF

3) Load Balancing LB ON

4) Content Switching CS ON

5) Cache Redirection CR OFF

6) Sure Connect SC ON

8) Priority Queuing PQ ON

9) SSL Offloading SSL ON

10) Global Server Load Balancing GSLB ON

11) Http DoS Protection HDOSP OFF

12) Content Filtering CF ON

13) Integrated Caching IC OFF

19) Rewrite REWRITE ON


show ns mode

> show ns mode

Mode Acronym Status

------- ------- ------

1) Fast Ramp FR ON

2) Layer 2 mode L2 OFF

3) Use Source IP USIP OFF

4) Client Keep-alive CKA OFF

5) TCP Buffering TCPB OFF

6) MAC-based forwarding MBF OFF

7) Edge configuration Edge ON

8) Use Subnet IP USNIP ON

9) Layer 3 mode (ip forwarding) L3 ON

10) Path MTU Discovery PMTUD ON

16) Bridge BPDUs BridgeBPDUs OFF


show info

> show info


NetScaler IP: 192.168.47.6 (mask: 255.255.240.0)

NW FWMODE: NOFIREWALL

Number of MappedIP(s): 0

Node: Secondary (Primary is 192.168.47.1)

System Time: Thu Mar 27 08:52:06 2014

Last Config Changed Time: Thu Mar 27 07:52:50 2014

Last Config Saved Time: Tue Mar 25 13:28:21 2014


show run | more

#NS10.1 Build 124.13

# Last modified Thu Mar 27 10:08:20 2014

set ns config -IPAddress 192.168.47.1 -netmask 255.255.240.0

enable ns feature LB CS SSL SSLVPN OSPF REWRITE AppFw RESPONDER HTMLInjection AppFlow

enable ns mode FR L3 Edge USNIP SRADV DRADV PMTUD

set system parameter -natPcbForceFlushLimit 4294967295

set system user nsroot -----------------SNIP---------------- -encrypted

set rsskeytype -rsstype ASYMMETRIC

set lacp -sysPriority 32768 -mac 11:22:33:aa:bb:cc

set ns hostName nstipster

set interface 1/1 -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype "Xen Virtual" -ifnum 1/1



set interface LO/1 -haMonitor OFF -throughput 0 -bandwidthHigh 0 -bandwidthNormal 0 -intftype Loopback -ifnum LO/1

add ns ip6 fe80::40da:5dff:fe08:296b/64 -scope link-local -type NSIP -vlan 1 -vServer DISABLED -mgmtAccess ENABLED -dynamicRouting ENABLED

What’s my NetScaler doing?


Processes and Uptime

> shell top

#ps –ax | more

#uptime


#top/nsppe

> shell top

last pid: 13825; load averages: 1.00, 1.04, 1.02 up 1+19:48:58 08:38:17

60 processes: 2 running, 58 sleeping

CPU states: 18.0% user, 0.0% nice, 32.1% system, 0.0% interrupt, 49.9% idle

Mem: 99M Active, 51M Inact, 1492M Wired, 15M Cache, 165M Buf, 2564K Free

Swap: 4500M Total, 12K Used, 4500M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND

50185 root 1 44 -52 814M 815M CPU1 1 20.2H 100.00% NSPPE-00

11834 nobody 1 4 0 26448K 18276K accept 0 0:02 0.05% httpd

50206 root 1 4 0 61364K 11632K kqread 0 1:03 0.00% nsaggregator

50251 root 1 4 0 30504K 7964K kqread 0 0:47 0.00% nsconfigd

44 root 1 4 0 15880K 1828K kqread 0 0:21 0.00% pitboss

995 root 1 8 0 25660K 17672K nanslp 0 0:13 0.00% httpd

987 root 1 96 0 3668K 876K select 0 0:11 0.00% syslogd

50188 root 1 4 0 59184K 9016K kqread 0 0:10 0.00% nsnetsvc

100%


#ps –ax | more

root@ns# ps -ax | more

PID TT STAT TIME COMMAND

0 ?? WLs 0:00.44 [swapper]

1 ?? ILs 0:00.09 /sbin/init --

2 ?? DL 0:02.35 [g_event]

3 ?? DL 0:09.00 [g_up]

4 ?? DL 0:02.21 [g_down]

5 ?? DL 0:00.00 [xpt_thrd]

6 ?? DL 0:00.00 [acpi_task_0]

7 ?? DL 0:00.00 [acpi_task_1]

8 ?? DL 0:00.00 [acpi_task_2]

9 ?? DL 0:00.00 [kqueue taskq]

10 ?? RL 0:10.39 [idle: cpu1]

--More—(byte 933)


IP related

> show route

> show ip

> show dns addrec -type proxy

# ping

# traceroute

# telnet


> show route

> show route

Network Netmask Gateway/OwnedIP State TD Type

------- ------- --------------- ----- -- ----

1) 0.0.0.0 0.0.0.0 192.168.32.1 UP 0 STATIC

2) 127.0.0.0 255.0.0.0 127.0.0.1 UP 0 PERMANENT

3) 192.168.32.0 255.255.240.0 192.168.47.1 UP 0 DIRECT

4) 172.16.200.0 255.255.255.0 192.168.47.2 UP 0 STATIC|ADV


Load Balancing

> show lb vserver

> show service

> show connectiontable | grep <IP Address|port>

> show persistentSessions


show lb vserver

> show lb vserver

1) LB_RGB (192.168.47.3:80) - HTTP Type: ADDRESS

State: UP

Last state change was at Wed Aug 14 09:17:14 2013

Time since last state change: 0 days, 00:30:42.140

Effective State: UP

Client Idle Timeout: 180 sec

Down state flush: ENABLED

Disable Primary Vserver On Down : DISABLED

Appflow logging: ENABLED

Port Rewrite : DISABLED

No. of Bound Services : 1 (Total) 1 (Active)

Configured Method: ROUNDROBIN

Mode: IP

Persistence: NONE

Vserver IP and Port insertion: OFF

IcmpResponse: PASSIVE

New Service Startup Request Rate: 0 PER_SECOND, Increment Interval: 0

Warning: Feature(s) not enabled [LB]

LB_RGB (192.168.47.3:80) - HTTP Type: ADDRESS

State: UPClient Idle Timeout: 180 sec


Disable Primary Vserver On Down : DISABLED

Appflow logging: ENABLED

Port Rewrite : DISABLED



Mode: IP

Persistence: NONE

Vserver IP and Port insertion: OFFWarning: Feature(s) not enabled [LB]


show lb vserver [vservername]

> sh lb vserver LB_RGB

LB_RGB (192.168.47.3:80) - HTTP Type: ADDRESS

State: UP



Effective State: UP

Client Idle Timeout: 180 sec




Mode: IP

Persistence: NONE

1) svc_blue (192.168.196.62: 80) - HTTP State: UP Weight: 1

1) svc_blue (192.168.196.62: 80) - HTTP State: UP Weight: 1


show service [servicename]

> show service svc_blue

svc_blue (192.168.196.62:80) - HTTP

State: UP



Server Name: Blue

Use Source IP: NO

Idle timeout: Client: 180 sec Server: 360 sec

Client IP: ENABLED ClientIP

1) Monitor Name: mon-http-ecv

State: UP Weight: 1 Passive: 0

Probes: 14887 Failed [Total: 124 Current: 0]

Last response: Success - Pattern found in response.

Response Time: 10.220 millisec

1) Monitor Name: mon-http-ecv

State: UP Weight: 1 Passive: 0

Probes: 14887 Failed [Total: 124 Current: 0]

Last response: Success - Pattern found in response.

Response Time: 10.220 millisec

State: UP


show persistentSessions

> sh persistentSessions

Type SRC-IP DST-IP PORT VSNAMETIMEOUT PERSISTENCE-PARAMETER

SOURCEIP 192.168.119.81 192.168.196.61 80 LB_RGB 118192.168.119.81


Authentication

> show vpn vserver

> show aaa session

> show aaa stats

# cat /tmp/aaad.debug

How do I troubleshoot deeper?

Time for Technical Support?


Core dump directories

# ls -la

total 8

drwxrwxr-x 4 root nobody 512 Mar 4 09:41 .

drwxr-xr-x 31 root wheel 1024 Dec 4 10:06 ..

drwxrwxr-x 2 root nobody 512 Oct 8 21:28 1

-rw-r--r-- 1 root nobody 2 Mar 4 09:36 bounds

# ls –la 1/

total 10292

drwxrwxr-x 2 root nobody 512 Jan 22 11:24 .

drwxrwxr-x 4 root nobody 512 Mar 4 09:41 ..

-rw------- 1 root nobody 9881665 Dec 18 11:15 NSPPE-00-1094.gz

-rw------- 1 root nobody 603327 Dec 18 11:15 nscac64p-1189.gz


Packet Engine (nsppe) Process Crashing

Oct 19 08:52:46 <local0.alert> vpx1 nsppe: PE 0 (pid 1077) got signal 6; signal mask is 0x0 0x0 0x0 0x0

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NSPPE-00 (1077) unexpectedly died due to receiving signal

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 There may be a delay restarting process while collecting core dump on NSPPE-00 (1077)

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 proc NSPPE-00 (1077) failure. Therefore initiating nCore NetScaler restart according to policy setting (0x29ac)

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NetScaler restart may be delayed if collecting core dump for NSPPE-00 (1077)

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 Pitboss declaring system failure: NSPPE-00 (1077) exited

Oct 19 08:53:44 <local0.info> vpx1 [44]: pitboss Sat Oct 19 08:53:44 2013 Deleting watch on NSPPE-00 (1077) for ()


/var/core

# cd /var/core/

# ls -la

total 14

drwxrwxr-x 6 root nobody 512 Oct 25 07:54 .

drwxr-xr-x 30 root wheel 1024 Oct 25 10:31 ..

drwxrwxr-x 2 root nobody 512 Sep 30 13:39 1

-rw-r--r-- 1 root nobody 2 Oct 25 07:54 bounds

root@vpx1# ls 1/

NSPPE-00-1077.gz nscac64p-1177.gz nsnetsvc-1086.gz

aslearn-1148.gz nscfsyncd-1158.gz nsrised-1164.gz

imi-1129.gz nsclfsyncd-1160.gz provserverd-1162.gz

monuploadd-1154.gz nsclusterd-1105.gz snmpd-1152.gz

nsaaad-1131.gz nsconfigd-1156.gz

nsaggregatord-1107.gz nsfsyncd-1110.gz


Show commandsSystem show node

show info

show licenseVserver/Service show lb vserver

show cs vserver

show service

show persistencesession

show connectiontableIP related show route

show ip

show dns addrec -type proxyDiagnostic show techsupport


show node

> show node

1) Node ID: 0

IP: 192.168.1.145 (NS145)

Node State: NOT UP

Master State: Secondary

Fail-Safe Mode: OFF

INC State: DISABLED

Sync State: ENABLED

Propagation: ENABLED

Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1

Disabled Interfaces : None

HA MON ON Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1

Interfaces on which heartbeats are not seen : 1/8 1/6 1/5 1/4 1/3 1/2 1/1

Interfaces causing Partial Failure: 1/8 1/6 1/5 1/4 1/3 1/2 1/1

SSL Card Status: UP

Hello Interval: 200 msecs

Dead Interval: 3 secs

Node in this Master State for: 0:15:2:13 (days:hrs:min:sec)

2) Node ID: 1

IP: 192.168.1.45

Node State: STAYSECONDARY


Fail-Safe Mode: OFF

INC State: DISABLED

Sync State: ENABLED

Propagation: ENABLED

Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1





SSL Card Status: UP

Local node information:

Critical Interfaces: 1/8 1/6 1/5 1/4 1/3 1/2 1/1

>

Node State: NOT UP

Master State: SecondaryMaster Enabled Interfaces : 1/8 1/7 1/6 1/5 1/4 1/3 1/2 1/1





Node State: STAYSECONDARY



NetScaler ProcessesProcess Description Process Description

nsppe NetScaler Packet Engine nsfsyncd Sync bookmarks and SSL certificates

nsaaadRBA and SSL VPN External Auth nsnetsvc

Used by the GUI for config changes

nsconf Writes the ns.conf file nsumond Runs the scriptable monitors

nslog.sh Controls Logging for newnslog nsconmsg Controls writing of newnslog

nssync HA sync nscollect Statistics gathering for historical reporting

nsreadfile Used to read SSL Cert Files imi/ripd/

ospfd/bgpdRouting processes

nscrlrefresh SSL CRL list update


stat commands

>stat ns

>stat cpu

>stat interface

>stat lb vserver

>stat cs vserver

>stat service

>stat ssl

>stat dns

>stat http

System

Entities

Protocols


stat ns

> stat ns

System overview

Up since Wed Aug 14 11:53:33 2013

CPU usage (%) 0.60

Packet CPU usage (%) 0.60

Management CPU usage (%) 0.80

Memory usage (MB) 190

InUse Memory (%) 18.73

Last Transition time We...013

System state UP

Master state Primary

# SSL cards UP 0

# SSL cards present 0


stat cpu

> stat cpu

CPU statistics

ID Usage

1 0

7 0

6 0

5 0

4 0

3 1

2 0


stat interface

> stat interface

Interface Summary

ID IntfState IntfAlias Rx Bytes Tx Bytes Rx Pkts Tx Pkts

1/8 DOWN 0 0 0 0

1/7 DOWN 0 0 0 0

1/6 DOWN 0 0 0 0

1/5 DOWN 0 0 0 0

1/4 DOWN 0 0 0 0

1/3 DOWN 0 0 0 0

1/2 DOWN 0 0 0 0

1/1 UP 14476M 21813M 442178k 56718611

10/2 DOWN 0 0 0 0

10/1 DOWN 0 0 0 0

0/1 UP 32027M 18048M 292060k 67610607

0/2 DOWN 0 0 0 0

LO/1 UP 831255M 1218G 6624M 13125M

LA/1 DOWN LA2 0 0 0 0


stat interface [interfacename]> stat interface 1/1

Interface [1/1]:Interface State UPLink uptime 00:40:21Link downtime 00:00:00

Throughput Statistics Rate (/s) TotalBytes received 42393 54497294Bytes transmitted 2584 20222135Packets received 629 710246Packets transmitted 47 69066Packet Statistics Rate (/s) TotalMulticast packets 18 41219NetScaler packets 85 98954

LACP Statistics Rate (/s) TotalLACPDUs received 0 0LACPDUs transmitted 0 0Error Statistics Rate (/s) TotalError packets received (hw) 0 0Error packets transmitted (hw) 0 0Inbound packets discarded (hw) 0 0Outbound packets discarded (hw) 0 0Packets dropped in Rx (sw) 539 599904Packets dropped in Tx (sw) 0 0NIC hangs -- 0Status stalls -- 0Transmit stalls -- 0Receive stalls -- 0Error-disables -- 0Duplex mismatches -- 0Link re-initializations -- 0MAC moves registered 0 0Times NIC became muted -- 0


stat dns

> stat dns

DNS Statistics

Runtime Statistics

Dns queries 106983

Multi queries 0

Dns responses 0

Server responses 102334

Total Record updates 0

Auth answers 0

Server queries 102347

Cache flush called 0

Cache entries flushed 0

Configuration Statistics

Non-authoritative entries 0

Authoritative entries 98

Error Statistics

Nonexistent domain 102359

Response class unsupported 0

Invalid query format 0

Stray answers 0

Incorrect RD length 0

Requests refused 0

Response type unsupported 0

Query class unsupported 0

Invalid response format 0

No answer responses 102334

Multi queries disabled 0

Other errors 0


stat http

> stat http

HTTP Statistics - Summary

Rate (/s) Total

Total requests 1 6251

Total responses 1 5885

Request bytes received 27 293191

Response bytes received 286 1744835


stat lb vserver [name]

> stat lb vserver LB_RGB

Virtual Server Summary

vsvrIP port Protocol State Health actSvcs

LB_RGB 192.168.47.3 80 HTTP DOWN 0 0

inactSvcs

LB_RGB 1

Virtual Server Statistics

Rate (/s) Total

Vserver hits 0 0

Requests 0 0

Responses 0 0

Request bytes 0 65

Response bytes 0 188

Total Packets rcvd 0 5

Total Packets sent 0 4

Vserver hits 0 0

Requests 0 0

Responses 0 0

Request bytes 0 65


Total Packets rcvd 0 5

Total Packets sent 0 4

Current client connections -- 0

Current Client Est connections -- 0

Current server connections -- 0

Requests in surge queue -- 0

Requests in vserver's surgeQ -- 0

Requests in service's surgeQs -- 0

Spill Over Threshold -- 0

Bound Service(s) Summary

IP port Type State Hits Hits/s

svc_andrews 192.168.33.130 80 HTTP DOWN 0 0/s

Req Req/s Rsp Rsp/s Throughp ClntConn SurgeQ

svc_andrews 0 0/s 0 0/s 0 0 0

SvrConn ReuseP MaxConn ActvTran SvrTTFB Load

svc_andrews 0 0 0 0 0 0


stat service [servicename]

> stat service svc_blue

Service Summary

IP port Type State

svc_blue 192.168.196.62 80 HTTP UP

Service Stats:

Rate (/s) Total

Requests 0 865

Responses 0 855

Request bytes 22 67683





Current Server Est connections -- 0

Connections in reuse pool -- 0

Maximum server connections -- 0

Average server TTFB -- 0

Current load on the service -- 0

Requests 0 865

Responses 0 855

Request bytes 22 67683





Current Server Est connections -- 0

Connections in reuse pool -- 0

Maximum server connections -- 0

Average server TTFB -- 0

Current load on the service -- 0


NetScaler Disk Partitions

# df -h

Filesystem Size Used Avail Capacity Mounted on

/dev/md0c 286M 245M 35M 88% /

devfs 1.0K 1.0K 0B 100% /dev

procfs 4.0K 4.0K 0B 100% /proc

/dev/ad0s1a 1.4G 965M 368M 72% /flash

/dev/ad0s1e 14G 3.1G 9.5G 24% /var

NetScaler File System


NetScaler File System examples

/var/log/ns.log

/var/nslog/newnslog/newnslog.ppe.x

/var/nsinstall/build-10.1-124.13_nc.tgz

/flash/ns-10.1-124.13.gz

/var/core/NSPPE-00-353.gz

/var/crash/vmcore.0

/nsconfig/ns.conf

/nsconfig/ssl/ns-root.cert

/nsconfig/monitors/nssmtp.pl

/nsconfig/license/FID__b0d70c6_13b16ab7034_573f.lic

/netscaler/nsconmsg

/var/nstrace/nstrace1.cap

Logs

Firmware

Crash FilesNetScaler Configuration

SSL CertificatesMonitor Scripts

License FilesNetScaler BinariesPacket Trace Files


dmesg

root@ns# dmesg

Copyright (c) 1992-2008 The FreeBSD Project.

Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

The Regents of the University of California. All rights reserved.

FreeBSD is a registered trademark of The FreeBSD Foundation.

FreeBSD 6.3-NETSCALER-10.1 #0: Thu Feb 20 18:54:22 PST 2014

[email protected]:/usr/obj/amd64/usr/home/build/rs_101_124_8/usr.src/sys/NS64

Preloaded elf kernel "/ns-10.1-124.13" at 0xffffffff930b3000.

Calibrating clock(s) ... i8254 clock: 1189606 Hz

CLK_USE_I8254_CALIBRATION not specified - using default frequency

Timecounter "i8254" frequency 1193182 Hz quality 0

Calibrating TSC clock ... TSC clock: 3325066248 Hz

CPU: Intel(R) Core(TM)2 Duo CPU E8600 @ 3.33GHz (3325.07-MHz K8-class CPU)

Origin = "GenuineIntel" Id = 0x1067a Stepping = 10

Features=0x789fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,CLFLUSH,MMX,FXSR,SSE,SSE2>

Features2=0x81282201<SSE3,SSSE3,CX16,<b19>,<b21>,<b24>,<b31>>

AMD Features=0x20000800<SYSCALL,LM>

AMD Features2=0x1<LAHF>

real memory = 2143289344 (2044 MB)


/var/log

# ls

auth.log httperror.log ns.log

auth.log.0.gz httperror.log.0 ns.log.0

callhome.log lastlog nscollect.log

callhomedebug.log license.log nscollect_cl.log

cron lpd-errs nsvpn.log

cron.0.gz maillog nsvpnd.log

ctxslsboc.log maillog.0.gz ntpd.log

db messages security

httpaccess.log messages.0 snmpd.log

httpaccess.log.0.gz nitro.log wicmd.log

httpd.scoreboard nitro.log.0.gz


/var/nslog

# cd /var/nslog/

# ls

asl lspci_tv.last nsagg.conf

aslearn.log lspci_tv.prev nsagg.log

aslearn_old_db.tar.gz lspci_vvvxxx.boot nslog.nextfile

conmsg.log lspci_vvvxxx.last nsumond

dmesg.boot lspci_vvvxxx.prev nsumond.log

dmesg.last newnslog snmpdebug.log

dmesg.prev newnslog.0.gz

lspci_tv.boot ns.log

# cd newnslog

# ls

newnslog.ppe.0


Troubleshooting Techniques

View eventsView console messagesView statisticsDebug system countersDebug load balancing issuesDebug CPU/Memory utilization

Use cases


Troubleshooting Techniques: nsconmsg

# cd /var/nslog

# nsconmsg –K newnslog -d event

# nsconmsg –K newnslog -d consmsg

# nsconmsg –K newnslog -d oldconmsg

# nsconmsg –K newnslog -s ConLb=2 –d oldconmsg

# nsconmsg –K newnslog -s ConDebug=1 –d oldconmsg

nsconmsg Common Syntax

Make sure to use –K, NOT -k


nsconmsg examples

# nsconmsg -d current -g cpu_use

# nsconmsg -K <newnslog-filename> -d event

# nsconmsg -d current -g ha_cur_master_state

# nsconmsg -s ConLb=2 -d oldconmsg

# nsconmsg -s ConCSW=2 -d oldconmsg

# nsconmsg -d current -g pol_hits

# nsconmsg -s ConSSL=2 -d oldconmsg

# nsconmsg -s ConCMP=2 -d oldconmsg

Live CPU related statsArchived events

HA Failover causeLB stats

CS related countersReal-time policy hitsSSL related counters

Compression related counters


Policy Hits

# cd /var/nslog/newnslog

# nsconmsg -K newnslog.ppe.0 -d current -g pol_hits

Displaying performance information

NetScaler V20 Performance Data


reltime:mili second between two records Fri Oct 25 09:45:52 2013

Index rtime totalcount-val delta rate/sec symbol-name&device-no

0 3038060 2 2 0 pol_hits Policy(CTX-LDAP)

1 3500159 15 15 2 pol_hits Policy(__ESNS_PREBODY_POLICY)

2 0 15 15 2 pol_hits Policy(__ESNS_POSTBODY_POLICY)


nsconmsg –K newnslog -d event

# nsconmsg -K newnslog -d event | more

Displaying event information



rtime: Relative time between two records in milliseconds

seqno rtime event-message event-time

1906 3910 PPE-0 'interface(1/1)' has been disabled Wed Aug 14 12:27:16 2013

1907 0 PPE-0 'interface(1/1)' DOWN Wed Aug 14 12:27:16 2013

1908 7 PPE-0 MonServiceBinding_192.168.47.16:4739_(ping-default)(service_192.168.47.16_33554): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013

1909 0 PPE-0 MonServiceBinding_192.168.224.5:53_(ping-default)(SVC_CTX_DNS1): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013

1910 0 PPE-0 MonServiceBinding_192.168.204.51:53_(ping-default)(SVC_CTXANG_DNS2): DOWN; Last response: Failure - Probe timed out. Wed Aug 14 12:27:25 2013


1912 0 PPE-0 'server_svc_cfg_NSSVC_DNS_192.168.204.50:53(SVC_CTXANG_DNS1)' DOWN Wed Aug 14 12:27:25 2013


1906 3910 PPE-0 'interface(1/1)' has been disabled Wed Aug 14 12:27:16 2013

1907 0 PPE-0 'interface(1/1)' DOWN Wed Aug 14 12:27:16 2013






nsconmsg –K newnslog –d consmsg

# nsconmsg -K newnslog -d consmsg

Displaying console message information



current time is Wed Aug 14 12:53:43 2013

LSM 1/1: Mon->Up on Link_Up

platform: NetScaler Virtual Appliance 450000 (9), manufactured at 2/17/2009

platform: serial HE2H91SCZ6

platform: unknown platform - using defaults!

LSM LO/1: Mon->Up on Link_Up

NetScaler: 420Mb of memory allocated

cell_pool: 0, (2 MB, 2 MB)

cell_pool: 0, map_addr 0xd2200000


nsconmsg –K newnslog –d oldconmsg

# nsconmsg -K newnslog.ppe.0 -d oldconmsg

Displaying debug performance information




CPU:2.4% MEM:199419576 UP:00.00:00:14 since:Wed Aug 14 10:16:57 2013






nsconmsg –K newnslog –s ConLb=2 –d oldconmsg

# nsconmsg -K /var/nslog/newnslog -s ConLb=2 -d oldconmsg

Displaying debug performance information




-------------------------------------------------------

NATSession : Free(6552)A(6553)InUse(1)

NATSession: Cur(Tcp[0] Udp[1] Icmp[0] Other[0])

NATSession: Op/s(Tcp[0] Udp[0] Icmp[0] Other[0])

Session: A:0 F:0 IUse:0 SEs: SIP:0 C:0 SSL:0 Svr:0 UserId:0 SIPDIP:0 DIP:0 SO:0

SSF: Conn (Srvr 0 Clnt 0) U:0

CM: Conn (Srvr 0 Clnt 0) Sessions PCB 0 NATPCB 0

Z(SIP[0], C[0], SSL[0] Server[0] SIPDIP[0] DIP[0] SO[0])

Mon: Probes: 0, Failed: 0

VIP(127.0.0.2:53:DOWN:WEIGHTEDRR): Hits(0, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 1024

Pkt(0/sec, 0 bytes) actSvc(0) DefPol(NONE) override(0) newlyUP(0)

Conn: Clt(0, 0/sec, OE[0]) Svr(0) SQ(Total: 0 OnVserver: 0 OnServices: 0)

slimit_SO: (Sothreshhold: 0 [Ex: 0] Consumed: [Ex: 0 Borrowed: 0 TotActiveConn: 0] Available: 0

VIP(127.0.0.2:53:DOWN:WEIGHTEDRR): Hits(0, 0/sec) Mbps(0.00) Pers(OFF) Err(0) SO(0) LConn_BestIdx: 1024

Pkt(0/sec, 0 bytes) actSvc(0) DefPol(NONE) override(0) newlyUP(0)

Conn: Clt(0, 0/sec, OE[0]) Svr(0) SQ(Total: 0 OnVserver: 0 OnServices: 0)

slimit_SO: (Sothreshhold: 0 [Ex: 0] Consumed: [Ex: 0 Borrowed: 0 TotActiveConn: 0] Available: 0-


IP Conflict

# cat /var/log/ns.log

Aug 14 12:39:11 <local0.info> 192.168.47.1 ipConflict (ipConflictAddr = 192.168.47.1, sysIpAddress = 192.168.47.1)

Aug 14 12:39:42 <local0.info> 192.168.47.1 last message repeated 169 times


Troubleshooting TechniquesCommon NetScaler issues/resolutions

HDD Issues Logging fails/var missing

Flash Issues Config fails to saveConfig saves partiallySync failsDevice fails to boot

Memory starvation Dropped sessionsCPU starvation All services failing

All VIPS downDegraded performance


NetScaler Data Collection

NetScaler software version including build – (from the ‘show version’ NetScaler Command Line Interface (NSCLI) command) – example: 10.1.124.13

Production setup or new installation

Whether an application/service that was working is now broken or whether the user wants to configure an application/service

Network topology information

What changes were performed on the NetScaler appliance prior to the issue

Any change(s) on the connected Switches, upstream Router, or backend server prior to experiencing the issue

ns.conf after saving the configuration - (from the save configuration NSCLI command)


show techsupport I

> show techsupport

showtechsupport data collector tool - $Revision: #1 $!

NetScaler version 10.1

Creating /var/tmp/support ....

The NS IP of this box is 192.168.47.1

Current HA state: Primary (or this is not part of HA pair!)

All the data will be collected under

/var/tmp/support/collector_P_192.168.47.1_25Oct2013_11_12

Copying selected configuration files from nsconfig ....

Copying WebInterface configuration files (if WI is installed)...

... WI is not installed on this system. Nothing to copy.

Running shell commands ....

Running CLI show commands ....

Running CLI stat commands ....

Running vtysh commands ....

Determining newnslog files to archive....


... copied 6 files from this directory.

Copying core files from /var/core ...(last 5 files created within the last week)

NSPPE core (2/NSPPE-00-1077.gz) file present! Skipping this file because of size restrictions..


Copying core files from /var/crash ...(last 5 files created within the last week)

... Nothing to copy...No files created within the last one week

Copying messages,ns.log,dmesg and other log files ....

Copying imported files and mapping files ...

Copying GSLB location database files ....

Archiving all the data into "/var/tmp/support/collector_P_192.168.47.1_25Oct2013_11_12.tar.gz"....Done.

Created a symbolic link for the archive with /var/tmp/support/support.tgz

/var/tmp/support/support.tgz ---- points to ---> /var/tmp/support/collector_P_192.168.47.1_25Oct2013_11_12.tar.gz

If this node is part of HA pair, please run it on the other node also!!

show techsupport II


techsupport archive structure

/etc

/flash

/nsconfig

/shell

/var/cfsynclog/clusterd/core/crash/download/log/netscaler/nslog/nsproflog/nssynclog


nstrace & nstcpdump

nstrace

Common syntax:Nstrace.sh –sz 0Nstrace.sh –sz 0 –filter “SOURCEIP = 10.198.4.10” –link enabled

Filter qualifiers and operators:SOURCEIP, SOURCEPORT, DESTIP, DESTPORT, SVCNAME, VSVRNAME, STATE==, eq, !=, neq, >, gt, <, lt, >=, ge, <=, le, BETWEENCompound filters using || and &&

nstcpdump

Common syntax:Nstcpdump.sh –X tcp port 80Nstcpdump.sh –w testcapture.cap –X src host 10.198.4.10 tcp port 80

Filter qualifiers and operators:tcpdump standard


Packet TracingDifferences between nstrace.sh & nstcpdump.sh

Nstcpdump.sh Nstrace.sh

Useful if traces are to be viewed on standard output

nstcpdump.sh –w <filename> option helps writing output to file

Useful for offline collection

nstrace.sh –sz 0 Saves traces in /var/nstrace in cap format

can be used with expressions so that you get to see filtered traffic

nstcpdump.sh host <IP> nstcpdump.sh port 21

useful for collection of traces in separate log files based on NICs

nstrace.sh –tcpdump1 –nic 1


nstrace.sh

# nstrace.sh --?

nstrace - utility to start NetScaler packets trace

usage: nstrace.sh [-h] [-nf <number_of_files] [-time <time>] [-m <mode>] [-nic <boolean>]

-h - prints this message - exclusive option

-nf - number of files to be generated in cycle (def. 24)

-time - seconds per file (def. 3600) (could be an expression)

-sz - size of the captured data (bytes from 60 to 1514)

-m - Capturing mode: sum of the values (def. 18):

-tcpdump - 0=nstrace-format (default) or 1=tcpdump-format

-nic - use separate trace files for each interface

(only works if -tcpdump option is set)

-name - name of the trace file

-filter - Filter expression for nstrace. The maximum length of filter expression is 255 and it can of following format: <expression> [<relop> <expression>]

-link - Log filtered connection's peer's (linked connection's) traffic. Works only with -filter option

-id - ID for the trace file name for uniqueness. Should be used only with -name option

-stop - can be used to disable tracing (when 'nstrace.sh' is run in the background)

#

-h - prints this message - exclusive option

-nf - number of files to be generated in cycle (def. 24)

-time - seconds per file (def. 3600) (could be an expression)

-sz - size of the captured data (bytes from 60 to 1514)

-tcpdump - 0=nstrace-format (default) or 1=tcpdump-format

-nic - use separate trace files for each interface (only works if -tcpdump option is set)

-name - name of the trace file

-filter - Filter expression for nstrace. The maximum length of filter expression is 255 and it can of following format: <expression> [<relop> <expression>]

-link - Log filtered connection's peer's (linked connection's) traffic. Works only with -filter option


Packet Tracing

nstrace.sh nstcpdump.shProprietary capture format TCPdump PCAP capture formatNative format captures more information Useful for live capture from CLIFiles are stored in /var/nstrace Option to write to a fileNeeds custom dissector in Wireshark (1.6+) Most TCPdump options supported

Works in standard Wireshark


Trace analysisnstrace.sh

By Default if nstrace.sh is executed from shell prompt

Trace files are stored in NetScaler proprietary (.cap) format

The trace capture runs for 1 hr. (3600 sec) if not interrupted

Files are cyclically numbered from 1-24. Trace mode is 6 (that is the TXB and RX packets are captured)

The size of the captured data for each packet is 164


Trace Analysisnstrace syntax examples

Command Purpose

# nstrace.sh Stores the traces in default (proprietary) format

# nstrace.sh -tcpdump 1 Begins to save the traces in the TCPDUMP format for a default 3600 seconds

# nstrace.sh -tcpdump 1 -nic 1 Logs the traces (in TCPDUMP format) into separate log files based on the NIC IDs

# nstrace.sh –nf <value> No of files to be generated in cycle by default is 24

# nstrace.sh –time <value> Seconds per file by default 3600 seconds

# nstrace.sh -sz 0 Size of the captured data( by default it is 164), -sz 0 is the entire packet length

# nstrace.sh –m Capturing mode: sum of the values (def. 6):1-Transmitted packets (TX)2 - Packets buffered for transmission (TXB)4 - Received packets (RX)

# nstrace.sh -stop Can be used to disable tracing (when 'nstrace.sh' is run in the background


Trace Analysisnstcpdump.sh example syntax

Command Purpose

# nstcpdump.sh <type> Possible types are host, net and port .If there is no type qualifier host assumed

# nstcpdump.sh <dir> Qualifiers specify a particular transfer direction to and/or from id. Possible directions are src, dst, src or dst and src and dst. If there is no dir qualifier, src or dst is assumed. src foo', `dst net 128.3', `src or dst port ftp-data'

# nstcpdump.sh <proto> ether, fddi, ip, arp, rarp, decnet, tcp and udp.

# nstcpdump.sh –c <value> Exit after receiving ‘value’ number of packets

# nstcpdump.sh –F –I –r Not supported and not to give them as options to the script

# nstcpdump.sh –w <file name>

Write the raw packets to file rather than parsing and printing them out. Read by typing tcpdump –r test


nstcpdump.sh

# nstcpdump.sh --?

Setting 1000 pages (8000 KB) of trace buffers ... Done.

Enabling all nic trace mode=6 ... Done.

Changing trace packet length from 0 to 0 ... Done.

tcpdump version 3.9.4

libpcap version 0.9.4

Usage: tcpdump [-aAdDeflLnNOpqRStuUvxX] [-c count] [ -C file_size ]

[ -E algo:secret ] [ -F file ] [ -i interface ] [ -M secret ]

[ -r file ] [ -s snaplen ] [ -T type ] [ -w file ]

[ -W filecount ] [ -y datalinktype ] [ -Z user ]

[ expression ]

Saving current trace data in file 'pipe' ... in TCPDUMP format

Disabling all nic trace ... Done.


Trace analysis

# nstcpdump.sh host 10.102.12.250

# nstcpdump.sh port 21

# nstcpdump.sh src host 10.102.12.250 and dst host 10.102.12.204

# nstcpdump.sh ‘(host 10.102.12.250 and host 10.102.12.204) or (host 10.102.12.204 and host 10.102.12.31)’

# nstcpdump.sh host storefront.nstipster.lab

Core Dumps/Crashes


Core dump directories

# ls -la

total 8

drwxrwxr-x 4 root nobody 512 Mar 4 09:41 .

drwxr-xr-x 31 root wheel 1024 Dec 4 10:06 ..

drwxrwxr-x 2 root nobody 512 Oct 8 21:28 1

-rw-r--r-- 1 root nobody 2 Mar 4 09:36 bounds

# ls –la 1/

total 10292

drwxrwxr-x 2 root nobody 512 Jan 22 11:24 .

drwxrwxr-x 4 root nobody 512 Mar 4 09:41 ..

-rw------- 1 root nobody 9881665 Dec 18 11:15 NSPPE-00-1094.gz

-rw------- 1 root nobody 603327 Dec 18 11:15 nscac64p-1189.gz


Packet Engine (nsppe) Process Crashing

Oct 19 08:52:46 <local0.alert> vpx1 nsppe: PE 0 (pid 1077) got signal 6; signal mask is 0x0 0x0 0x0 0x0

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NSPPE-00 (1077) unexpectedly died due to receiving signal

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 There may be a delay restarting process while collecting core dump on NSPPE-00 (1077)

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 proc NSPPE-00 (1077) failure. Therefore initiating nCore NetScaler restart according to policy setting (0x29ac)

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 NetScaler restart may be delayed if collecting core dump for NSPPE-00 (1077)

Oct 19 08:52:46 <local0.alert> vpx1 [44]: pitboss Sat Oct 19 08:52:46 2013 Pitboss declaring system failure: NSPPE-00 (1077) exited

Oct 19 08:53:44 <local0.info> vpx1 [44]: pitboss Sat Oct 19 08:53:44 2013 Deleting watch on NSPPE-00 (1077) for ()


/var/core

# cd /var/core/

# ls -la

total 14

drwxrwxr-x 6 root nobody 512 Oct 25 07:54 .

drwxr-xr-x 30 root wheel 1024 Oct 25 10:31 ..

drwxrwxr-x 2 root nobody 512 Sep 30 13:39 1

-rw-r--r-- 1 root nobody 2 Oct 25 07:54 bounds

root@vpx1# ls 1/

NSPPE-00-1077.gz nscac64p-1177.gz nsnetsvc-1086.gz

aslearn-1148.gz nscfsyncd-1158.gz nsrised-1164.gz

imi-1129.gz nsclfsyncd-1160.gz provserverd-1162.gz

monuploadd-1154.gz nsclusterd-1105.gz snmpd-1152.gz

nsaaad-1131.gz nsconfigd-1156.gz

nsaggregatord-1107.gz nsfsyncd-1110.gz


show techsupport

> show techsupport

----SNIP----

Copying core files from /var/core ...(last 5 files created within the last week)

NSPPE core (1/NSPPE-00-1077.gz) file present! Skipping this file because of size restrictions..


Copying core files from /var/crash ...(last 5 files created within the last week)

... Nothing to copy...No files created within the last one week

----SNIP—---


Backup conf files

# cd /nsconfig/

# ls -lath ns*

-rw-r--r-- 1 root wheel 16K Oct 30 14:01 ns.conf

-rw------- 1 root wheel 16K Oct 30 14:01 ns.conf.0


-rw------- 1 root wheel 12K Oct 25 08:09 ns.conf.NS10.1-124.13





Useful Links

CTX109304 Data Collection Procedure to Troubleshoot NetScaler Related Issueshttp://www.slideshare.net/davidmcg/common-pitfalls-when-setting-up-a-net-scaler-for-the-first-time http://support.citrix.com/search/basic?searchQuery=counters&refinement=Content+Type,Technotes&refinement=Product+Family,NetScaler CTX114999 How to Troubleshoot Authentication with Aaad.debughttps://taas.citrix.com/

http://www.slideshare.net/davidmcg/common-pitfalls-when-setting-up-a-net-scaler-for-the-first-time

http://www.slideshare.net/davidmcg/common-pitfalls-when-setting-up-a-net-scaler-for-the-first-time

http://support.citrix.com/search/basic?searchQuery=counters&refinement=Content+Type,Technotes&refinement=Product+Family,NetScaler

http://support.citrix.com/search/basic?searchQuery=counters&refinement=Content+Type,Technotes&refinement=Product+Family,NetScaler


Maximize your knowledge.Continue your journey with Citrix Education.

Recommended next step for hands-on technical training:

CNS-205 Citrix NetScaler 10 Essentials and Networking Identify the capabilities and functionality of the NetScaler Explain basic NetScaler network architectureObtain, install, and manage NetScaler licensesExplain how SSL is used to secure the NetScalerImplement NetScaler TriScale Technology, including Clustering

Visit bit.ly/NSCOURSE to save 10% now through April 30.**Not valid with any other promotions, packages or discounts. Applies only to new purchases. Regional limitations may apply.

http://deliver.citrix.com/WWWE0314TRNGSUPPORTSERIESATTACH.html

http://deliver.citrix.com/WWWE0314TRNGSUPPORTSERIESATTACH.html


AboutCitrix Services

Citrix Services make sureyou succeed with yourvirtualization programs.

How we can help

Citrix Education – The fastest, most efficient way toget your team the virtualization skills they need. Online,on-site or in class.citrix.com/training

Citrix Consulting – Intensive engagements forcomplex, critical or just plain massive projects.citrix.com/consulting

Citrix Support – Always-on support services thatleverage everything we know about best-practicedeployment and maintenance.citrix.com/support

Educate | Guide | Support | Succeed

http://citrix.com/training

http://www.citrix.com/support/consulting.html

http://citrix.com/support


• 40 insider troubleshooting tips

• Covering XenDesktop, XenServer, XenApp and NetScaler

• Citrix Support top engineers

• FREE eBook

• Citrix Auto Support

• Now available!

Secrets of the Citrix Support Ninjas

http://deliver.citrix.com/WWEL1212SUPTNINJASEBOOK


Premier Support Calculator

Check it out

http://www.citrix.com/pscalculator

Work better. Live better. Use NetScaler