Best Practices for Deploying Enterprise Applications on UNIX

Best Practices Deploying Enterprise Applications on UNIX

Noel McKeownDeploy Partners Ltd.

Objective of this GuideObjective of this Guide

• Gain some insight to a UNIX based operating system and the types of tasks a build team or vendors perform.• How to prepare a UNIX server for typical enterprise deployments.• The best practices applied to a UNIX platform in a telco environment.• Some hints, tips, troubleshooting and practical knowledge.

2www.deploypartners.com

AgendaAgenda

• About• Intro• Preparation• Hardening• Troubleshooting


About the AuthorAbout the Author


Noel McKeown:• Principal Consultant/Pre-Sales - Deploy Partners Ltd.• Network and service assurance expert.• Many years consulting on solutions in the enterprise and telecommunications space.• Worked globally from small engagements to large scale transformation projects.• Operational/Business Support Systems consultant.For business enquires:E [email protected] www.deploypartners.com | www.noelmckeown.comP +61 (0)2 9940 0288F +61 (0)2 9940 0478M +353 (0)87 669 3510A Suite 5130 Unit 1, 77 Sir John Rogerson's Quay, | Dublin 2 | Ireland

About Deploy PartnersAbout Deploy PartnersDeploy Partners has a proven track record for delivering best-of-breed Service Assurance solutions.Our solution and service portfolio includes:• Solution Consultancy• Service Assurance Solution Design and Build (OSS)• Service Assurance Maturity Assessment• Business Analytics and Optimisations solutions • IT Service Management Solutions (ITSM and BPM)• Customer Portal Design and Build• Cloud Design and Build and Migration solutions• Infrastructure and Data Centre solutions• Bespoke Business System Application Development (BSS)• Product Training• 24x7 Solution Support


Telstra, NBNCo, Vodafone Australia, Vodafone NZ, SMART, CelCom, AAPT, BCT, First Data, SP Ausnet, Ergon Energy, TransPower, Telecom NZ, Westpac, CBA, Royal Australian Navy, CIOG, IBM GBS, Cisco Services, Telekom Malaysia, SMART, China Unicom, NTT, Harvey Norman, Downer Edi, Tennis Australia, Coles, Singtel, Celcom, Telkomsel, PT Telkom, Colt, Sprint, AT&T…..

Trusted by Blue Chips Globally

Amongst The BestAmongst The Best

We are the IBM C&SI Service Assurance Solution Delivery Experts

Our Technology PartnersOur Technology Partners

LearnQuest

Ask about our deploynow cloud solution

• Where are you in the Service Assurance maturity level range? Talk to a DeployPartners Service Assurance expert about improving your Service Assurance strategy.

• How will disruptive technologies such as Cloud & Data Analytics affect your company’s effectiveness and ROI strategy?

Is your Service Assurance Strategy aligned?Is your Service Assurance Strategy aligned?

1. INTRO1. INTRO

IntroIntroThe UNIX operating system is a set of programs that act as a link between the computer and the user.• Unix was originally developed in 1969 by a group of AT&T employees at Bell Labs.• There are multiple UNIX variants available in the market..• Redhat Enterprise Linux (RHEL) or SUSE tends to be most popular now with service providers and telcos. Solaris used to be the de facto but their hardware prices could not compare with cheaper x86 Linux boxes and the performance of Linux got better and was easier to support.• Several people can use a UNIX computer at the same time; hence UNIX is called a multiuser system.• A user can also run multiple programs at the same time; hence UNIX is called multitasking.


UNiplexed Information and Computing System!!

IntroIntro


IntroIntroEverything in Linux is a file or directory, even if they are unusual. In UNIX there are three basic types of files:1. Ordinary Files: An ordinary file is a file on the system that contains data, text, or program instructions. 2. Directories: Directories store both special and ordinary files. 3. Special Files: Some special files provide access to hardware such as hard drives, CDROM drives, modems, and Ethernet adapters. Other special files are similar to aliases or shortcuts and enable you to access a single file using different names.


2. PREPARATION2. PREPARATION

PreparationPreparation• Typically servers are prepared by an infrastructure group in the organisation with a baseline UNIX image using kickstart or something similar.• The infrastructure team usually configure the server for their environment.• The following areas is a combination of what the infrastructure team and the vendor need to address and/or configure to successfully deploy their application software to best practices:


Naming convention OS installation Baseline server Access MOTD Shared storage RAID Tiered Storage Host file Aliases Profile

Environment variables Tmp System accounts Sudoers DNS NTP/Timezone Security

Iptables SELinux Telnet

Resource limits Package management Configuration management Backups/Snapshots Startup/Shutdown scripts SSH Networking High Availability Manage/Monitor VM Cloning

PreparationPreparation• Naming Convention - It is recommended to devise a naming convention where certain attributes identify a particular server and its function. Mostly the organisation will have a naming standard but sometimes they give the vendor the freedom to create one. • An example of a naming convention:

TIBCO BusinessWorks (TBW):BSSPRDTBWPP01


Attribute Code DescriptionDomain: <OSS><BSS> Functional DomainEnvironment: PRD Production

PPD Pre-ProductionTST TestDEV Development

Function: <FFF> 3 char code to describe functionalityLayer: C Collection

P ProcessingD DisplayA AdministrationN Network

Role: P PrimaryB Backup

Instance: O-N Server number

PreparationPreparationThe following are the typical best practices and tasks when preparing a UNIX server for an enterprise software deployment:• OS Installation - Strip down the operating system to only the essential packages. This is to ensure the server only performs the functions it was designed to. This is done at installation time.• Baseline Server - Take a baseline of the server, I typically use UnixBench. This is good practice to do at the start and before you go live in production. Provides some evidence of the server performance before you started working on it and when it was handed over:https://github.com/kdlucas/byte-unixbench

tar -xzf UnixBench.tgzmake./Run


PreparationPreparation• Access – It is essential to lock down access before you start. • Access to systems is controlled by /etc/security/access.conf• By default this:

• Grants access to all users that are LOCAL• Specifically denies root access remotely• All other access is initially denied.

• It is important that the local console have restricted access, as all users have access via this path by default:# grep access /etc/pam.d/system-authaccount required pam_access.so# cat /etc/security/access.conf+ : ALL : LOCAL- : root : ALL# Add additional role groups here- : ALL : ALL


PreparationPreparation• MOTD - Set the message of the day (MOTD) for SSH logins. The below is a recommended format for an environment with multiple servers with various purposes in a high availability environment.

vi /etc/motdPRECEDENCE (Primary/Backup)PURPOSE (Alias)ENVIRONMENTe.g. PRIMARY PROBE01 PRD

• It would be wise to also include a legal statement for the MOTD banner like so:“This system is restricted to authorized access only. All activities onthis system are recorded and logged. Unauthorized access will be fullyinvestigated and reported to the appropriate law enforcement agencies.”


PreparationPreparation• Shared Storage – It is recommended to use either SAN or NAS shared storage across all your servers in the functional domain. This shared storage will be used to access common files, scripts and configurations across the servers. It can be used as a repository for binaries, documentation and notes.• Shared storage can be very powerful. The distributed approach greatly enhances the enterprise solution from a management, configuration and efficiency approach.• Shared storage can also be used for storing binaries, fixpacks, test fixes, backups and rollbacks for upgrades and configuration changes.• Devise a file structure for the environment that works for all the various software applications and shared files.• P.S. It is not recommended to use Network File System (NFS) if the NFS server will act as a single point of failure.


PreparationPreparation• Example of shared storage in an assurance solution “Tivoli Netcool”:


PreparationPreparation• RAID - Redundant Array of Independent Disks• Install your application on RAID 1+0 SAN mount• This RAID 1+0 (10) Mirrored Striping• Provides the best performance• Requires 4 or more drives• Only 50% of actual capacity used


PreparationPreparation• Tiered Storage - The assignment of different categories of data to different types of storage media in order to reduce total storage cost. • Categories may be based on levels of protection needed, performance requirements, frequency of use, and other considerations.• If would be advised to work out the storage requirements of the application. This can then be used to plan appropriately. Factors could be data volume, the high performance, future capacity and business continuity.


PreparationPreparation• Tiered Storage – If your application is a multi layer architecture, it would be advised to work out the data requirements for each functional component.• Some good practices/examples: o Assign Tier 1 storage to the presentation servers where there is a high volume of user clients. There will be less performance issues, hence less user backlash, greater uptake.o A reporting database that analyses long term and inactive data should use cheaper archiving storage.


PreparationPreparation• Host File – The hosts file is a computer file used by an operating system to map hostnames to IP addresses.• Once your shared storage has been setup. It is advised to set a centralized host file for your environment.• Typically a host file can be shared between a functional domain and environment as there would be a lot of inter server communication in a functional domain; it is essential to have accurate host files. The host file may be widened to include all servers in enterprise if there is a lot of integrations.• The centralized host file makes it easy to update hostnames across the platform as only one file needs to be touched. Create a symbolic link from the local host file to the one on the SAN mount. In example below, the SAN mount is /repo:

cd /etccp hosts hosts.origrm hostsln -s /repo/system/env/hosts hosts


PreparationPreparation• Aliases– Enables a replacement of a word by another string.• Once you’re the centralised hosts file has been configured, you should assign aliases for all your server hostnames. Never use the IP of FQDN.• An example would be an enterprise data warehouse in production; BSSPRDEDHPP01 could become WAREHOUSE01• The hosts file would look like:• 192.168.1.1 BSSPRDEDHPP01 WAREHOUSE01• This gives the server a more friendly name which is easier for communicating with your peers and in mails ..etc. server gets a nickname!• The biggest advantage of this, is you can use the alias name of the server in configurations where the hostname is required. This allows you to use the same configuration across your environments, be it Production, Pre-Production, Test or Development.


PreparationPreparation• Profile – The profile is used to configure the environment globally or for a particular user. The profile sets the environment variables and PATH for processes and the command interpreter.• The PATH specifies a set of directories where executable programs are located.• Create a custom profile script that sets up the environment variables, PATH and aliases. Place this script in the profile.d directory.• The /etc/profile.d directory contains other scripts that contain application-specific startup files, which are also executed at startup time by the shell.• Create a profile script on your shared storage all your servers can use. Create a symbolic link from profile.d to your script:

cd /etc/profile.d/ln -s /repo/system/env/profile.sh profile.shls -l profile.shprofile.sh -> /repo/system/env/profile.sh


PreparationPreparation• Environment Variables - are a set of dynamic named values that can affect the way running processes will behave in the UNIX environment. • Environment variables can dictate the way an application behaves.• It is nearly always necessary to set environment variables for enterprise applications. These will need to be set from when the application starts.• The best method is call your profile script from the previous slide from the application startup script. This will ensure the environment is setup for the applications. e.g. An example Tibco script would be:

vi /etc/init.d/tibco# Environment Setup ## Sources:# Variables# PATH# Aliases. /repo/system/env/profile.sh


PreparationPreparation• Tmp – In Unix and Linux, the global temporary directories are /tmp and /var/tmp. Typically, /var/tmp is for persistent files (as it may be preserved over reboots), and /tmp is for more temporary files.• Most application installers will use the /tmp directory.• To increase security, /tmp can be mounted with nodev, nosuid, and noexecoptions.The installer will not work if noexec/nosuid is enabled on /tmp• Check exec/suid of /tmp before installing the software:

mount | grep '/on tmp‘• Before:

mount -o remount,exec /tmp• After:

mount -o remount,noexec,nosuid,nodev /tmp


PreparationPreparation• Sudoers - Add your system accounts to the sudoers file. This will make it easier to support once the root account has been revoked, This typically depends on the organizations security policy though. In the example, we set the netcool user as a sudoer:

/usr/sbin/visudo -f /etc/sudoers%netcool ALL=(ALL) NOPASSWD: ALL:wq!

• System Accounts - Add your application/system users and groups. It is bad practice to run your applications as root. Running applications as root is seen as a security risk. Having consistent users and groups for you applications increases supportability.


PreparationPreparation• Ex: Adding a user and group of ‘netcool’ with ID of 511. The getent command is just a pre-check to see if the user already exists, it Is not essential but good practice.

getent group netcoolgetent passwd netcoolgroupadd -g 511 netcooluseradd -u 511 -g 511 netcoolpasswd netcoolNew UNIX password: PASSWORDRetype new UNIX password: PASSWORDvi /etc/passwd


PreparationPreparationThe following are some security settings that may need to be configured to enable your software to work.• Iptables - act as the Linux kernal firewall. This can typically be turned off if firewalls are already protecting the environment. Sometimes inter server communication might not work because of iptables and it wont be too obvious.

service iptables stopchkconfig iptables off

• SELinux - is a Linux kernel security module that provides a mechanism for supporting access control. Many enterprise applications require this to be disabled as pre-req. Set SELinux to permissive, this does not deny access, but denials are logged for actions that would have been denied if running in enforcing mode.vi /etc/selinux/configSELINUX=permissiveRestart server for selinux mode to take affect


PreparationPreparation• Telnet - In general users should not connect to a machine using telnet since the password is sent in clear text and, in principle can be captured and used by others. • The SSH program is a secure replacement for telnet and should be used where ever possible.• You disabling telnet by editing the inetd.conf file and commenting out the entry:

vi /etc/inetd.conf#telnet stream tcp nowait root /usr/sbin/tcpd in.telnetdRestart inetd (internet service daemon)cd /etc/rc.d/init.d./inet reload


PreparationPreparation• DNS - Domain Naming Service will be typically required in all environments, especially if the server is network facing and doing lookups based on IPs. Check to make sure it is configured.

# domainnamenslookup <domain># vi /etc/resolv.confdomain deploypartners.comnameserver ipaddressnameserver ipaddress

• Add a DNS round robin friendly name for your portal, admin UI and dashboard servers.• NTP - Network Time Protocol is a networking protocol for clock synchronisation between computer systems. Having correct clocks is critical for most applications. Organisations will have their own NTP server if they do not allow access to the internet from their private server networks.• All NTP servers eventually get their sync from an atomic clock.


PreparationPreparation• You can use clustered NTP servers on the internet. The following was taken from pool.ntp.org . Copy the stanza from the region you are in:


PreparationPreparation• It is important to factor into the solution the timezone that the server resides. Especially if the servers are across large geographic area with different timezones. Some organisations use coordinated universal time (UTC) for all their servers and offset the time in the users client or browser; most enterprise applications support this functionality:

# Set timezonetzselect/etc/timezone/usr/share/zoneinfo//etc/localtime -> binary file referenced by /etc/timezonesudo cp /usr/share/zonefinfo/Ireland/Dublin /etc/localtime# date commanddate MMDDhhmmYYYYTZ=‘Ireland/Dublin'export TZ# System clock used when OS is running# Hardware clock keeps time when system powered off# System clock and hardware clock keep time.hwclock


PreparationPreparation• Package Management - check that your package management is configured. Depending on the organisation, the servers will access the repositories provided by the vendor like redhat or they will maintain their own RPM server internally that will pull updated packages from the internet. If the utility has access to neither, the RPMs needed to be download manually and installed. Enterprise application typically have a few dependency packages , compilers and libraries that need to be installed on the server before installation. Having a utility like yum or apt-get essentially makes this task much easier.• Clean your package repo and update:

yum clean allyum makecacheyum update –y


PreparationPreparation• Resource Limits - is the concept where you regulate several resources consumed by a process on an UNIX operating systems. Although the resource limits are set on a per user basis, they are applied per process basis. Therefore, if a user is executing hundreds of processes, the user may consume huge amount of resources, even if the resource setting values for the user are relatively small numbers.• Some example settings for applications, these may be app specific:

- Set File Limits for Non Root Usersvi /etc/sysctl.conf# Open File Limitsfs.file-max=100000- Set File Limits for Non Root Usersvi /etc/security/limits.conf#<domain> <type> <item> <value>* soft nofile 65536* hard nofile 100000


PreparationPreparation• Some packages recommended to ease the software deployment and management. Subversion is used for configuration change management, dos2unix is used for formatting files that are transferred between UNIX and windows, NMAP is a useful connectivity toolset, libXpm is for using X sessions to remotely launch windows, ksh is the korn shell, screen can be used to prevent timeouts when remotely installing software, kexec-tools contains the kdump service which is used to analyse system crashes. The ‘y’ argument will automatically select YES at the prompt to install.• To install packages:

yum -y install subversionyum -y install dos2unixyum -y install nmapyum -y install libXpmyum -y install kshyum -y install screenyum -y install kexec-tools


PreparationPreparation• Configuration Management – it is highly recommended to use configuration and change management processes and software for the system and application configurations and files.• This mitigates against the risk of catastrophic server failure or an erroneous condition being introduced to the system.• I typically configure subversion(SVN) or GIT for deployments but tools like puppet, hg, chef and go are becoming very popular.• Create a branch in your repository for each environment in the solution and create folders for the servers using the alias name as described earlier.


PreparationPreparation• Backups/Snapshots – If you are taking responsibility for a system that someone else set up, you should review the backup decisions to make sure that you understand them, and that they are still appropriate. • A yearly test restore and review of your backup model is a good idea.• Take regular backup/snapshot of your servers during development and backup before they go live in production and for any patch, upgrade, release or test fix.• Types of backup:

• Full State – A system snapshot used for recovering the entire platform. Useful for disaster recovery or complete replacement of the system. • All Configurations – All system configuration and logical configurations. • System Configurations – All system configurations which include usernames, roles, locales etc. • Logical Configuration – Includes service profiles, VLANs, VSANs, pools, policies etc.


PreparationPreparation• Startup Script – the Init daemon is used to add startup services on the UNIX server. It is critical to have startup scripts for the applications in the case of server failure and reboot.• The init daemon can be used to start the application software on the server. It will also be used to start any custom monitoring software of any other management software installed.• Example of adding custom TIBCO startup script from the share repo. The S99 indicates it will be one of the last processes to start; this is good practice to give the system time to fully boot up.:

- Copy the tipco statup to /etc/init.d/cd /etc/init.d/cp /repo/system/startup/tipco /etc/init.d/chmod +x /etc/init.d/tipcoln -s /etc/init.d/tipco /etc/rc3.d/S99TIBCOchkconfig --list tipcochkconfig --add tipco


PreparationPreparation• Shutdown Script – It is best practice to add scripts to shut down any applications that are installed on a server• This tends to be typically overlooked and usually not configured but to ensure a clean shutdown of your critical applications, kill scripts should be added.• This is an example for TIBCO:

cd /etc/rc0.dln -sf /etc/init.d/tibco K65tibcocd /etc/rc1.d/ln -sf /etc/init.d/tibco K65tibcocd /etc/rc2.d/ln -sf /etc/init.d/tibco K65tibcocd /etc/rc4.d/ln -sf /etc/init.d/tibco K65tibcocd /etc/rc5.d/ln -sf /etc/init.d/tibco K65tibcocd /etc/rc6.d/ln -sf /etc/init.d/tibco K65tibco


PreparationPreparation• Boot/Shutdown Best Practice• Make sure the permissions on your init script are 755. This script need to be owned by root and must not be writeable by others.• The script must not execute other scripts with root, which or not owned by root. Otherwise you have a root backdoor. The subsequent scripts can be changed by non-root users

The “normal” startup scripts need to be executed by the functional user:/bin/su - <functionaluser> -c “control script”For example, starting Netcool:/bin/su - netcool -c “/opt/ibm/tivoli/omnibus/bin/nco_objserv”


PreparationPreparation• Boot/Shutdown Best Practice• Softlinks will be created and managed by chkonfig command. The “init.d” startup script needs to define the details within the header. E.g.

#!/bin/sh## /etc/init.d/mysystem# Subsystem file for "MySystem" server## chkconfig: 2345 95 05# description: MySystem server daemon

• Although these are comments, they are used by chkconfig command and must be present. This particular line defines that on runlevels 2,3,4 and 5, this subsystem will be activated with priority 95 (one of the lasts), and deactivated with priority 05 (one of the firsts).


PreparationPreparation• SSH - is a cryptographic (encrypted) network protocol to allow remote login and other network services to operate securely over an insecure network. • Recommended clients for SSH are PUTTY, MobaXTerm and SecureCRT.• Some recommended settings for SSH are:1. Enable DenyHosts2. Display Banner3. Disable Root Logins4. SSH Passwordless Login5. Enable X11 Session Forwarding6. SSH Keepalives


PreparationPreparation• Enable DenyHosts

• Used to monitor and analyse SSH server access logs for failed login attempts knows as dictionary based attacks and brute force attacks. The script works by banning IP addresses after set number of failed login attempts.• Display Banner

• Display SSH warning messages to UN-authorized access and welcome messages to authorized users before the password prompt and after the user logged in.• Disable Root Logins

• By default Linux systems are per-configured to allow SSH remote logins for everyone including root user itself, which allows everyone to directly log in to system and gain root access. Despite the fact that SSH server allows a more secure way to disable or enable root logins, it’s always a good idea to disable root access, keeping servers a bit more secure.


PreparationPreparation• SSH Passwordless Login

• A SSH Password-less login with SSH keygen will establish a trust relationship between two Linux servers which makes file transfer and synchronization much easier. This is very useful if you are dealing with remote automated backups, remote scripting execution, file transfer, remote script management etc without enter password each time.• Enable X11 Session Forwarding

• Typically, you will not have GUI/Desktop access to these servers and it may even be disabled. A good idea is to enable X11 forwarding so you can remotely launch GUI programs. Essentially you can login to a remote desktop machine and run some X windows program like Gnumeric, Gimp or even Firefox and the program will run on the remote computer, but will display its graphical output on your local computer.• SSH Keepalives

• Sometimes you may have trouble keeping your SSH session up and idle. For whatever reason, the connection just dies after X minutes of inactivity. Usually this happens because there is a firewall between you and the internet that is configured to only keep statefulconnections in its memory for 15 or so minutes. You can set the TCPKeepAlive to keep the session up.


PreparationPreparation• Networking – You can tune the TCP settings to increase the performance of the network interface. I recommend looking online for tweaks you can make for especially busy servers that may be EMS or network element facing.• The TCP time wait interval specifies the length of time that a socket waits for a FIN packet from the sender before forcibly closing. In high-load environments, the server may appear to stall if a large number of sockets remain suspended in the TIME_WAIT state after their connections are closed. Reducing the TCP time wait interval will reduce the number of suspended sockets and, in high-load environments, may prevent the server from appearing to stall.

echo "1024 61000" > /proc/sys/net/ipv4/ip_local_port_rangeecho "5" > /proc/sys/net/ipv4/tcp_fin_timeout


PreparationPreparation• Kdump - is a feature of the Linux kernel that creates crash dumps in the event of a kernel crash. When triggered, kdump exports a memory image (aknown as vmcore) that can be analysed for the purposes of debugging and determining the cause of a crash.• This can be very useful in determining the issue with a problematic server in your environment.• Some guidelines on setting up this service:https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/s2-kdump-configuration-cli.html


PreparationPreparation• High Availability – In a typical enterprise, there will be a Primary Data Center(PDC) and a Secondary Data Center (SDC). Usually these are geographically distributed to mitigate any local disasters like fire, explosions, natural disasters..etc• Usually mission critical applications are deployed in a high availability configuration. It is best practice to cross deploy your primary and backup components across the data centers.• In a dual stack environment, usually the application will have native high availability built into the product for redundancy.• Recently, virtualisation is being used more to provide a HA solution, this saves on licenses and power/cooling as a VM can be booted to take over as the primary component in an architecture and be moved across the infrastructure between data centers.• Server clustering and load balancing is also used in a HA solution and can either be virtual or bare metal solution.• The High Availability solution usually depends on the data center standard operating environment (SOE) that the organisation implements.


PreparationPreparation• Representative drawing of high available architecture with load balancers, clustering and shared storage.


PreparationPreparation• Manage/Monitor – The organisations operations team will want to monitor and manage the infrastructure that the applications sit on.• This can utilise system tools like audit logs, Syslog and SNMP.• In an enterprise, there is usually a monitoring or assurance product installed on the ICT infrastructure to ensure reliability and uptime.• An example of a monitoring solution is IBM Tivoli Monitoring:


PreparationPreparation• Manage/Monitor – A graphic of command line tools to monitor the various planes on a Linux operating system.


PreparationPreparation• Manage/Monitor – There are multiple tools that can be installed to assist in system monitoring. One such tool that is recommended is glances.• Glances is a cross-platform command-line curses-based system monitoring tool written in Python language which use the psutil library to grab information from the system. With Glance, we can monitor CPU, Load Average, Memory, Network Interfaces, Disk I/O, Processes and File System spaces utilization.


https://nicolargo.github.io/glances/

PreparationPreparation• VM Cloning – is a copy of an existing virtual machine. The existing virtual machine is called the parent of the clone. When the cloning operation is complete, the clone is a separate virtual machine.• Setup one or more virtual machines as deployment template for your environment.• Clone this for duplicate TEST, DEV, PILOT or PRODUCTION environments.• Ensure each /etc/hosts is updated with the FQDN name for the server• Setting up aliases as described earlier will ensure the short names in the application configurations will be valid across all environments so no need to reconfigure.


3. HARDENING3. HARDENING

HardeningHardening• Something that was overlooked for some years, hardening a server is essential in any commercial operation to protect the business and their assets.• This is a huge topic, too big for this presentation. We’ll just touch on a few points.• I recommend the following two guides for any enterprise deployment:1. http://benchmarks.cisecurity.org/2. https://www.nsa.gov/ia/mitigation_guidance/security_configuration_guides/operating_systems.shtml• The CIS one is very good, it gives you the command to run to check for the security breach and supplies the remediation to fix.• Yep, the NSA one is actually very good as well, well maybe!!


HardeningHardening• Patches - If possible, configure your software to automatically install security patches. Security patches are often released in response to attacks that are actively being used throughout the Internet.• Accounts - Set a root password that is at least 8-10 characters. Only use the root account for systems administration. Login as yourself and su to root when you are doing systems administration. At least once a year, you should remove or lock accounts that are no longer needed, and make sure that the passwords on active accounts are changed.


HardeningHardening• Password Hashing- Storing all user passwords as cleartext can result in a massive security breach if the password file is compromised. One way to reduce this danger is to only store the hash digest of each password. To authenticate a user, the password presented by the user is hashed and compared with the stored hash.• The example below sets this to the SHA-512 algorithm:

grep CRYPT /etc/login.defsMD5_CRYPT_ENAB noENCRYPT_METHOD SHA512


HardeningHardening• Password Strength and Complexity – The organisations identity and access control will outline the password complexity requirements.• Example:

• Passwords are to be at least 8 characters long. • Passwords are to consist of the characters A-Z, a-z, 0-9.• At least one upper case character and one numeric digit. • Unless exempted in accordance with this guideline, passwords are to expire at least every X days.

# grep 'password requisite' /etc/pam.d/system-authpassword requisite pam_cracklib.so try_first_pass retry=3 minlen=8password requisite pam_passwdqc.so min=disabled,disabled,disabled,8,8 enforce=everyone use_authtok# egrep ^PASS /etc/login.defsPASS_MAX_DAYS XPASS_MIN_DAYS 0PASS_MIN_LEN 8PASS_WARN_AGE 7


4. TROUBLESHOOTING4. TROUBLESHOOTING

TroubleshootingTroubleshootingIn this section, we’ll look at some methods for troubleshooting the following:

1. Gathering details2. Server resources3. Configuration4. Networking5. Connectivity/Firewalls


TroubleshootingTroubleshooting1. Gathering details - If the system or application is performing badly, you will need gather details for the infrastructure team or vendor to troubleshoot:

## Get hostnamehostname –a3# Get redhat releasemore /etc/redhat-releasecat /etc/issue## Determine Hardwaredmesg | grep nexus## Get ENV variables set for particular process e.g. Netcoolps -ef | grep ncocd /proc/23822(cat environ; echo) | tr "\000" "\n" | grep NETCOOL## List all serviceschkconfig --list


TroubleshootingTroubleshooting1. Gathering details - Get operating system and resource details:

## Get System Informationprtdiag –vv## Get OS, Resources and Versionuname -arvcat /proc/meminfocat /proc/cpuinfocat /etc/*leaseulimit -a## uname commandsuname -o // OS typeuname -n // network node nameuname -s // kernel nameuname -m // architectureuname -v // kernel builduname -r // current kernel releaseuname -i // hardware platformuname -a // all system info


TroubleshootingTroubleshooting2. Server Resources - Check how the server resources like CPU, memory and disk usage:

## Check Number of Processes by Non-Root Userps h -Tu netcool | wc –l## Perform Memory Audit: Capture free memory every minute while true; do free -t | grep "buffers/" >> /tmp/free.out; sleep 60; done &## Process Using the Most Memoryps -e -o "vsz size rss pmem time pcpu pid comm" | sort –n## Display Utilisationtoptop -d 1 // change update to 1 secondtop -p PID,PID // Display particular PIDStop -n 2 // Display n updates and quit## Check disk partitions and usage in human readibledf -h


TroubleshootingTroubleshooting2. Server Resources - Check how the server resources like CPU, memory and disk usage:

## Check Memoryfree –m## Check memory for particular processpmap <pid> | grep total## Check Memoryprtconf | grep memoryprtconf | grep "Memory size“ |/usr/bin/awk 'print $3

## Get Size of Directorydu -hs | sort -r –n## Process CPU and Memory utilisationprstat -a -s sizeprstat -T 1 1


TroubleshootingTroubleshooting3. Configuration – Troubleshoot server and application configurations.

VI is a very powerful UNIX editor for searching and editing files:



## Visual DIFF on two filesvimdiff file1 file2



## Check Sumcksum file.txt // First number is checksum, Second number is size of file## Find Large Files e.e. 900Mfind . -size +940000000c## Directory file countls -l | wc –l## Check Sub directoriesls */*## Line Countcat file | wc -l



## Find text in a file e.g. “deploy”find . -type f -exec grep -il deploy \;## Print Subdirectoriesfind . -printtree## User Loginslast -w | more## Command Line historyHistory## Modified in last 24 hoursfind . -mtime -1 -print## Modified in last 30 minutesfind . -mmin -30


TroubleshootingTroubleshooting4. Networking – As applications are very heavily integrated and rely on data from various networks, it may be necessary to troubleshoot the networking:

## snooping an interface for a particular port and PINGstcpdump -i eth2 'port 16311'tcpdump -v -i eth0 src x.x.x.x and dst x.x.x.x and icmp## Check IP Routing Tablenetstat –nr## check if port '27000' in Usenetstat -an | grep 27000## Get Default Routesroute -n | grep 'UG[ \t]' | awk 'print $2‘## Add Routeroute add -net x.x.x.x netmask 255.255.255.0 gw 0.0.0.0


TroubleshootingTroubleshooting4. Networking – As applications are very heavily integrated and rely on data from various networks, it may be necessary to troubleshoot the networking:

## Display network interface configurationifconfig –a## Display routing tableroute# List of portscat /etc/services## Display network hopstraceroutetracepath# Display and configure network interfacesip link show


TroubleshootingTroubleshooting4. Networking – As applications are very heavily integrated and rely on data from various networks, it will be necessary to troubleshoot the networking:

## hostname/etc/hostname## Basic DNS Info for host/etc/host## Address for DNS Server, needs to be IP address:/etc/resolv.conf## Determine order for name resolution/etc/nsswitch.conf## Network Files/etc/sysconfig/network-scripts/ifcfg.eth0## Restart Networking/etc/init.d/networking restart


TroubleshootingTroubleshooting3. Firewalls/Connectivity – As there is a lot of inter server and system communication, it may be necessary to test connectivity and test if firewalls have been configured to allow the traffic:

## Use telnet:Log on to communicating server and telnet to destination IP$ telnet x.x.x.x 2020If you get controlcharacter, it was successful## Use netcatLog on to target server Create listening TCP on desired portconnection:$ nc –l 2020Log on to source serverConnect to listeningport:$ nc x.x.x.x 2020


TroubleshootingTroubleshooting3. Firewalls/Connectivity – As there is a lot of inter server and system communication, it may be necessary to test connectivity and test if firewalls have been configured to all the traffic:

## Test connectivity (ICMP) - Send 1 pingping -c 1 <host>## Use nping:Log on to source server and nping the destination and port$ nping –tcp x.x.x.x –p 2020## If the target has a MIB, you can test SNMPsnmpget -v 2c -c public x.x.x.x sysName.0## If target is Web Server, you can test with CURL or WGETcurl -v https://server:16311/ibm/console/webtop/cgi-bin/filterTagging.cgi


www.deploypartners.comwww.deploypartners.com

www.deploypartners.com

formally Eirteic Consulting (Australia)




•  DeployPartners exist to empower our customers with world class technology while making it look easy to use and manage

•  Have experienced exponential growth over the last 10 years that is continuing to new continents across the globe

•  Very channel focused as a means to grow our business •  Very strong delivery capability and we are well respected as a

solutions provider by major vendors

Our independence and global experience allow us to adapt quickly and provide the right solution at the best price

About Us



•  “IBM 2013 Australia Middleware Partner of the Year” •  IBM Focus Partner for C&SI Cloud and Mobile product suites •  IBM Gold Accreditation, most certified IBM C&SI Partner in the

APAC region

•  The assisting Cisco Partner for Cisco Prime product suite for the APAC region. ( ie Currently expanding our technical expertise with Cisco Prime Fulfillment suite of products through the Cisco training and certification program. )

We are…

We are the Service Assurance Solution Delivery Experts



We provide services for these Best of Breed Technology Vendors

Our focus Technology Partners

LearnQuest



5

Successful, timely completion of projects across diverse markets

–  Telco/Internet Service Providers

–  Financial Services & Banking

–  Utilities (Power, Water & Transport)

–  Military & Defense

–  Retail & Sports

Small enough to move quickly -- large enough to have depth



6

Telco/Internet Service Providers

Ncell




7

Telco/ISP’s

•  End-to-end service management (Telephony, IPTV, VoIP, Internet etc.) with real-time monitoring

•  An improvement in process efficiency - (in resolution time by correlation and

root cause analysis)

•  Eliminating insidious or repetitive failures on high priority services

•  Reduction in CAPEX and OPEX

•  Automating the provisioning and activation process of services, reducing time to market

•  Customer portals for self-service, with automated workflows

•  Improved service availability through proactive network monitoring, service enrichment and predictive analytics.



8

Financial Services & Banking

•  Real-time, transactional level monitoring and reporting

•  Ability to display business impacts which result from an outage.

•  Improved customer experience through consistent service availability.

•  Provide dynamic thresholding and performance analytics to improve incident avoidance. This proactive monitoring permits incident resolution before it impacts users, business applications or business services.

•  Collects data you can use to drive timely performance and capacity planning activities to avoid outages from resource over-utilization.




9

Utilities

•  Improved prioritisation of any fault action – by correlating the business service priority (and therefore business impact) to a communications failure event.

•  A reduction in truck rolls – improving the probability of having the right person

dispatched to the right job. •  Operational Health and Safety / Regulatory

o  Minimize Pole Climbs reducing OH&S exposure and resulting Insurance and compensation costs. o  Avoid regulatory fines and class actions resulting from poorly tuned/maintained

equipment related fire.




10

Defence & Military

•  Ensuring high levels of availability and security of Defence enterprise service bus both for internal and external partners such as the Joint Strike Fighter alliance.

•  Discovery, monitoring and management of Defence terrestrial, remote and tactical networks.

•  Network asset and software license management for cost recovery.

•  Deployment of Defence Operational Platforms for network and server monitoring in standalone environments such as remote or mobile bases and on ships.

Depended upon by Defence



11

Retail & Sports

•  Proactively monitor performance of online, mobile, and fixed applications and infrastructure with automatic trend and threshold based alerting.

•  Detect, diagnose, & resolve network issues before users know there's an issue. •  Understand how applications are using shared resources such as

databases, storage, and the network.



12

Tool Set – C&SI/Tivoli Portfolio

•  C&SI Netcool Operations Insight o  Analytics for unstructured data

•  C&SI Predictive Insights o  Predict application, middleware of infrastructure problems

•  C&SI Netcool/OMNIbus and Impact o  Fault Management and automation engine

•  IBM Tivoli Network Manager (ITNM) o  Network discovery

•  Tivoli Netcool Performance Manger (TNPM) •  Network performance polling

•  Smart Cloud Application Performance Manager (SCAPM) •  Application performance monitoring

•  Tivoli Netcool Configuration Manager (TNCM) •  Configuration Management of Network Devices



13

Tool Set – C&SI/Tivoli Portfolio cont •  Tivoli Business Service Manager (TBSM)

o  Service modelling

•  Smart Cloud Control Desk (SCCD) o  Asset & Service Management

•  IBM Tivoli Storage Productivity Center (TPC) o  Storage Monitoring & Provisioning

•  Tivoli Application Discover & Dependency Manager (TADDM) •  Application relationships and inter-dependancies

•  IBM Tivoli Storage Manager (TSM) o  Storage Backup & Recovery

•  IBM Cloud Orchestrator (SCO) o  Cloud Management

•  IBM Endpoint Manager (IEM) o  Device Management



14

Tool Set – Cisco Prime Portfolio

•  Cisco Prime Central o  Centralized access & user interface

•  Cisco Prime Provisioning o  Service Lifecycle

•  Cisco Performance Manager o  Network Performance Monitoring

•  Cisco Prime Optical o  Automated configuration & provisioning

•  Cisco Prime Network o  Device operation, administration, and fault management



15

Tool Set – Edge Technologies

•  enPortal o  Vendor-neutral network management

integration platform o  Transforms event data to service information o  Integrates data from service-oriented

applications o  Provides simple visualizations of information

derived from disparate data sources

•  AppBoard o  Build interactive dashboards through a

graphical configuration interface o  Secure & scalable o  Data adapters connect to various files,

application API's, and databases o  Accelerates development and reduces

implementation costs of dashboard solutions o  Wizard-style configuration



16

Tool Set – Analytics

•  Smart Cloud Analytics - Predictive Insights

o  Early problem detection

•  Smart Cloud Analytics - Logfile Analysis o  Analysis of unstructured data to help identify, isolate

and resolve problems

•  DeployPartners Context o  Knowledge Base o  Analytics o  Event Enrichment



IT Service Management (ITSM)

•  Implementation and management of quality information technology services

•  Improvement in people, process and information technology

•  Alignment with ITIL best practice methodology •  Incident Management •  Problem Management •  Change Management •  Service Requests •  Service Level Management •  Asset Management •  Knowledge Base •  Service Catalog •  Process Management •  Configuration Management

•  DeployPartners has vast experience developing and implementing ITSM solutions to meet the customers specific needs

•  IBM Smart Cloud Control Desk (SCCD) •  BMC Remedy and Atrium Orchestrator



Software Development

•  DeployPartners has a capable and versatile Software Development team

•  We have extensive experience in developing and implementing OSS and BSS solutions to meet the customers specific needs

•  Our solutions include:

•  RFQ (Telstra Cable Modem Performance Management solution)

•  Noderly (“deploynow availability monitoring tool”)

•  RFSR (Radio Frequency Schematic Report - Schematically representing the Telstra mobile base-station design database eliminating the reliance on CAD drawings)

•  Whisper (Dynamic workflow, task and information management providing a paperless environment for the commissioning of Telstra’s mobile base-Stations)

•  Context (Service Assurance Analytics Tool and Knowledgebase)



19

Training DeployPartners offer comprehensive training classes for all our Technology Partner products.

•  Top of the range, in-house training facilities.

•  Best available experts in the field.

•  Vendor-certified trainers and experienced engineers.

•  Provide real-world scenarios and anecdotes throughout the training.

•  Focus technologies include Tivoli CSi products, Cisco Prime products and the Edge Technology Portals and Appboards.



20

Delivery DeployPartners provide a proven delivery approach to guarantee Business outcomes

•  Industry standard methodologies - Prince II,

PMBOK, Agile. •  Repeatable solutions and the best skilled

resources in the industry. •  Project & Program Management to support

small engagements through to large programs. •  Extensive Multi Sector & Gov’t experience in

Australia and Pacific region. •  Certified PM Practitioners (PMI & AIPM, Prince

II) with 20+ years industry experience. •  Quality Management (ISO Compliant) –

continuous improvement approach.

Projects delivered On Time on Budget



21

Support

DeployPartners Support Services consolidate your organization’s support requirements with a single, experienced support provider and partner

Benefits •  Maximize your support investment in current tool sets •  Unlock the full value from your existing products •  Peace of mind – 24x7 access to Support expertise •  Rapid problem resolution with services based on ITIL & ITSMF methodologies and best practices •  Control your upgrade strategy •  Multiple tiers of Support to meet your requirements •  Access to DeployPartners KnowledeBase •  Access to the third party vendors partner website for documentation, patch/bug fixes, software updates and problem escalations. •  Outsourcing for support and administration services



Ask about our deploynow cloud solutions

•  Where are you in the Service Assurance maturity level range? Talk to a DeployPartners Service Assurance expert about improving your Service Assurance strategy.

•  How will disruptive technologies such as Cloud & Data Analytics affect your company’s effectiveness and ROI strategy?

Is your Service Assurance Strategy aligned?


