SAITJS Complete

8/6/2019 SAITJS Complete

1/66

sponsored by

Solutions for

AutomatingIT Job Scheduling


2/66


3/66

SolutionsforAutomatingITJobScheduling GregShields

ii

Introduction to Realtime Publishers ................................................ ............................................. .................... i

Ch apter 1: Do I Need Job Scheduling? Ten Questions to Ask Yourself ............................................ .. 1The Pain of the NonHomogeneous IT Environment .................................................. ......................... 2Constructing an Unfriendly Application ........................................... ............................................. ............ 4Defining IT Job Scheduling .............................................................................................................................. 7

Te n Questions to Ask Yourself .................................................................................................................... 101. How Much Time Have You Wasted in Completing Tasks Manually? ............................... 102: What Is the Cost of an Error Incurred During a Manual Task? ........................................... 103: Where Can You Go for a HeadsUp Display of IT Activities? ................................................ 114: How Do You Manage CrossSystem Communications? ........................................... ............... 125: Are You Concerned About Idle Time During a Task? .............................................................. 136: How Many Tasks Exist in Your Environment that You Dont Know About? ................. 137: Can You Monitor IT Tasks Across Every Team, Platform, and Application? ................ 14

.......... 148: Build Versus Buy: Is a Homegrown Scheduler Good Enough? ..................................9: What Are Your Steps for Linking the Results of One Task with the Actions ofAnother? .......................................................................................................................................................... 15

10: How Do You Handle Errors in a CustomCoded Script? ...................................................... 15Do You Need Job Scheduling? ...................................................................................................................... 15Ch apter 2: Seven Use Cases for Automating IT Job Scheduling .................................................. ....... 16

Seven Stories in IT Job Scheduling ............................................................................................................ 17Story #1: Five AdministratorsFive Scheduled Tasks Libraries ............................................. .. 17Story #2: Consolidating Tasks from Every Application ............................................ ....................... 18Story #3: The Script that Wasnt a Job..................................................................................................... 20Story #4: When Gathering Data Is More than Gathering Data ............................................... ....... 22Story #5: Controlling Data Transfer as a Workflow .......................................................................... 24Story #6: Complex Jobs and the Need for Triggers ............................................................................ 26

Story #7: Securing the Data Center from the Ease of Script Execution .................................... 28


4/66


iii

Job Schedulings Stories Are its Value Add ............................................................................................ 30

Ch apter 3: What Makes an IT Workflow? A Technical Deconstruction ......................................... 31A W orkflow Is an IT Activity, Quantified ................................................................................................ 31

Capturable....................................................................................................................................................... 32Monitorable and Measurable ............................................. ........................................... .......................... 33Repeatable and Reusable ......................................................................................................................... 34Securable ......................................................................................................................................................... 35A GroundUp IT Workflow Construction ................................................... ........................................... .. 36Job Libraries and the Value of Triggers .................................................................................................. 44An IT Workflow Tells the Computer What to Do ............................................................................ 45

Ch apter 4: Implementing Enterprise Job Scheduling: A Requirements Checklist .................... 46Cr eating a Requirements Spec for Job Scheduling ............................................................................. 46

Requirement #1: The Solution Shall Integrate with All Platforms and Applications thatComprise the Business System .............................................................................................................. 48

Requirement #2: The Solution Shall Expose All Properties and Methods Used by Those...................... 48Systems ......................................................................................................................................

Requirement #3: The Solution Shall Include Support for Script Language

49Independence ................................................................................................................................................Requirement #4: The Solution Shall Support Queues for Job/Plan Prioritization andPerformance Management ...................................................................................................................... 49

Requirement #5: The Solution Shall Support FileBased Triggers ........................................ 50Requirement #6: The Solution Shall Support MessageBased Triggers .............................. 50Requirement #7: The Solution Shall Support EventBased Triggers .................................... 51

1Requirement #8: The Solution Shall Support TimeBased Triggers ..................................... 5Requirement #9: The Solution Shall Support Variables and Dynamic Data Across Jobs

...................... 51and Plans ...................................................................................................................................Requirement #10: The Solution Shall Support Intra and InterWorkflow

...... 52Communication ......................................................................................................................................Requirement #11: The Solution Shall Support Job/Plan Execution via a BusinessCalendar ........................................................................................................................................................... 52


5/66


iv

Requirement #12: The Solution Shall Support and Report on Job Object Dependencies ............................................................................................................................................................................. 53Requirement #13: The Solution Shall Include Client, Browser, and Mobile Device UserInterfaces ......................................................................................................................................................... 54Requirement #14: The Solution Shall Support Nesting, Chaining, and Load Balancing ofJobs and Plans ............................................................................................................................................... 54

e 55Requirement #15: The Solution Shall Use an ObjectOriented Management InterfacRequirement #16: The Solution Shall Use a Centralized Job and Script Library that

. 56Includes CommonlyUsed BuiltIn Items .........................................................................................Requirement #17: The Solution Shall Consolidate Error and Output Messages fromExecuted Jobs ................................................................................................................................................ 57Requirement #18: The Solution Shall Support a Centralized Security Model Across Jobs

and Plans ......................................................................................................................................................... 58

Requirement #19: The Solution Shall Include a Centralized Change ManagementModel........ 58.....................................................................................................................................................................

Requirement #20: The Solution Shall Include a Centralized Audit Database thatIncludes Metrics and Alerting ................................................................................................................ 59

So, Do You Need a Job Scheduling Solution? ......................................................................................... 60


6/66


v

Copyright Statement

2011 Realtime Publishers. All rights reserved. This site contains materials that havebeen created, developed, or commissioned by, and published with the permission of,Realtime Publishers (the Materials) and this site and any such Materials are protectedby international copyright and trademark laws.

THE MATERIALS ARE PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND,EITHER EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO, THE IMPLIEDWARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE,TITLE AND NON-INFRINGEMENT. The Materials are subject to change without noticeand do not represent a commitment on the part of Realtime Publishers its web sitesponsors. In no event shall Realtime Publishers or its web site sponsors be held liable fortechnical or editorial errors or omissions contained in the Materials, including withoutlimitation, for any direct, indirect, incidental, special, exemplary or consequentialdamages whatsoever resulting from the use of any information contained in the Materials.

The Materials (including but not limited to the text, images, audio, and/or video) may notbe copied, reproduced, republished, uploaded, posted, transmitted, or distributed in anyway, in whole or in part, except that one copy may be downloaded for your personal, non-commercial use on a single computer. In connection with such use, you may not modifyor obscure any copyright or other proprietary notice.

The Materials may contain trademarks, services marks and logos that are the property ofthird parties. You are not permitted to use these trademarks, services marks or logoswithout prior written consent of such third parties.

Realtime Publishers and the Realtime Publishers logo are registered in the US Patent &Trademark Office. All other product or service names are the property of their respectiveowners.

If you have any questions about these terms, or if you would like information aboutlicensing materials from Realtime Publishers, please contact us via e-mail [email protected].
mailto:[email protected]:[email protected]


7/66


1

[EditorsNote:This book was downloaded from Realtime NexusThe Digital Library for ITrofessionals. All leading technology books from Realtime Publishers can be found atttp://nexus.realtimepublishers.com

Ph .]

Chapter1:

Do

INeed

Job

Scheduling?

Ten

QuestionstoAskYourselfI still remember that day the boss walked into my office. And I still remember her fatefulyet enticing words that talked me into tackling the project, TheProjectThatWouldChange

sEverything. Youre the Scripting Guy. Youll make it work, she said. I agreed, and the rest ihistory.

Youre probably familiar with this story: Big projects in IT start that way. Big projects thatmay seem great on paper but then get very, very complex as theyre developed. Big projects

onwhose multiple applications across multiple platforms require substantial integratieffort.

But this story isnt only about that one big project. Its also about all the other littleautomationsscripts for Windows and Oracle and Active Directory (AD), SQL packages,Linux cron jobs, and so onthat creep into every IT environment over time. Thoseautomations absolutely solve a business immediate needs. But without centralmanagement, they also come with a cost. That cost arises as scripts age, technologychanges, and the script owners relocate or leave the company.

Back then, I was known as The Scripting Guy. If you needed a quick data transformation, ora scheduled movement of files from one system to another, I was your goto person. I had

developed a command of the major scripting languages, along with all the other necessaryaddons one needed to be The Guy. Over the course of several years, my integrationMI,prowess had grown to include platform and applicationspecific technologies such as W

ADSI, SQL, some Oracle, and even a bit of Linux and IBM AIX.

I put that knowledge to what I thought was good use. Over the years, I had gotten to thepoint where much of my daily responsibility was automatedsort of. Sometimes my littleautomations broke. Sometimes they were accidentally deleted or otherwise wiped outthrough the regular change that happens in any data center. Sometimes their need wentaway, or a servers configuration was updated, and as a result, I had to go find them onceagain and remember what they were intended to do.

The result was a notverywelloiled machine; one that created even bigger problems theday I moved on to a new employer. You see, I had little automations scattered around the

heard, theyre still finding them years later,company servers with my name on them. Last Iusually after some process breaks that nobody ever knew existed.

You can probably guess what we were missing. WeneededenterpriseITjobscheduling.Thats why Im writing this book, to explain what this approach is and help you realize youcould probably use it as well. Throughout this book, I intend to return back to that bigprojects story along with a few of the other little ones to show you why.
http://nexus.realtimepublishers.com/http://nexus.realtimepublishers.com/http://nexus.realtimepublishers.com/http://nexus.realtimepublishers.com/


8/66


2

ThePainoftheNonHomogeneousITEnvironmentThis book wouldnt exist if every IT technology seamlessly talked with each other,transferring data, events, and instructions across platforms and applications with ease. Ifevery technology could perfectly schedule its activities with itself and others, you wouldntbe reading these pages.

But you are, and consequently this guide indeed exists.

It exists because IT job scheduling is a task that every enterprise needs, as do many smalland midsize organizations as well. Jobs in this sense represent those little packages ofautomation I discussed earlier. Some work with a single system. Others integrate theservices of multiple systems to create a kind of mixed workload that produces a resultyour business needs.

Consider a few of those jobs youre probably already creating today:

Reports about activities on an email system need to be collected and distributed to

security groups

Customer files transferred inbound to a Linux FTP server need to be ingested uponreceipt into a SQL database

Provisioning AD users requires several additional steps across systems such asLinux, email applications, and Oracle databases

New records in a Microsoft SQL or Oracle database trigger actions to occur in one ormore middleware system

Your jobs might be less complex, working with only a single system or application. Or theymight be exceedingly so, requiring the participation of multiple applications across

different platforms in different parts of the world. At issue here isnt necessarily howcomplex your jobs are. The simple ones have many of the same requirements andnecessitate as much due diligence as the complex ones. Rather, the issue has more to dowith the workflow that surrounds those jobs, and the solutions you implement to manage,monitor, audit, and otherwise keep tabs on every activity at once.

It also has to do with the very different languages and techniques that each IT technologyuses and requires. Those differences represent a big headache inside todaysheterogeneous data centers. Your IT operating environment surely has Windows systems.But it probably also has Linux, Oracle, HPUX, Solaris, and others. You probably need totransfer XML documents, DOCX files, and XLSX spreadsheets over multiple protocols like

SMB, FTP, and SSH. Even your monitoring comes in many flavors: SNMP for switches androuters, WMI for Windows systems, and all the various UNIX and Linux widgets for keepingtabs on their activities.


9/66


3

You can imagine just a few of the headaches these radical differences in applications andplatforms create:

languageEvery operating system (OS) speaks its own

Every application speaks its own language

Every application and OS uses its own security model that doesnt necessarilyintegrate well with each other

Transferring data or scheduling actions within and between each is difficult andsometimes impossible with native tools alone

Most importantly, centrally managing every job everywhere just isnt possible

Solving these five problems is the primary mission of an IT job scheduling solution. From acentral location (see Figure 1.1), an IT job scheduling solution creates a platform on whichto run all your little automation packages that might otherwise be spread acrosstechnologies. Using a centralized solution, a database job, a UNIX/Linux job, and an FTP job

are all parts of the same management framework. All begin their execution from the sameplace, and all are managed and monitored from that single location.

Figure1.1:AnITjobschedulingsolutioncentralizesjobsofalltypes,acrossall

platformsandapplications.

As you can probably guess, such a solution has to be exceptionally comprehensive. Asolution that works for your company not only needs to support your management,

monitoring, and workflow needs. It must be more than just a morepowerful version of theWindows Task Manager. It also needs to support the integrations into every OS, platform,and technology that your business processes incorporate. Thats why finding a solution for

ctivity.automating IT job scheduling can be such a challenging a


10/66


4

To help you out, you can consider this guide to be a kind of automation idea factory. Itsfour chapters will present you with questions to ask yourself, helping you frame your needfor a job scheduling solution. It delivers a set of realworld use cases for seeing schedulingin action. It deconstructs an IT job so that you can peer inside its internal machinery andunderstand the power of a centralized solution. And it will conclude with a checklist of

requirements you should consider when seeking the software that creates your solution.Ill be your guide, and throughout this process Ill share a few of my own stories to bringsome realworld experience into this complex topic.

Note

In this book, youll hear me use the termjobscheduling. Another commonlyused term for job scheduling is workloadautomation. For the purposes of thisbook, you can assume that the two are interchangeable.

ConstructinganUnfriendlyApplication

Now, back to my story from long ago. Every OS and application comes equipped withmultiple ways to perform its core functions. You already know this. An OS includes one ormore scripting languages to enact change and read data. Every modern database has itsown scheduling and automation functions, enabling the creation of packages for insertingand selecting data. Even middleware technologies and applications have their own APIs,which can be interfaced either inside or outside the application.

But the internal languages and automations that come with a product are rarely equippedto handle actions outside that product. Ever try to use an XML document to instruct a SQLServer to update an Oracle database row so that an SAP application can provision a processto an AIX mainframe? Whew! Thats pretty close to the situation I experienced as I started

on TheProjectThatWouldChangeEverything.

Lets start with a little background. Why that project was needed is really unimportant, asis what we were doing with its data. What is important, however, are the interconnectionsbetween each of its disparate elements. Multiple applications running atop more than oneOS, integrating with different databases, and requiring data from both inside and outsidethe organization was just the start.

To get going, I attempted to diagram its components, creating something close to what yousee in Figure 1.2. At a high level, this system was constructed to aggregate a set of datafrom outside our organization with another set on the inside. Our problem was the many

ifferent locations where that data needed to go.d


11/66


5

SQL DatabaseIIS Server

Email Server

Oracle DatabaseSAP Server

Active DirectoryServer

UNIX Mainframe

FTP Server

External

Data Source

User

Figure1.2:Myunfriendlyapplication.

Let me break down the mass of arrows you see in Figure 1.2. The flow of data in this systemstarted via an FTP from an external data source. That data, along with all its metadata,needed to be stored in a single, centralized SQL database. There, permissions fromWindows AD would be applied to various parts of the data set. Some data was appropriatefor certain users, with other data restricted to only a certain few. Information inside theFTP data stream would identify who should have access to what.

Users could interact with that data through a Microsoft IIS server running a homegrownWeb application. That Web application used XML to transfer data to and from the SQLdatabase. Certain types of data also needed to be added to our company SAP system

running atop Oracle, requiring data transformations and delivery between those twosystems.

Occasionally, portions of that data would need to be ingested into a UNIX mainframe forfurther processing. There, it would be consolidated with data from other locations forgreater use elsewhere in the company. An email server would ensure users were notifiedabout updates, new data sets, and other systemwide notifications.


12/66


6

Thatsalotofarrows, and each of those arrows represent an integration that needs to belaid into place in order for the entire system to function. Each arrow also represents anactivity that needs to happen at a particular moment in time. Data heading towards theOracle database obviously couldnt be scheduled to go there until it was actually received atthe SQL Server system. Users shouldnt be notified unless something important to them

was actually processed. Just the scheduling surrounding each arrows integration was acomplex task unto itself.

Does this look like one or more of the systems that are currently in your data center? Ifyoure doing much with data transformation and movement, you might have the samescheduling headaches yourself. Thats why there are four critical points that are importantto recognize:

Youcanfindinterconnectedsystemslikethiseverywhere. In companies across theglobe, IT systems are constructed in ways just like this, using technologies just likethese. So although this project would indeed change everything for my company,others elsewhere are already dealing with slightlydifferentbutmostlythesameintegrations for other reasons.

Constructingandmanagingthissystemisnotnecessarilyanactivitylimitedtojustdevelopers. Im not a developer. Im a systems administrator who just happened tobe considered The Scripting Guy. Thus, in the minds of my superiors, I made theperfect candidate for leading this project to completion. The same holds true withthe worlds other systems as well. IT administrators are tasked with constructingsystems that are made up of many moving parts. Getting them all to talk with eachother and schedule their activities is only the first step in that battle.

Itisentirelypossibletobuildthisusingscriptingandapplication specificautomations,butdoingsoisareallybadidea. My strewnabout SQL SSIS packages and VBScriptsand cron jobs never prepared me for the administrative overhead of a real systemrather than some jumble of unconnected actions. Successfully scheduling theactivities across every piece of the infrastructure isnt a task that works well unlessits centralized. Even worse, managing and maintaining those automations becomesa risk to the systems operation as they grow in number and get more complex.

Theschedulingofactivitiesinasystemlikethisrequiresmorethanjusttheclockonthewall. Think about the data flow between these individual components, and theevents that need to trigger other events. Simple actions inside simple systems mightwork just fine with a time and datebased schedule. Is it 3:00AM? Start the backups!But systems with greater complexity and more dependencies require more

powerful means of determining when to do something. Those decisions can be madebased on receiving a piece of data, seeing a change, reading a log file, or any of themyriad other state changes that occur in any IT system. A good IT job schedulingsystem will give you plenty of conditions that you can customize for identifyingwhen an action should occur.


13/66


7

It is the combination of these four realizations that helped me understand that I needed tostep outside my applicationspecific mindset. It helped me realize I needed to look tosolutions that schedule activities across every platform and every application. Thats whenI started looking into enterprise IT job scheduling solutions.

DefiningITJobSchedulingLets now take a step back from the storyline and think for a minute about what IT jobscheduling should be. Ive already suggested that a job represents some sort ofautomation that occurs within an IT system. But lets get technical with that definition. Isubmit that an IT job represents anactiontobeexecuted. An IT job might be running abatch file or script file. It might be running a shell command. It could also be the executionof a database job or transformation. Essentially, anything that enacts a change on a systemis wrapped into this object well call a job.

Using an objectoriented approach, it makes sense to consolidate individual actions into

separate jobs. This singleactionperjob approach ensures that jobs are reusableelsewhere and for other purposes. It means that I can create a job called Connect to OracleDatabase and use that job any time I need to make an Oracle database connectionanywhere.

Now if each job accomplishes one thing, this means that I can string together multiple jobsto fully complete some kind of action. Ill call that string an IT plan. A plan represents aseriesofrelatedjobsthatcanbeexecutedwith the intended goal of carrying out somechange. Figure 1.3 shows a graphical representation of how this might work.

Figure1.3:Multiplejobsareconnectedtocreateaplan.

In Figure 1.3, you can see how three different jobs are connected to create the plan. Job 27connects to an Oracle database. It passes its result to Job 19, which then extracts a set ofdata from that database. Once extracted, the data needs to be sent somewhere. Job 42completes that task, as it FTPs the data to a location somewhere.

Theres obviously an art to creating good jobs. Thats a topic that Ill discuss in greaterdetail in Chapter 3, but I need to introduce some of the basics here. A good job, for example,might not necessarily have any specific data or hard information thats stored inside thejob. Rather than a connection string to a specific server for Job 27, a much better approachwould be to use some kind of variable instead.


14/66


8

Developers use the techie term parameterization to represent this generalizing of jobobjects and the subsequent application of variables at their execution. Figure 1.4 showshow a parameterized plan can link three generic jobs. At the point this plan is run, thosejobs are fed the variable information they need to connect to the right database, extract theright data, and eventually pass it on to the correct FTP site.

Figure1.4:Feedingparameterstojobsinaplan.

By parameterizing the plan in this way, I now get reusability of the plan in addition to allthe individual jobs that make up that plan. Should I down the road need to attach to adifferent database somewhere, pull off a different set of data, and send it to some other FTPsite, I can accomplish this by reusing the plan and modifying its variable information. Thatsreusability on top of reusability!

Theres obviously quite a bit more to this whole concept of working with jobs and plans. Illspend more time in Chapter 3 helping you understand the various characteristics that canbe assigned to a job and a plan as well as other objects a typical IT job scheduling solutionwill use.

But there is one characteristic that merits attention before moving on. That characteristic isthe schedule itself, which needs to be applied to the object to tell it when to run. Imentioned earlier that scheduling for large systems like The Project requires a kind offlexibility you just cant get by looking at the clock on the wall. Rather, the kinds of jobs thatproject needs tend to be more related to actions or state changes that occur within the

system.


15/66


9

Lets assume that Figure 1.4s Plan 7 relates to some data transfer that needs to happeninside The Project. In this case, lets assume that the data transfer occurs between its SQLServer and UNIX mainframe. Figure 1.5 shows a graphical representation of how this mightbe applied. There, you can see how three different schedules could potentially be attachedto the newlycreated plan:

Invoke the plan at 3:00PM

Invoke the plan when a set quantity of new data arrives

Invoke the plan when processor utilization is less than 30%

Figure1.5:Applyingascheduletoaplan.

Any of these three schedules can be appropriate, depending on the needs of the system andits components. For example, the first schedule might be appropriate if a daily data dump isall thats necessary. In that case, a date/timecentric schedule might be all thats necessary

to complete the action. Very simple.

The second and third tasks highlight some of the more powerful scheduling options thatcould also drive the invocation of the plan. In the first, the plan is executed not based onany time of day. Rather, it executes when a set quantity of new data has been added to thedatabase. This could be a smart solution if you want these two databases to stay roughly insync with each other. It is really powerful when you consider how difficult that kind of

lone.scheduling would be to create if you were using just the native SQL or UNIX tools a

That third schedule is particularly interesting, because it could be used alone or incombination with the second. That third schedule instructs the plan to run only if the

server isnt terribly busy. Using it in combination with the second allows you to maintain alevel of synchronization while still throttling the use of the server. A good job schedulingsolution will include a wide range of conditions that you can apply to plans to direct whenthey should kick off.


16/66


10

TenQuestionstoAskYourselfAgain, Ill dive deeper into this deconstruction of an IT workflows components in Chapter3. But before you can truly appreciate the power of this modular approach, there are likelya few questions that youre probably asking yourself. If youre not, let me help you out witha list of ten good questions about your own environment that you should probably ask

yourself. Your answers to these ten questions will determine whether youll want to turnthe next chapters. If youre experiencing zero headaches with the tools you have today for

activities, you wont need the rest of this book.scheduling your IT

Everyone else will.

1.HowMuchTimeHaveYouWastedinCompletingTasksManually?

It wasnt many years ago that one of my jobs was in keeping a set of servers updated.Monthly updates were de rigueur with some on even shorter schedules. Each came with avery short time window when they could and should be applied. The big problem resultedfrom the fact that these updates typically required a server reboot to get them applied.

At that time, our reboot window was in the wee hours of the morning, many hours past theusual 8to5 workday. For me, sticking around once a month to complete these updatesrepresented a hardship on self and family. Thats why I created my own automation thatwrapped around these updates installation. For my solution, when updates were droppedinto a particular location, they were applied at the next window. My mobile device notifiedme should any problems occur.

From that part on, adding updates to servers meant simply adding them to the rightlocation and making sure my mobile device was near the bedside. Yes, sometimes theydexperience a problem, but those could be fixed through a remote control session. Successfulmonths could go by without loss of sleep or important family time.

Although your IT job scheduling needs might not necessarily go down the path of systemupdate installation, this time (and money) savings becomes an important parable.Computers are designed by nature to be automation machines. Thus, it stands to reasonthat any manual activity should have an automationfriendly adjunct. It is that adjunct thatcan be a part of your greater scheduling solution.

Can you afford to pay the risk of inappropriate execution, forgetfulness, or user error inyour critical activities? If not, creating flexible and reusable workflows via an IT jobscheduling solution should pay for itself in a very short period of time.

2:WhatIstheCostofanErrorIncurredDuringaManualTask?That first question introduces the possibility of three kinds of risk in any manual system.First is the risk of inappropriate execution. Any task that requires manual intervention alsointroduces the notion that it could be executed at an inappropriate time. Or, moredangerously, such a task could be reparameterized to send data to the wrong location orexecute it in an inappropriate way. There is a recognizable cost associated with this risk.


17/66


11

I remember a situation where a script was created that would apply a set of data to aspecific server upon execution. That script took as parameters a list of servers to send thedata. One day, a junior administrator accidentally invoked the script with the * wildcardin place of a list of servers. As a result, data was distributed to every server all across thecompany. That single invocation cost the company significantly to clean up the mess.

Forgetfulness and user error are both additional risks that can be addressed through a jobscheduling solution. In such a solution, jobs and plans are run within the confines of thesystem and its security model. Dangerous jobs can be specifically restricted against certainindividuals or execution models. Centralizing your job execution security under a singlemodel protects the environment against all three of these costly manual errors.

3:WhereCanYouGoforaHeadsUpDisplayofITActivities?

You probably have monitoring in place to watch servers. Youve probably got similarmonitoring for network components, perhaps even as part of the same product. But doesyour data center also leverage a unified headsup display for monitoring jobs along withtheir execution success? A failure in a job can cause the same kinds of outages and servicelosses as a failure in the network or its servers.

If your business systems interconnect through multiple scheduling utilities across multipleproducts and platforms, there usually isnt a way to centralize all those activities under onepane of glass. What you need is the same kind of monitoring for IT jobs that youve alreadygot in place for your other components.

As you can see in Figure 1.6s mockup, you can get that by using a centralized approach.There, every action across every system and application is centralized into a single screen.Determining which jobs ran successfully is accomplished by looking in one place.

Figure1.6:Dailyactivityunderonepaneofglass.


18/66


12

4:HowDoYouManageCrossSystemCommunications?

Your data center environment already has multiple scheduling engines in place today.Nearly every major business service technology comes with its own mechanism forscheduling its activities. In fact, those mechanisms are likely already performing a set ofduties for your services.

Yet the problem, as you can see in Figure 1.7, has to do with the languages each of theseplatformspecific and applicationspecific scheduling tools speaks. SQL, for example, comesequipped with a wide range of tools for manipulating SQL data and SQL Server systems; buthow rich are those tools when data needs to exit a SQL Server and end up on a UNIXmainframe?

SQL Database

Oracle DatabaseSAP Server

UNIX Mainframe

SAP Scheduling Engine

Oracle Scheduling Engine

UNIX Scheduling Engine

SQL Scheduling Engine

Figure1.7:Multipleschedulingengines.

Often, the native tools arent sufficient, forcing an external solution to bridge the gap. Thatsolution can be in the form of individual little automations like the scripts this chapterstarted with. Or they can be wrapped underneath the banner of a holistic job schedulingsolution. Chapter 4 will discuss the capabilities youll want to look for in the bestfitsolution.


19/66


13

5:AreYouConcernedAboutIdleTimeDuringaTask?

Considering the answer to question 4, some platform and applicationspecific schedulingtools indeed include limited crossplatform support. Their scheduling capabilities may beable to fire jobs based on actions or state changes.

However, one state change that is particularly difficult to measure across platforms is whentasks taketoomuchtime. Task idling in a statebased scheduling system can cause theentire workbook of plans to come to a halt if not properly compensated for. Essentially, thisidle time represents when part of when a task does not complete, leaving the next onewaiting.

SQL Server

Send Data

Oracle

Process Data

SAP

Present Data

Figure1.8:Unmanagedtaskidlingcankillanon automatedworkflow.

Idling need not necessarily be a problem within a piece of code or script. It can be simplythe waiting that is natural in some types of onsystem activities. For example, not knowingwhen a person will submit a file or not knowing when a piece of data is ready for the nextstep in its processing.

These idle states are notoriously difficult to plan for using timebased scheduling alone.With timebased scheduling, your jobs are built with no intelligence about changes thatoccur within a system. Rather, they simply run an action at some set point in time. Your jobscheduling solution must include the logic necessary to add that intelligence. As youll learn

in later chapters, that intelligence can occur through eventbased scheduling or triggerbased scheduling. In either of these cases, an onsystem event or trigger recognizes when achange has been made and initiates the next step in processing.

6:HowManyTasksExistinYourEnvironmentthatYouDontKnowAbout?

If you havent yet standardized on an enterprise job scheduling solution, can you honestlysay how many tasks are operating everywhere in your data center? I used to think I knewwhere all of them were in that former job of mine. But then I left, and took with me the sumtotal of that knowledge. As I mentioned at the beginning of this chapter, those littleautomations are still being found years lateroften after one of them breaks and causesdowntime. More importantly, those are only myscripts. There were others in that company

as well with scripts of their own that probably eventually got lost.

A centralized job scheduling solution creates a single point of control for automation. Itenables auditors and IT teams to know where changes are being sourced from. It isessentially a single point of control, which makes auditors, security officers, and thetroubleshooting administrators very happy.


20/66


14

7:CanYouMonitorITTasksAcrossEveryTeam,Platform,andApplication?

If you cant, you how can you correlate issues across those teams, platforms, andapplications? If you cant, troubleshooting becomes a game of fingerpointing and provingwhy not.

I was once told a story about a company in real need of an enterprisewide job schedulingsolution. Their business system was much like The Project in that it involved multipletechnologies across some very different platforms. Like The Project, managing that systemfell to a somewhat distributed group of individuals. SQL Server was managed by the SQLServer team. SAP was administered by SAP administrators. Even the AD had its own groupof people responsible for its daily care and feeding.

The problem in this company was not necessarily its applicationspecific scheduling tools.It was in its people. Those widelydistributed people feared the centralization that a jobscheduling solution brings. That fear in part was due to the usual technologists fear ofcentralization, but it was also a result of the assumption that a centralized tool would meanrecreating SQL, SAP, AD, and other jobs on a new and completely different system outsidetheir direct control.

An effective enterprise job scheduling solution shouldnt require the complete recreationof existing jobs within each platform and application. Recall that a job itself represents thechange that is to be made, the individual script or package that must be executed. A jobscheduling solution represents the wrapper around that invoked action.

This story ends as youd expect, after a very small but very major problem in onesubsystem impacted the system as a whole. Fully unable to track down that minor job witha major impact, the company discovered why centralizing is a good idea.

8:BuildVersusBuy:IsaHomegrownSchedulerGoodEnough?I once built my own scheduling system in the nowancient scripting language of VBScript.VBScript is still in use many places, and it has a long history. But its known for not havingsuperior builtin methods for scheduling activities. That said, its scheduler worked fine forthe task I assigned to it. But the next time I needed a scheduler, I found myself reinventingthe wheel. Even with the limited code modularization VBScript can present into a script, myschedulers reusability was very limited.

Imagine having to replicate that scheduling across multiple applications and platformsusing different languagesand even using different approaches, both objectoriented andstructured. Homegrown schedulers are indeed an acceptable way of handling the triggering

needs of individual scripts and packages; however, a global scheduler that works across allr job execution.jobs and plans obviously creates a superior framework fo


21/66


15

More importantly, the human resources that are necessary to keep a homegrown jobscheduler can be much greater than they seem at first blush. Those resources need to keepan eye on the logical code itself alongside the jobs that the scheduler attempts to run. Inmany cases, youll find that the extra costs associated with creating your own schedulerinclude the fact that this project will take away from time that could be better served by

working on other more valueoriented projects.

9:WhatAreYourStepsforLinkingtheResultsofOneTaskwiththeActionsof

Another?

Nowhere is that valueadd more pertinent than when two jobs rely on each other. This kindof job construction happens all the time within distributed systems. In it, the first task in astring completes with a set of data. That data is needed by the next task in the string. Thissharing of information can be handled through file dropboxes or richer mechanisms likeMicrosoft Message Queue or database triggers.

Like with the problem of multiple languages, these queuing solutions tend towards being

very platformcentric. It becomes very difficult using a database trigger to invoke an actionin AD, for example. A centralized job scheduling solution with rich support for applicationswill become the central point of control for all crosstask action linking.

10:HowDoYouHandleErrorsinaCustomCodedScript?

Last is the handling of error messages in customcoded scripts, a process that itselfconsumes a vast quantity of script development time. Errors are notoriously difficult totrack down, and become even more challenging when scripts need to span platforms andapplications. Error handling requires special skills in trapping variables and determiningtheir intended and actual values. All of these activities grow even more difficult whenscripts are run automatically as opposed to interactively because error messages in many

cases cannot be captured. Chapter 3 will go into greater detail on the errorhandlingfunctionality of a good job scheduler. For now, recognize that a homegrown script withouterror handling is ripe for troubleshooting headaches down the road.

DoYouNeedJobScheduling?So do you have good answers to these questions? Do you feel that your existing schedulingtools bring you zero headaches? If yes, then thanks for reading. If no, then it is likely thatyour next thoughts will be toward the types of IT challenges that an enterprise jobscheduling solution can support. With the basics discussed and the initial questionsanswered, the next chapter introduces seven realworld use cases for automating IT job

scheduling. It should be an interesting read because the types of use cases outlined in thatgot under management today.chapter are probably pretty close to those youve already


22/66


16

Chapter2:SevenUseCasesforAutomatingITJobSchedulingFind me the business that runs atop a single applicationa single instance!and Ill show

.you a business that doesnt need IT job scheduling. Everyone else probably does

In fact, most data centers have far more. Your average midsize data center runsapplications for handling its databases, along with middleware systems for processing thedata. That data center requires servers and protocols for staging of data in and out of theorganization. Applications run atop client/server operating systems (OSs) and mainframes,servers, and perhaps even a few desktops. All of these elements need to communicate witheach other, many dont share the same OS, and all suffer under the managementcomplexities brought about by productspecific toolsets.

Todays IT technologies are fantastic in the business processes they automate, but rare aretwo that seamlessly talk with each other. Rarer still is the IT product that is superior all byitself in creating and scheduling workflows that meet business requirements. Needed tointegrate activities among disparate technologies is a central solution that can interact witheach at once.

An IT job scheduling solution is thatRosettaStone between different platforms, OSs, andapplications. It is intended to be the data centers solution for converting raw technologyinto business processes. In this book, I hope to show you how to incorporate such asolution into your own business.

Youve already experienced a taste of how an IT job scheduling solution might work.Chapter 1 was constructed to help you recognize that job scheduling is a service your ITorganization probably needs. That said, Chapter 1s discussion intentionally stayed at ahigh level. You havent yet explored deeply the features and capabilities such a solutionmight bring.

You wont get that deep dive in this chapter either. Thats because Ive found that the bestexplanation of IT job scheduling requires first a look at the problems it intends to solve.Once you understand where it fits, youll then appreciate the logic behind its behaviors. It ismy hope that by the conclusion of Chapter 1 you began nodding your head, affirming thatthis purported solution is something your data center desperately needs.


23/66


17

SevenStoriesinITJobSchedulingMy task now is to further enlighten you with a series of ideas to help you find that best fit.These ideas will take place in the form of a seven use cases; essentially, seven little storiesabout issues that have been resolvedor made easierthrough the incorporation of a job

scheduling solution. These stories themselves will be mostly fictitious but are based on realevents and real problems. Ill use faux names to keep the narrative interesting.

Theres an important point here. Even if some portion of these stories is made up, youshould find that the problems and solutions in each arent far from those youreexperiencing.

Story#1:FiveAdministratorsFiveScheduledTasksLibrariesThe first of these stories has nothing to do with a customerfacing solution. Neither is itdirectly related to a lineofbusiness application. Rather, the first of these stories starts

simple. It explains the administrative situation at Company A, a mature company with aprocedurallyimmature IT organization. Lacking many centralized processes, operatingwith marginallyeffective change and configuration control, and managed by five differentadministrators, Company As data center is a mishmash of fiefdoms and technology silos.Problem is, these fiefdoms need to communicate with each other, even if their managing ITadministrators wont.

John, Bob, Jane, Sara, and Jim are those five IT administrators (see Figure 2.1). Each isresponsible for some portion of the data center infrastructure, with each having someoverlap of responsibilities. To accomplish administration, theyve created scripts, tasks,and packages that keep the individual business workflows running. Those automations

indeed enact change on servers and get data moved from system to system but with nointerconnection of intelligence.

Figure2.1:Aninterconnectionofautomations.


24/66


18

Figure 2.1 explains this problem in graphical form. In it, you can see that individualautomations are sourced without considering their context. If John creates a job, hisinformation cannot be based on instrumentation that is gained through another created bySara. As a result, there is no way to orchestrate the activities between each individual, noway to schedule activities so that they do not conflict, and no way to base information or

scheduling from one off of the results of another.

A much better solution is in aggregating these five peoples automations into a single andcentralized solution. Through that single solution, each administrators jobs can be seen bythe others. The jobs of each person can also be aligned with the needs of the others toensure resources arent oversubscribed. Additionally, because jobs are collocated in asingle location, information and instrumentation from any automation can be used to driveother automationsor feed into their future scheduling.

IT Job

Scheduling

Solution

Jim

John

Sara

Jane

Bob

Figure2.2:SourcingautomationsthroughanITjobschedulingsolution.

In short, even if your automations are administrative in nature, an IT job schedulingsolution can bring substantial benefit.

Story#2:ConsolidatingTasksfromEveryApplicationYet an IT job scheduling solution isnt solely about its actors. In fact, in many ways, theactors can be one of that solutions leastimportant impacts. An IT job scheduling solutionreally hasmoredowiththedatainadatacenter. Thats why the second story in thischapter deals with the different applications that are used by Company B.


25/66


19

Different from the IT administration example told in the previous story, Company Bs storycenters around their lineofbusiness (LOB) application. That LOB application is comprisedof several components, each of which is represented in Figure 2.3. Transactions amongthese systems occur through a carefully choreographed set of tasks, jobs, packages, andworkflows. As you can probably imagine, the system in aggregate crosses Windows and

UNIX boundaries, and includes multiple database management systems and even a bit ofmiddleware. It is the classic business service.

Figure2.3:Individualschedulersforeachindividualapplication.

All of these individual components enable the functionality of the LOB application. But all

also leverage their own builtin toolsets for scheduling activities: The SQL server runs itsSSIS packages, the Linux SAP server runs its own tasks and cron jobs, and even theInformatica server enacts change through its workflows.

You are correct in assuming that this oneschedulerpercomponent configuration canindeed work for many systems. Data and actions that occur inside Informatica can be basedon wallclock time or other schedule characteristics. The SQL database can run its SSISpackages based on its own settings, and so on. However, like the actors in the first story,this environment is likely to experience problems as individual system activities conflictwith those on other systems.

Contrast this situation to the superiority in design one gets through job consolidation. InFigure 2.4, the individual task schedulers in that same LOB application have been replacedby a single and centralized IT job scheduling solution. This is possible because, as Imentioned in Chapter 1, a primary benefit of such a solution is its ability to speak thelanguage of every application in the business service.


26/66


20

IT Job

Scheduling

Solution

At Job, Scheduled Task

Cron Job, SAP Task

SSIS Package

Oracle Job

Informatica Workflow

Figure2.4:Consolidatingtasksacrossapplications.

With that centralization of data and actions comes an enhancement to job scheduling,based on results or data in other jobs. This chapters sixth story will explain in greaterdetail how triggering capability dramatically improves service performance; but know herethat centralization of scheduling brings to bear greater instrumentation about the health ofjobs across the entire application infrastructure.

Story#3:TheScriptthatWasntaJobJohn is an Oracle DBA who has been with Company C since the very beginning. In his role asdatabase administrator, he built the companys business system infrastructure nearly fromthe ground up. As a result, he understands those systems inside and out: He has tuned thesystem over time to improve its performance and weed out nonoptimizations. Hes builtnumerous scripts and other automations that gather data, translate it, process it betweenapplication components, and present reports for review by stakeholders. That system iscritical to the company. It provides important revenue data for its sales teams andexecutives. It also means a lot to John.

Then one day the company grew. Substantially. Overnight. Acquiring a completely new lineof business, Company C suddenly found its internal IT systems insufficient to handle thenew reality of work and its associated data needs.

John was approached shortly after the merger by some very important people in the nowlarger company. Those people recognized his strengths in creating and managing theoriginal revenue system that brought much value to the smaller company. They wantedanother system, just like the first one, but this time for selling sprockets instead of cogs.


27/66


21

Graciously accepting the offer to improve the company and fortify his resume, Johnimmediately realized that simply replicating the original system would not be a trivial task.Although his scripts absolutely did everything requested of them in the original system,they were also hardcoded into that original system. Its database architecture was designedto deal with cogs. His transformations were cogbased in nature. Even the server names

and script names were hardcoded into each individual script, task, and package. Worseyet, there were hundreds of tiny automations spread everywhere.

Translating even a simple database job from cogs to sprockets, like what you seegraphically in Figure 2.5, would take months of detective work, recoding, and regressiontesting. John was in for a great deal of work, and the result might not be as seamlesslyvaluable as his original system.

Figure2.5:Replicatinganautomationtoacompletelynewsystem

Had Johns scripts, tasks, and packages instead been objects within a central jobmanagement system (see Figure 2.6), this new company need might not be fraught with somuch risk.

.

IT Job

Scheduling

Solution

IT Job

Scheduling

Solution

Original System Additional System

Figure2.6:Objectsremainobjectsastheyretranslatedintoanewsystem.


28/66


22

Recall Chapter 1s conversation about how good IT jobs are those that are coded forreusability. IT plans are then constructed out of individual job objects to enact change. Ingood jobs, variables are used to abstract things like server names and script namessprockets and cogs, if you willso that entire plans can be rebaselined to new systemswith a minimum of detective work, recoding, and regression testing. An IT job scheduling

solution takes the risk out of business expansion, giving IT the flexibility to augmentservices as the business needs.

Story#4:WhenGatheringDataIsMorethanGatheringDataCompany Ds current situation is a product of its own success. Starting as a smallorganization with a single mission, their focus has changed and evolved over the years aslines of business come and go. Indeed, even whole businesses have been grafted on andlater spun off as the winds shift in Company Ds industry.

As a result, Company D suffers under many of the problems you would associate with any

classic enterprise company. It has thousands of applications under management, some ofwhich are used by only a very few people. Some homegrown solutions only remain becausethey were coded years ago to solve a specific problem for a specific need that has notchanged.

Being a company that is more like the summation of lots of little companies, these businessapplications are nowhere near homogeneous. One budgeting application might store itsdata in SQL, another in Oracle, a third in some obscure database language spoken only byIT professionals long past retirement. Tying these applications and their data together is abig job with big consequences to the organization.

Many enterprise organizations that rely on disparate databases leverage Business

Intelligence (BI) solutions like Crystal Reports. These BI solutions aggregate informationacross the different architectures. Using BI tools like Crystal Reports, data in an Oracleformat can be compared and calculated against data in a SQL format, and so on. These toolscome equipped with rich integrations, enabling them to interconnect nearly all databaseformats all at once (see Figure 2.7). Company D uses Crystal Reports to gather budgetarydata across business units and individual project teams.


29/66


23

Figure2.7:ConnectingsolutionslikeCrystalReportstomultipledatabases.

Yet Figure 2.7 doesnt fully show the reality of how Company Ds data is generated. Beforethat data ever becomes something tangible that can be ingested into a BI solution, it startsits life inside any number of downlevel systems. Figure 2.8 shows just a few of thoseunderlying systems, all of which integrate to create the kind of data a BI solution desires tomanipulate.

Oracle SQL Other Database

Crystal Reports

ExternalCompany

E-Commerce Server

Figure2.8:UnderlyingjobsmakeBusinessIntelligencedatausable.


30/66


24

Notice in Figure 2.8 how a portion of the first business units information comes from apartner company external to the data center. The second and third business units haveprojects in combination that require orchestration and synchronization betweendatabases. The second business unit has further integrations into an ecommerce server inorder to gather a full picture of budget levels.

BI solutions can indeed present a moreunified view of data across different platforms, butthey do not provide a mechanism to unify transactions between downlevel systems. ForCompany D, creating that unified workflow lies within the realm of an enterprise IT jobscheduling system. If gathering data is more complicated than simply gathering data, a jobscheduling system ties together the entire system to accomplish what you really need.

Story#5:ControllingDataTransferasaWorkflowCompany E had a big problem not long ago when they began extending customer servicesonto the Internet. They quickly found out that you can indeed interact with customers

there, but creating a holistic system that gets customer data in all its various types into thehands of the right person isnt as easy as it looks.

What Company E found out is that dealing with customers over the Internet automaticallycreates a lot of data. That data arrives in various formats, with each format requiring adifferent mechanism for handling.

You can see a graphical representation of Company Es story in Figure 2.9. Internetcustomers that desired services interacted primarily through a Web server. Inside thatWeb server was contained the requisite logic to inform customers about products, andinteract with them as purchases are made. Interesting about Company E is that theirsystem involved twoway communication with customers not only via the Web site and

email but also in the transfer of data files.


31/66


25

Figure2.9:Differentdataformatsrequiredifferentdatahandling.

Data files, as you know, are much different than XML files in Web transactions or emailsback and forth through an email server. Theyre larger, they can come in many differentformats which create particular issues when youre working with unstructured customers,and the management tools to work with them dont necessarily integrate well into otherformats and workflows.

Company E needed a multipart mechanism to solve their formatting workflow problem.They needed to recognize when an order was placed, generate an FTP URL for the user toupload data, move that uploaded data from a lowsecurity FTP server to highsecuritydatabase server, and finally notify the user when the transaction was complete. Adding tothe complexity, those exact same steps were required in reverse at the time the order was

fulfilled.


32/66


26

You can imagine the protocols and file formats at play here: XML, SMTP, FTP and SFTP,along with a little SSH and SOAP to tie the pieces together, just like you see in Figure 2.10.Complex needs like Company Es require data transfer handling that can support therecognition that files have been downloaded. Such handling can either monitor for a filespresence or use event or messagebased notification. An IT job scheduling solution wraps

file transfer logic into the larger workflow, enabling XML to trigger SSH, to fire off SMTP,and finally to invoke SOAP at the point the application requires.

Figure2.10:Multipleprotocolsatplay,eachwithitsownmanagement.

For complex file transfer needs, those that must exist within a business workflow, IT jobscheduling solutions solve the problem without resorting to lowlevel development.

Story#6:ComplexJobsandtheNeedforTriggersThis chapters sixth story brings me back to the one I started in Chapter 1. There Iexplained some of the introductory pieces in TheProjectThatWouldChangeEverything. Ialso explained how an IT job scheduling system was very quickly identified as the onlyclass of solution that could enable the kind of functionality our complex project needed. Letme tell you a little more in that story.


33/66


27

Our determination happened shortly after whiteboarding the various components weknew the project needed. Figure 2.11 shows a reenactment of that whiteboarding activity,with only a few of its myriad interconnecting magic marker lines in place. My team knewthat when external company data arrived on the FTP server, transferring that data to theSQL server must occur in as close to realtime as possible.

Figure2.11:WhiteboardingthetriggersforTheProject.

Fulfilling that requirement with traditional FTP alone created a fantastically problematicdesign. The idea of constructing an alwayson FTP session that constantly swept for newdata was a ridiculous notion. And it wasnt a good idea for security. Needed was some kindof agent (or, better yet, an agentless solution) that would simply know when data arrived.

Then it could provision that data from the FTP servers data storage to the SQL database.

But that wasnt our only challenge. At the same time, our SQL and Oracle databases neededto remain in strong synchronization. Changes to specific values in SQL must replicate toOracle, also in as realtime as computationally possible. Synchronizing Oracle with SQLmeant also synchronizing metadata with SAP.


34/66


28

Even a single one of these nearrealtime requirements can be challenging for a developerto build. Lacking developers, a development budget, and the desire to complete this projectusing offtheshelf components, we demanded a solution that would accomplish the task

wlevel coding.without reverting to lo

What we needed were triggers.

Triggers are the real juice in an IT job scheduling solution. The kinds and capabilities oftriggers a job scheduling solution supports makes the determination between one thatsenterprise ready and one thats not much more than the Windows Task Scheduler.

The Project required a wide range of these triggers to get the job done: Our projects FTPtoSQL integration required afile basedtrigger, kicking off a job or plan when a fileappeared at the FTP server. Message basedtriggers were also necessary for the SQLtoOracle integration, enabling the two applications to notify each other aboutsynchronization activities. Eventtriggers were necessary in the OracletoSAP integration,allowing Oracle to create events about changes to its state and alerting SAP to make

associated changes based on event characteristics. Those same event triggers gave ActiveDirectory (AD) the data to quickly tag permissions into the data. Finally, time basedtriggerskicked off occasional data transfers between the SQL database and the UNIX mainframe.

Wildly, simple triggers alone werent sufficient. All by itself even the best trigger couldntfulfill the multiserver and multiaction realtime requirement our system demanded. Wealso needed the ability to tether or chain individual triggers together. By chainingtriggers, we could speed the process, get data where needed, and ensure the systemremained in convergence. Ill talk more about chaining triggers in the next chapter.

Pay careful attention to the triggering capability of your selected IT job scheduling solution.

The very best will come with the richest suite of triggering abilities.

Story#7:SecuringtheDataCenterfromtheEaseofScriptExecutionCompany F was a midsize company with a midsize IT organization. Responsible for all thetasks typically associated with IT, its IT team got along well and generally provided goodservice to the company at large.

But even the most wellmeaning of IT organizations occasionally makes mistakes, andsometimes those mistakes are large in impact. Thats the situation that occurred one dayafter two administrators began sharing their quiver of scripts, tasks, and packages.

You already know that one of the primary benefits of moving to scriptbased automation isconsistency in reuse. By packaging a series of tasks into a script, that script can be executedover and over with a known result. Parameterizing those same scripts makes them evenmore valuable to IT operations. Once created, the same script can be used over and overagain across a range of different needs. Yet reusability can sometimes be a risk as well.When a person can simply doubleclick a script to activate it, the chance presents itself thatthat doubleclick might happen inappropriately.


35/66


29

Thats exactly what was intended the day Sara borrowed one of Janes scripts for use inanother system. Janes scripts were brilliantly designed, smartly parameterized, and welldocumented. Built right into each script was all the necessary information anotheradministrator would need to reuse the script elsewhere. For Sara, Janes script perfectlysolved her problem at hand.

The central problem, however, was that Sara wasnt really authorized to run Janes script.In fact, the wellmeaning Sara wasnt supposed to be working on the system at all. Whenshe executed Janes script, it brought the system down unexpectedly. Company F learned avaluable lesson that day in the openness of simple scripts.

Thats why shortly thereafter Company F invested in an IT job scheduling solution foraggregating their automations into a unified store. Unifying automations within arestrictedrun framework enabled Company F to apply privileges to their scripts. Becausethey chose a bestinclass IT job scheduling solution, they were able to not only applyprivileges on the scripts themselves but also the jobs, plans, and even variables associatedwith those scripts. Having correct permissions in place reduces the risk that a Sara willinappropriately execute a script. But, more importantly, it also reduces the risk that a Saratype worker will inappropriately attach the wrong variables to the right script, or the rightvariables to the wrong script.

Figure2.12:Applyingsecurityatvariouslevelsinscriptexecution.

Consolidating IT automations into a job scheduling framework provides visualization ofscripts within the enterprise. It adds security to what might otherwise be highlydangeroustext files. It creates a location where their successful execution can be proven toadministrators as well as auditors. And it creates an auditable environment of approvedexecution that protects the data center.


36/66


30

JobSchedulingsStoriesAreitsValueAddThese seven stories are told to help you understand the value add to an IT job schedulingsolution. In them, youve learned how IT job scheduling works for administration as well ascomplex tasks that might otherwise be relegated to lowlevel developers. Youvediscovered how triggers and file manipulations are as important as database tasks and

middleware actions. Youve also learned how job scheduling creates that framework forapproved execution that your auditorsand, indeed, your entire businesstrulyappreciate.

Yet as I mentioned as this chapter began, I still havent dug deeply into the inner workingsof jobs themselves. Now that youve gained an appreciation for where job schedulingbenefits the data center, lets spend some time discussing a technical deconstruction of anIT workflow. At the conclusion of the next chapter, youll gain an even greater appreciationfor how these job objects and their plans fit perfectly into the needs of your businessservices.


37/66


31

Chapter3:WhatMakesanITWorkflow?ATechnicalDeconstructionComputersareusefulbecausetheyllperformanactivityoverandoverwithoutfail.Theartis

intellingthemexactlywhattodo.

I hope that reading Chapter 2 was as enjoyable as writing it. Although Ill admit I took alittle literary license in telling its stories, I did so to highlight the use cases where IT jobscheduling makes perfect sense. Coordinating administrator activities, consolidating tasks,generalizing workflows, gathering data, orchestrating its transfer, triggering, and securityare all important facets of regular data center administration. Yet too often these facets areadministered using approaches that dont scale, introduce the potential for error, or cantbe linked with other activities. The ultimate desire in each of Chapter 2s stories was the

creation ofworkflow. That workflow absolutely involved each storys actors; but, moreimportantly, it involved the appropriate handling of those actors data.

In many ways, workflows, jobs, and plans represent different facets of the same desire:Tellingacomputerwhattodo. You can consider them the logical representations of thelittle automation packages I referenced in the first two chapters. Although I spent much ofthose chapters explaining why theyre good for your data center and how theyll benefityour distributed applications, I havent yet shown you what they might look like.

Thats what youll see in this chapter. In it, youll get an understanding of how a workflowquantifies an IT activity. Youll also walk through a set of mockups from a model IT jobscheduling solution. Those mockups and the story that goes with them is intended to

d.solidify your understanding of how an IT job scheduling solution might look once deploye

But for now, lets stay at a high level for just a bit more. In doing so, I want to explain howworkflows bring quantification to IT activities.

AWorkflowIsanITActivity,QuantifiedA workflow has been described as asequenceofconnectedsteps. More importantly, aworkflow represents an abstraction of real work. It is a model that defines how data getsprocessed, where it goes, and what actions need to be accomplished with that datathroughout its life cycle.

You can find workflows everywhere in business, and not all are technical in nature. Thinkabout the last time you took a day off from work. You know that taking that day off requiresfirst submitting a request. That request requires approval. Once approved, you notifyteammates and coworkers of your impending unavailability. In the world of paid time off,you cant just miss a day without following that process.


38/66


32

And yet sometimes people do just miss days. Perhaps they were very sick, or got stuck onthe side of the road far from cell phone service. In any of these cases, the workflow breaksdown because the process isnt followed. What results is confusion about the personswhereabouts, and extra effort in figuring out what they were responsible for accomplishingduring their absence.

You can compare this people workflow to the data workflows in an IT system. Data inan IT system needs to be handled appropriately. Actions on it must be scheduled withprecision. Data must be transferred between systems in a timely manner. Failure states inprocessing need to be understood and handled. The result in any situation is a systemwhere data and actions can be planned on.

To that end, lets explore further the IT plan first introduced back in Chapter 1. Figure 3.1gives you a reproduction of the graphic you saw back in that chapter. There, you can seehow three jobs have been gathered together to create Plan7SendDataSomewhere.

Figure3.1:AnexampleITplan.

I wont explain again what this plan intends to do; the activities should be selfexplanatory.More important is the recognition that this example shows how an IT workflow quantifiesan activity along a set of axes: capturability, monitorability and measurability, repeatabilityand reusability, and finally security. Lets explore each.

Capturable

I find myself often repeating the statement, Always remember, computers aredeterministic! Given the same input and processing instructions, they will always producethe same result. Yet even with this assertion, why do they sometimes notproduce the result

were looking for?

That problem often centers on how well the established workflow captures theenvironments potential states. A welldesigned workflow (and the solution used to createit) must have the ability to capture a systems states and subsequently do something basedon what it sees.


39/66


33

I recently heard a story that perfectly highlights this need for capturability. In that story, acompany ran numerous missioncritical databases across more than one databaseplatform. Most of these databases were part of homegrown applications that the companyhad created over time.

Backing up these databases was a regular chore for the IT department. Although thecompanys backup solution could indeed complete backups with little administrator input,the configuration of many databases required manual steps for backups to completecorrectly. Due to simple human error, those manual steps sometimes werent completedcorrectly. With more than 25 databases to manipulate, that human error became thebiggest risk in the system. Fixing the problem was accomplished by implementing asolution that could capture the manual portions of the activity into an IT job.

Such capture is only possible when an IT job scheduling solution is richly instrumented.That solution must include the necessary vision into backup solutions, database solutions,and even custom codebases. Vision into every system component means knowingwhenthetaskneedsaccomplishing.

MonitorableandMeasurable

You cant capture something unless you can monitor and measure it. Just as important asvisibility into a system is visibility into the workflow surrounding that system. An effectiveIT job scheduling solution must be able to instrument its own activities so that the job itselfcan recover from any failure states.

This is of particular importance because most IT jobs dont operate interactively. Oncecreated, tested, and set into production, a typical IT job is expected to accomplish its taskswithout further assistance. This autonomy means that welldesigned jobs must includemonitoring and measurement components to know when data or actions are different from

expected values.

Its easiest to understand this requirement by looking at the simple IT plan in Figure 3.1.Such a workflow is only useful when its activities are measurable. More important,measurement of a plans logic must occur at multiple points throughout the plansexecution. Figure 3.2 shows how this builtin validation can be tagged to each phase of theplans execution. In it, you see how the handoff between Job 27 and Job 19 requiresmeasuring the success of the first job. If Job 27 cannot successfully connect to the database,then continuing the plan will be unsuccessful at bestanddamagingatworst. You dont

ant bad data being eventually sent via FTP to a remote location.w


40/66


34

Figure3.2:Validationlogicensuresmeasurability.

Similar measurements must occur in the handoff between Job 19 and Job 42 and again atplan completion. A successful IT job scheduling solution will create the workbench wherevalidation logic like that shown in Figure 3.2 can be tagged throughout an IT plan. Thislogic should not impact the execution of individual jobs, nor is it necessarily part ofwhatever code runs beneath the job object. Effective solutions implement validation logicin such a way to be transparent to the execution of the job itself.

RepeatableandReusable

Transparency of measurement along with parameterization of job objects combine tocreate a repeatable and reusable solution. You can imagine that creating a well

instrumented IT plan like Figure 3.2 is going to take some effort. Once expended, that effortgains extra value when it can be reused elsewhere.

Reusability comes into play not only within each IT job but also within each plan. Recall myassertion back in Chapter 1 that an IT job is an action to be executed. This definitionmeans that the boundary of an IT job must remain with the execution of an action. Figure3.3 shows a graphical representation of an IntegratedJobsLibrary. In that library, is acollection of previouslycreated jobs: Job 17 updates a database row, Job 27 opens a

onnection to a database, and so on.c


41/66


35

Figure3.3:ReusingITjobsinaplan;reusingITplansinaworkf

Each of those discrete jobs can be assigned to a workflow for the purposes ofaccomplishing some task. They can also be strung together in infinite combinations tocreate a morepowerful IT plan. You can see an example of this in Figure 3.3. Notice howJob 27 represents the beginning step of Plan 15; it also represents a middle step for Plan

low.

22.

Once created, both jobs and plans reside in an IT job scheduling solutions Integrated JobsLibrary. From there, created jobs can be reused repeatedly as similar tasks are required. InFigure 3.3, two new databases require synchronization. Since a plan has already beencreated to accomplish this task, reusing that plan elsewhere can be as simple as a dragand

drop. After dragging to create a new instance of the plan, the only remaining activitiesinvolve populating that plan with new server characteristics.

Securable

Chapter 2 introduced the notion of job security. In the seventh story, you read howindividual jobs and entire plans can be assigned security controls to prevent misuse. Thatlevel of security is indeed an important part of any IT system; however, Chapter 2 onlybegan the conversation.

Consider the situation where an IT plan updates data in a database. Correctly constructingthis IT plan requires parameterizing the plan to eliminate specific row values or items of

data to update. However, parameterizing the plan in this way introduces the possibilitythat someone could accidentally (or maliciously) reassign the plan and update the wrongdata.


42/66


36

This risk highlights why deeplevel security is fundamentally important to an IT jobscheduling solution. You want controls in place to protect someone from invoking a planinappropriately. But you also want controls in place to protect certain instances of ortriggers for that plan to be executed. Each platform and application tied into your IT jobscheduling solution has its own security model, as does the job scheduling solution itself.

Mapping these two layers together is what enables a job scheduling solution to, forexample, apply Active Directory (AD) security principles to some application with a nonWindows security model. Doing so enables you to lean on your existing AD infrastructurefor the purpose of assigning rights and privileges in other platforms and applications.Figure 3.4 shows how such an extended access control list (ACL) might look, with triggers,trigger characteristics, and even instances of such a plan being individually securable.

Figure3.4:Applyingdeepsecuritytoajoborplan.

AGroundUpITWorkflowConstructionAt its core, an IT workflow is still a piece of code. Some kinds of code a solutions vendorwill create and include within a job scheduling solution. These represent the builtin jobobjects in your solutions Integrated Jobs Library. Other code must be customcreated bythe administrators who use that solution. No vendor can create objects for every situation,so sometimes youll be authoring your own. Notwithstanding who creates the code, at theend of the day, it is that code that needs to be scheduled for execution.

With this in mind, lets walk through an extended example of constructing a workflow outof individual parts. You can assume in this example that an IT job scheduling solution hasbeen implemented and will be used to author the workflow.


43/66


37

A diagram of that workflow is shown in Figure 3.5. In it, each block represents an activity tobe scheduled. Its story goes like this: Data in a system needs to be monitored for changes.As changes occur, an IT plan must be invoked to gather the changes, run scripts against thedata, and move it around through file copy and FTP transfers. While all these processesoccur, individual jobs within the workflow must trigger each other for execution as well as

monitor for service availability.

Query Web

Services

Select Oracle

Data

ProcessScript A

File Copy Data FTP Data

Monitor Service

Monitor for File

Presence

Process

Script A

Figure3.5:Anexampleworkflow.

You should immediately notice that scheduling is an important component of thisworkflow. That scheduling isnt accomplished through some clockonthewall approach. Itis instead based on monitoring the states present within the system (presence of files, WMIqueries, log file changes, and so on), and firing s

Documents

SAITJS Complete