Case Study : Organization Performance Capability
ReviewsConfidential
Asia’s Largest Global Software & Services Company
Confidential
PROJECT
SANDBOX
PARAMETERS
KNOWN PROBLEMS
Confidential
What is EME?
EME, Enterprise Meta Environment, is an object oriented data
storage system that version controls and manages various kinds of
information associated with Ab Initio applications, which may range
from design information to operational data. In simple terms, it is
a repository, which contains data about data – metadata.
Why EME?
Source Control
dependency and impact analysis for the graphs in the
repository
Asia’s Largest Global Software & Services Company
Confidential
Re>Posit -- Base technical metadata product
BM>R -- Business metadata repository on top of Re>Posit
These two products are now bundled together with a new name
EME
Asia’s Largest Global Software & Services Company
Confidential
Confidential
EME Data store/Repository connection settings
EME datastore is a specific instance of EME in the environment.
This is a repository
where different versions of code and its related data like the
record formats,
transformations etc are maintained. At any point of time a user can
connect to only
one such EME repository instance. To access an EME Datastore, go
to
Project>EME Datastore Settings in the GDE Menu and details are
to be filled up in the
following boxlike:
Confidential
EME Data store/Repository connection settings
After filling in the detail press on the Connect button to test the
connection. If the
details are filled in correctly you will get a message box
confirming the connection.
Following details are to be filled up in the EME Datastore
Settings
Method : Remote Execution (Rexec)/Telnet
Login and Password : Unix Login credentials for the host
Co>Operating system Location : Path to where the Ab Initio
Co>Operating
system is installed
EME Datastore Location : Path to where the EME Datastore is
located
Mode : Source Code Control
Confidential
Project
A Project is a collection of related graphs and its associated
elements like
dml, xfr etc in the EME Datastore.
Project structure
Typically a project should contains maximum of 5 to 10 graphs. This
helps in organising the code efficiently within EME. With increase
in the number of graphs in a Project, the time taken to perform
dependency analysis on the graphs and related data increases.
Before adding a Project to an existing application, which already
has a number of Projects in place, the impact it might have on
other Projects and on the Application as a whole must be
considered.
Asia’s Largest Global Software & Services Company
Confidential
/Projects
Project1
Project2
mp
run
xfr
dml
db
Sub directory for deployed shell scripts
Sub directory for transform files
Sub directory for record format files
Sub directory for database interface files
bin
sql
Sub directory for sql queries
Asia’s Largest Global Software & Services Company
Confidential
Different Types of Projects
Private vs. Public Project
There is often information common to multiple Projects. For
instance several Projects may share some record format files or
transform files. Such elements which are used across Projects can
be made widely reusable by making them part of a Project and
including this Project in other projects to access the common
elements. A Project that is included by other Projects is termed as
a Public Project and the Projects including public Projects are
known as Private Projects.
A public Project is public in the sense that their data and
metadata are expected to be shared with other Projects and a
private Project is private in the sense that their data and
metadata are not expected to be shared with other Projects.
Asia’s Largest Global Software & Services Company
Confidential
Different Types of Projects
The Environment Project (Stdenv)
There is a special Project associated with every instance of Ab
Initio environment
known as the Environment Project or stdenv (Standard Environment).
This is no
different from a regular Project in the structure. It contains
machine and Application
specific settings like the data directory mount points, max-core
settings and application
wide parameters like current date, which are used across all
Projects. During creation
of any Project, stdenv is included in it by default. A single
stdenv is required for an entire
set of applications on a single machine and sharing a single EME
Datastore.
Asia’s Largest Global Software & Services Company
Confidential
What is a Sandbox?
Projects held in the EME Datastore can’t be manipulated directly.
To work on
Projects, they must be checked out to a working area on the file
system where
we can develop and modify code. This working area on the file
system is known
as a Sandbox. It has exactly the similar directory structure as
that of a Project
in the Datastore.
Each object that needs to be worked on is checked out to a sandbox
where
modifications or enhancements are carried out. After the changes
are complete
the code is checked in from the sandbox area to the EME
Datastore.
This action creates a new version of the code in the EME
Datastore.
Asia’s Largest Global Software & Services Company
Confidential
Sandbox Projects vs. EME Datastore Project
Sandboxes are work areas used to develop, test or run code
associated with a given project. Only one version of the code can
be held within the sandbox at any time. The EME Datastore contains
all versions of the code that have been checked into it. A
particular sandbox is associated with only one Project where as a
Project can be checked out to a number of sandboxes.
Check-in
Check-out
Project1
Confidential
Parameters
A parameter is a name-value pair with some additional attributes
that determine when and how to interpret or resolve its value.
Parameters are used to provide logical names to physical location
and should always be used instead of hard coded paths in graphs. We
can have two types of parameters, graph and Project
parameters.
Graph parameters
Graph parameters, as the name suggests are specific to the
individual graphs and are private to them. They affect execution of
the graph for which they have been defined. Graph parameters can be
defined by navigating to Edit>Parameters in the GDE which opens
the graph parameters editor.
Asia’s Largest Global Software & Services Company
Confidential
Project parameters
Project parameters are inherited by all the graphs in the Project
and are accessed from the GDE by the sandbox parameter editor in
Project>Edit Sandbox>Parameters. This shows a dialog box
prompting to enter the sandbox path. Choose the correct host and
the sandbox path and press OK to open the sandbox parameter editor,
which exactly like the graph parameter editor shown as above.
Asia’s Largest Global Software & Services Company
Confidential
Editing parameters
To add a new Project parameter or to modify the value of an
existing one, we
should first lock the parameters in the sandbox parameter editor by
clicking the
lock button on the menu. If nobody has locked it in their
sandboxes, then the lock
symbol turns green indicating a successful lock. This implies we
can add or modify
the parameters now. If a lock is already there before, then while
opening the
parameter editor it shows a warning saying the parameters are
already locked and
the lock symbol is red in such a case. After getting a lock, others
are disabled from
editing the parameters.
Confidential
Ab Initio EME
Major Parameter Attributes
Scope: Scope of a parameter can be formal or local. A local
parameter is internal to the
sandbox and most of the parameters have their scope as local. Its
value is taken from
the value column in the parameter editor. A formal parameter is one
whose value can
be set from outside, i.e. from the environment where the graph is
run. Its value is
supplied from the command line. A green diamond can identify the
formal parameters
with an arrow mark.
Kind: If scope is local, kind is left unspecified, but if it is
formal, the kind is automatically
set to keyword.
Type: This determines the nature of the parameter. Project
parameters have four types
as string, common Project, switch and dependent. Graph parameters
have different set
of types.
Export: When this check box is checked, the corresponding parameter
value is
exported as an environment variable, otherwise it is generated as a
local shell variable.
Private Value: If a parameter is specified as a private value, any
subsequent
changes to it remain private to the local sandbox and are not
checked in into the EME.
This is useful when different users want different values for the
same parameter.
Asia’s Largest Global Software & Services Company
Confidential
Value: This column specifies the value of the parameter.
Interpretation: This determines how the parameter is going to be
evaluated.
Constant: Value is taken literally.
$ Substitution: Variables with $ prefixes are replaced with their
values
${} Substitution: Variables within {} and with $ prefixes are
replaced by their values but other occurrences of $ are
ignored.
Shell: Korn shell syntax is used to evaluate the value of the
parameter.
Required: This attribute can take two values, required (the
default) or optional. If it is required, the value column can’t be
left blank but if it is optional, it can
be left blank.
Confidential
Version Control and Tagging
Each object under EME source control, which may be a file, a
directory or a Project, exist as a series of versions, each of
which is a representation of what was checked in by some user. It
can optionally have a textual description attached to it called a
tag and a description as a comment. Each version is separately
numbered and can be accessed by either the version number or the
tag attached to it. Version numbers, which are integers and tags,
are global to the whole EME datastore. Tags are the basic units
during migration of code across EME datastore instances.
Asia’s Largest Global Software & Services Company
Confidential
GDE
TBD ??? Display web interface .. URL ??
Asia’s Largest Global Software & Services Company
Confidential
Check out
The Ab Initio GDE provides wizards to check out code from the EME
to sandbox.Check out updates the sandbox with the particular
version of code that is being checked out from the EME. By default
the latest version of any object is checked out, but we can check
out any version of code we want. Any object that is version
controlled in the EME datastore can be checked out to a sandbox,
which may be pre-existing or may be created during check out
process itself. While checking out a Project or any objects
belonging to the Project to a sandbox, stdenv and any common
Projects associated with it also need to be referenced in the
sandbox. If the sandbox to which you are checking out is an
existing one, it would have the information as to where to
reference for the common projects (The stdenv sandbox and the
public sandboxes). In case it is a new sandbox, during check out we
have to point to the stdenv and public sandbox (if any)
paths.
Asia’s Largest Global Software & Services Company
Confidential
Check out of files using GDE
Check out wizard is invoked by navigating to Project>Check Out,
which looks as follows:
Select the Project /directory or file you want to check out by
browsing to the particular Project /directory or file.
In sandbox host dropdown list select the host on which the sandbox
resides.
Enter the path to an existing sandbox (the sandbox must be
associated with the concerned Project, which is being checked out)
or mention a new one in the directory field, which would be created
during check out.
Asia’s Largest Global Software & Services Company
Confidential
Check out of files using GDE
The advanced options dialog can be seen by clicking on advanced
button.
The first two options specify whether to check out the required
files from the parent project and whether to check out required
files from the common Projects. The default is check out the
required files from the parent project. A file is required if it is
directly referenced in a graph or if it is referenced in an include
in a dml or xfr. While checking out a whole project these two
options are disabled as shown above.
Run host setup script makes sure to run the host profile’s set up
script before check out and mark files read only on check out does
exactly what it says. The default is on for both of these
options.
Asia’s Largest Global Software & Services Company
Confidential
Check out of files using GDE
We can select a particular tagged version of the object we want to
check out from the tag drop down list. By default the latest
version is checked out.
On clicking next, if the sandbox doesn’t exist then a confirmation
is asked whether to create the new sandbox or not. Clicking yes
creates the sandbox and checks out the object mentioned to this
sandbox.
Asia’s Largest Global Software & Services Company
Confidential
Check out of files using GDE
You will be prompted to enter the sandbox locations of stdenv and
any common projects associated with the project, unless the sandbox
has already these values specified or the sandbox is a pre-existing
one.
Asia’s Largest Global Software & Services Company
Confidential
Check out of files using GDE
Clicking on Do Checkout performs the checkout operation and on its
completion a window shows the operations performed.
Asia’s Largest Global Software & Services Company
Confidential
Locking
A lock must be acquired on the object to be modified in the sandbox
after successful completion of check out. To modify a graph that
has been checked out, first open the graph in the GDE and then
click on the lock symbol on
the menu. This checks whether the version in the sandbox is the
latest version of the object in the datastore and if it is, the
lock symbol turns green showing
that the graph is now locked and is editable.
If the graph has already been locked in some other sandbox, after
opening the graph in the GDE the lock is red in colour denoting
that there is already a lock on it. A lock can be acquired on an
object only if the sandbox version and the current version of the
object in the EME are the same.
Once a lock is acquired and the changes are complete the object
must be checked in to the datastore to create a new version in the
Datastore.
For non-Ab Initio objects which can’t be locked from the GDE, a
lock can be obtained from the Unix command line using the air
commands available to obtain a lock on the particular object.
Asia’s Largest Global Software & Services Company
Confidential
Check in of files using GDE
Once the project files have been edited and updated they need to be
checked in to create a new version in the EME datastore, which will
be available for other users. Check in wizard is invoked by
navigating to Project>Check in. Before starting the check in
wizard, it checks for any unsaved file in the sandbox and prompts
whether to save them or not. The check in wizard looks as
follows:
Asia’s Largest Global Software & Services Company
Confidential
Choose the Sandbox host from the drop down list.
In the Directory or file field, browse to the particular file in
the sandbox that you want to check in. You may select a file under
the sandbox or you may also select the whole sandbox in which case
the whole project would be checked in to the EME datastore.
Browse to the parent Project in Project Directory field, which
points to the Project directory in the EME datastore where the
object would be checked in.
To go to the advanced options in check in click on the advanced
button.
The check in tab indicates how you want the check in to be
performed. By default “Force overwrite” is unchecked. Once it is
checked the object is checked in even if there are conflicts and
becomes the latest version in the datastore. “Run Host Setup
script” causes to run the host profiles set up script before each
check in. It is advised not to change any settings here.
Asia’s Largest Global Software & Services Company
Confidential
Check in of files using GDE
The analysis tab specifies how much dependency analysis is done and
on which objects during check in.
Asia’s Largest Global Software & Services Company
Confidential
Check in of files using GDE
A tag, which is a descriptive piece of text and a comment, can be
attached to the version that will be checked in. This can be
mentioned in the tag tab of advanced options dialog box. The
tagging standards are described in another document.
Asia’s Largest Global Software & Services Company
Confidential
Check in of files using GDE
After filling in the tag information, on clicking next in the check
in wizard a check in ready dialog is displayed.
Clicking on “Do Checkin” performs the actual check in and displays
a window similar to the “check out finished” window with the
results of check in and dependency analysis (if specified in the
advanced option).
Asia’s Largest Global Software & Services Company
Confidential
of graphs/objects
Many a time a previous version of a graph may be required to check
out and update rather than working with the latest or current
version of the graph as available in the EME data store. Using
check out wizard in GDE, you may check out a tagged version of a
graph, which is not the latest version available. But GDE doesn’t
allow locking such versions. In such a case, the procedure to be
followed is:
Asia’s Largest Global Software & Services Company
Confidential
of graphs/objects
Check out the required previous tagged version of the graph to your
sandbox.
(V1 in figure below).
Check it back in with “Force Overwrite” in advanced option in check
in wizard.
This will make it the current version in the data store. (V4 in
figure below).
Lock the graph now to make the changes.
Check in the graph back to the EME data store. This updated version
will
become the latest version in the EME data store. (V5 in figure
below)
EME data store
Confidential
E.g …. To be Done ????
Define Project --- Module Leads
Define Parameters --- Module Leads
Confidential
Known Problems Accessing EME from offshore
E.g …. To be Done ???? With resolution…...
Few experienced resources in EME -->>> Training
Considerable Time taken to check out , check in from offshore
-->>> check out at start , check-in at end , no
intermediate movement
Help on EME
Confidential