Some important new ehancements

My colleague, Wayne Bucek, in his Wayne_z_World blog has some information on a recent set of enhancements to OMEGAMON and ITM that optimize queries and displays that may request large numbers of rows of information. The more data requested, the bigger the savings. Like in the example Wayne mentions, if you are monitoring large numbers of MQ Q managers, this can have a big savings.

Here is a link to Wayne's blog, and the specifics of the maintenance:
https://www-950.ibm.com/blogs/9051592f-7640-466e-8524-6ae7dcc20c79/entry/monitoring_large_numbers_of_queues_better_faster_cheaper17?lang=en

Planning for ITM 6.22

If you are working on a plan to get to ITM 6.22, here is a link to some good information on platform support and compatibility for ITM 6.22. This link will show in detail information on supported operating systems, platforms, databases, etc.

http://www-01.ibm.com/support/docview.wss?uid=swg27018000&myns=swgtiv&mynp=OCSSZ8F3&mync=R#Database%20-%20DB2_R-swgroup

View threads in a data sharing group from OMEGAMON classic

For those OMEGAMON DB2 users that run DB2 in a data sharing environment, this is an enhancement that was added a while back to the classic interface, but you may have not noticed. The top line of the classic interface is the command line. The left portion of the command line is where you can enter various commands, screen space names, and navigational commands. The data sharing group support added some options on the right portion of the command line. You can now specify the DB2 subsystem, and if you want a single system (S option) view, or the data sharing group view (G option).

In the example I show first, on the top left, a single system view (DB2S option S), and notice there are no threads to view. I then specify option G for the data sharing group view. Now I can see threads connected to the DB1S DB2 member in the data sharing group. Also, I can drill in for thread detail, just like any thread in the classic interface.

This is a handy navigation option that you may, or may not have noticed before.

Situation overrides provide added flexibility

I've talked before about situation management, optimizing situation usage to make situations more relevant, and doing things like using policies to manage situations, such as for time of day processing. If you are running ITM 6.21 or above you have the ability to take advantage of an interesting function called Overrides. The override mechanism provides a way to specify a different threshold settings for different requirements, such as Prime shift or Non-prime shift.

I include an example of how to use overrides. To access the override logic you have to go to the distribution tab for the situation editor, select a managed system and then click the override button. You can then specify the value setting, and here I specify if the setting is for Prime or Non-Prime shift. It's an interesting option, and I plan to work with it some more to see how it may be effectively exploited.

DB2 10 for z/OS goes beta

Being a DB2 guy from a ways back, this is a cool announcment. DB2 10 For z/OS is going beta. Here is a link for more info.

http://www-01.ibm.com/software/data/db2/zos/db2-10/

OMEGAMON DB2 Near Term History

OMEGAMON DB2 has a very useful Near Term History (NTH) function. NTH provides an easy way to be able to retrieve and review DB2 Accounting and Statistics records from the past few hours of DB2 processing. The data is stored in a set of VSAM files allocated to the OMEGAMON collection task. How far back the history goes depends upon the size of the files and the amount of data being written to these files. Now some of the data volume is driven by the DB2 workload activity. Accounting records are typically written when a DB2 thread terminates processing, and it is the Accounting data that is often looked at by the analyst when studying what DB2 applications have been doing. Statistics records are created on a time interval basis. Usually, you will have much more accounting data than statistics data. Also, OMEGAMON has the ability to pull in additional trace IFCIDs to get information on things such as dynamic SQL activity.

To understand the amount of data being gathered by NTH, there are displays that show the number of records written to the NTH files, by type. In the example I show, you see an example of common NTH settings/options, and then you see the record count in the NTH record information display. If you look carefully you see that 'Perf-Dyn SQL' has a lot of records written relative to the other record types. This is a good way to understand the impact of enabling certain collection options, such as dynamic SQL collection, and see how many trace records are being gathered, as a result.

Did you know? About ServerPac....

Tivoli products on z, like OMEGAMON , can be ordered from Shopz in the form of a ServerPac.
You get the benefits of preinstalled datasets, and current maintenance already in those datasets, and the ability to configure the tools more quickly. Among the products supported include TWS for z/OS, IBM SA for z/OS, Tivoli Decision Support for z/OS (various components) , NetView for z/OS, ITCAM for SOA, and of course OMEGAMON (DB2, CICS, z/OS, IMS, Storage, Mainframe Networks, Messaging).

If you are pondering rolling out V4.20, this is a good way to get started quickly.

System z - Technology Summit

I will be presenting on IBM Tivoli Monitoring and the most effective uses of the TEP to manage z/OS as part of the System z - Technology Summit on Tuesday, February 9th in St Louis. This technology summit looks to be a nice event with multiple tracks on System z, Database, Application development, and of course Systems Management (that's me). If you're in the St Louis area, we would love to have you attend.

Here's a link on information, and how to register:

http://www-01.ibm.com/software/os/systemz/summit/

The System z - Technology Summit will also be in the following cities:
February 2, 2010 - Atlanta, GA
February 9, 2010 - St. Louis, MO
February 11, 2010 - New York, NY
February 18, 2010 - Toronto, Ontario
March 4, 2010 - -Chicago, IL
March 9, 2010 - Dallas, TX

Take advantage of custom queries

Earlier I had shown some examples of what I call Management By Exception displays, one example of an IMS view and another that showed an integrated z/OS view. One portion of the z/OS management by exception workspace showed DASD with high MSR times. Because in many shops you may have thousands of DASD devices, you may want to consider your options when displaying DASD information.

The data that appears in the various tiles of the workspace are gathered via a mechanism called a query. When creating a workspace that includes queries that can request potentially large amounts of data, it may be more efficient to create and use custom queries to make the rendering of the workspace more efficient. The DASD High MSR portion of the workspace is a good example of this technique.

If you look at the example, you can start with the basic default query, and then click 'Create Another' to make a new query to optimize. You can then specify a filter option, as in this example, to get information only for DASD devices with MSR times higher than 8 ms. You can then click 'Advanced' and get a pop up to specify the data sort sequence, and also in this example we are only asking for the 10 worst devices.

By filtering at the level of the query the workspace will render more quickly and efficiently. use this technique for displays that may request large amounts of information.

The value of persistence

Persistence has value, especially when we are talking about monitoring and situation alerts. When creating situation alerts, different monitored attributes have different characteristics. When choosing attributes to use for creating situations, keep in mind that some attributes may be more volatile than others.

In other words, some monitored information may be more "spikey". For example, certain resources, such CPU rates, page rates, I/O rates, and DASD MSR times may fluctuate quite a bit during normal workload processing. It's often times useful to monitor, and perhaps alert on these types of items. But, you don't necessarily want to send out a bunch of alerts every time CPU usage or page rate spike up.

That's where the persistence option comes into play. Like in the example I show here, in the situation editor you click on the 'Advanced' button. This will show the persistence option. The default is one interval, but you can click this option to set the persistence option higher. In the example I show a persistence option of 5. What that means is the situation logic has to be true for 5 intervals before the situation will fire. This has the net effect of helping tune out spike alerts, and reducing the occurrence of phony alerts.

Check out this upcoming webcast

There is an interesting new webcast coming up shortly, on January 28th. "Exploring your z10 and zOS systems with OMEGAMON XE for zOS" will talk about some of the powerful features of OMEGAMON z/OS.

Among the topics covered will be how to use OMEGAMON to: Monitor processor usage at the system and address space level, Manage resources and workloads across LPAR clusters, Identify service classes that are missing their performance goals, Monitor enqueues across systems and Sysplexes, Identify and ease bottlenecks that prevent a workload from achieving its service goal.

The speaker is Joe Winterton. Joe knows his stuff, so I'm sure this will be an excellent event. Here is the URL to sign up and attend:

http://www-01.ibm.com/software/os/systemz/telecon/28jan/

More on ITM 6.22

ITM 6.22 is the latest/greatest version of the IBM Tivoli Monitoring (ITM) infrastructure. While I presume you will find quite a few users running at the ITM 6.21 level, ITM 6.22 provides some interesting new features, some changes in how the TEP looks, and some useful new functons.

Here's a list of some of the enhancements provided by ITM 6.22:

The toolbar icons have been moved around in ITM 6.22

Historical data collection has been enhanced
The Historical Collection Configuration window has been redesigned, and looks quite a bit different. There is a new distribution method called Managed System (TEMA) that enables you to specify the managed systems that data collection will occur on. This is a biggie in that it gives you a lot more flexibility and control in terms of how and where history data will be gathered. There are also options for granular data collection with historical configuration object groups. Again, more granularity and control.

Managed system lists renamed to managed system groups

Modeling conditions for situations
Allows you to capture data from a query and use it to model possible threshold scenarios for situations.

Baselining added to charts for trend analysis
Bar charts, plot charts, and area charts have a new baseline tool. You now have support for such things as Statistical baselines, and Historical baselines.

Agent autonomy
Agent autonomy is the ability of a monitoring agent to execute independently of the TEMS. Aspects of this have been around for quite a while in the infrastructure. This just takes it to a new level in terms of having private situations and private history colelction that runs independent of the TEMS infrastructure.

Send SNMP traps
Send SNMP alerts from a TEMA directly to a receiver, without ever connecting to a monitoring server (pretty cool).

These are some interesting enhancements. If anyone is using ITM 6.22, I would like to hear about your experiences.

Integration of IBM System Automation (SA) and ITM

In prior posts I've discussed the Tivoli Enterprise Portal as an integration point for a variety of IBM Tivoli solutions: mainframe monitoring, distributed monitoring, ITCAM, storage solutions, NetView, you name it. One of the more interesting integration points is how IBM SA, System Automation, integrates with IBM Tivoli Monitoring (ITM) infrastructure. In IBM SA the INGOMX command has been enhanced to support sending commands in the form of XML requests to the SOAP server function of the TEMS (Tivoli Enterprise Monitoring Server). What this means is that from SA you can issue commands such as:

CT_Alert - Raise an existing situation
CT_Reset -Close a situation
CT_Acknowledge - Acknowledge a situation event
CT_Resurface - Change an acknowledged event to open
CT_WTO - Send a Universal Message
CT_Activate - Start a situation or policy
CT_Deactivate - Stop a situation or policy
CT_Execute - Execute a file on the TEMS server

This provides a very nice interface that allows SA to interact very directly with the monitoring infrastructure, and allows for more sophisticated and robust automation. The SOAP interface of the TEMS is a powerful and interesting feature that is well worth looking at.

More on OMEGAMON XE For CICS V4.2

I mentioned a while back that OMEGAMON XE for CICS V4.20 is now GA. With OMEGAMON CICS now at the V4.20 level that means all the core OMEGAMONs on z/OS have seen new versions be released in the past year.

Some of the new features of OMEGAMON XE For CICS V4.2 include:

Dynamic Terminal Integration Support. This is an extension of the Tivoli Enterprise Portal (TEP) client terminal emulator feature that allows you to launch from a Tivoli Enterprise Portal workspace to a related 3270 screen space in the target application. Predefined links have been provided with OMEGAMON XE for CICS on z/OS to OMEGAMON II for CICS screens.

Event forwarding and synchronization for Tivoli Enterprise Console® and Netcool/OMNIbus.

Improved migration. You can combine V3.1.0, V4.1.0 and V4.2.0 agents in your environment during migration.

ICAT Configuration tool improvements.

OMEGAMON XE for CICS on z/OS now contains the capability to monitor CICS Transaction Gateway V7.0.0 and higher for the mainframe. Now CICS TG support is bundled in the core OMEGAMON CICS tool, you don't need to purchase a separate tool to monitor CICS TG.

Consolidation of SMP/E FMIDs for OMEGAMON CICS installation.

Here's a link to the documentation for OMEGAMON CICS:

http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/index.jsp?topic=/com.ibm.omegamon.cics.doc_4.2.0/welcome_OMXEforCICS.htm

The return of the mainframe

An interesting article in the Economist magazine, of all places, on the IBM mainframe. The title is "The Return Of The Mainframe". It's a short, but interesting read. My only thought is, did the mainframe ever really go away? I've been in this business more years now than I care to admit to, and the mainframe has been purring away the whole time.


http://www.economist.com/businessfinance/displaystory.cfm?story_id=15276714

More on using policies in the TEP

Paul had a very good comment on the example that I used on policies managing the start/stop of situations. "ITM V6 has a new "Activity" located in the "Extensions" tab. It is called "Wait until a situation is False".I would add this to the policy after it starts the situation. Have it wait until situation EW_check_Prime_Time is False. This way , it will start the situation once and then wait. Without this, it will restart the situation each interval. (Interval of the situation EW_Check_Prime_time). This has two benefits. First, less overhead when you don't restart the situation every interval. Second, if the situation EW_Demo_DB2_Alert is true, restarting it would re-drive the event and make it fire True again."

Paul brings up a very valid point. If you referenced my article on policies (link on the right side of the page), I mention the loop aspect of policies and how it works relative to the interval defined on the situation. To control this loop mechanism, there is a feature I havent't really gone in to up until now, the "Wait Until xxx False" option. The example I show here demonstrates how it would be set up. In this example, we do the situation check, issue the command, then wait for the situation to go back to false before going back around to the start at the beginning of the policy again. This wait option is available from the "Extensions" tab in the policy editor.

Another example of management by exception

Here's another example of a management by exception display. Monitoring IMS with the management by exception approach works well. Often when I'm monitoring an IMS system I'm looking for such things as workload that is queuing, and resources that have failed or stopped/abended altogether. The example I show here is a typical example of an IMS display I might create using OMEGAMON IMS and the TEP. In this example I show message regions over a certain percentage busy, transactions that have queued, and PSBs and databases that have stopped or are in a problem status. Just like with the other example, I'm focusing on potential problem scenarios, and tuning out the background 'noise'.

Be sure to set your JRE settings when using the TEP

If you are logging on to the Tivoli Enterprise Portal (TEP) and you get this informational pop-up, don't tune it out and just click 'Continue'. It's telling you that you need to set some of your Java JRE settings for optimal use of the TEP. If you click on 'Help' it will take you to the online documentation that will spell out exactly what you need to do with the Java JRE settings. While it may not seem like a big deal, it can make a big difference in how well the TEP runs on your desktop.

Use Response Time (RTA) groups with OMEGAMON IMS and CICS

OMEGAMON XE for IMS and OMEGAMON XE for CICS provide powerful and useful Response Analysis and Bottleneck Analysis features. OMEGAMON XE for IMS and OMEGAMON XE for CICS each provide a mechanism to group together related transactions and programs, based upon the needs of the user and the nature of the applications being monitored. What this means is the user is able to optimize OMEGAMON to monitor and analyze CICS and IMS workload from the application perspective.

For example, the picture shows an example of how OMEGAMON IMS Bottleneck Analysis data is shown for transactions and programs related to the users ATM application. This means that when there is an issue with the ATM application, the user is able to determine the source of the potential bottleneck more quickly and precisely.

Definition of groups is driven by macro mechanisms, such as KOIGBLxx in OMEGAMON IMS, or may be specified in real time using OMEGAMON commands for 'on the fly' analysis.

The recommendation is to take advantage of this feature of the product to add more application relevance to the performance data. Many users simply do not exploit this feature.

Exploit policies to manage OMEGAMON situations

Policies extend alert command and control concepts established with OMEGAMON situations and add additional functionality to the Tivoli Enterprise Portal. While situations remain the essential starting point for alerts and automation, policies add essential function and flexibility to situation capabilities. Policies are probably one of the potentially most powerful, and yet most under utilized features of OMEGAMON and the Tivoli Enterprise Portal.

Policies are not a replacement for an automation engine, such as System Automation or AF/Operator. But they do provide a valuable extension to the command functions of situations, and enable the user to do things like issue mutiple commands, or sequences of commands based upon a situation being true. And, what is nice is this can be done using a GUI interface, and does not require coding things such as REXX code to get the job done.

One very interesting use of policies is as an overseer mechanism to manage the start/stop of situations within the Tivoli monitoring environment. Some situation alerts are sensitive to certain times of day or day of week considerations, due to operational or off-hours processing issues. In addition, some issues may be critical during prime time and less critical during off hours. You can reduce monitoring overhead and eliminate unnecessary alerts by running situations only when needed. The picture shows an example of an ‘Overseer’ policy, which manages situation start and stop. Using an overseer policy can simplify the coding and maintenance of the underlying situations, because the policy will be able to handle the time sensitivity logic.

If you want to find out more about policies, you can check out my article on policies (see the link on the right side of the blog page).