Collecting appropriate log data for IBM support

When you are having an issue with the IBM Tivoli Monitoring infrastructure, IBM support will often ask you for various logs for analysis purposes. There is a procedure called PDCOLLECT that will aid in gathering all the various logs you will commonly need. Here is a link to a write-up on how to use PDCOLLECT:

http://www-01.ibm.com/support/docview.wss?uid=swg21446655&myns=swgtiv&mynp=OCSSZ8F3&mync=R

New capability for the z/OS Event Pump

The Event Pump for z/OS is an IBM tool that enables the integration and forwarding of z/OS events from z/OS to IBM Netcool OMNIbus. The Event Pump can pass event information, plus resource status, and the severity of an event to OMNIbus. The mechanism is very similar to how OMEGAMON XE uses the EIF interface to pass alerts to OMNIbus.

There is one APAR of interest that just became available last week, OA34085. This particular APAR enables the integration of BMC Mainview alerts into the Event Pump. So the bottom line is, if you are BMc Mainview customer, and you have IBM OMNIbus for event management, you can now more easily forward alerts and events from Mainview into OMNIbus.

For more information, here is a URL:
http://www-01.ibm.com/support/docview.wss?uid=swg1OA34085&myns=swgtiv&mynp=OCSSXTW7&mync=R

Tips on customizing "Take Action" in the Tivoli Portal

The "Take Action" function in the Tivoli Portal (TEP) can be very useful. One of the nice things about "Take Action" is that it is very flexible and customizable. You can easily set up a pre-arranged set of commands and then select the desired command from a drop down list. One thing that makes "Take Action" flexible is the ability to use the attribute substitution function as part of the command definition. What this means is that you can have the Action command insert an attribute where needed to customize the command to be submitted. Here is an example of how you can use this function, in this example to set up a DB2 Thread Cancel function.

Webcast on "Monitoring Options in OMEGAMON XE for Messaging"

There will be a free webcast on October 7th on "Monitoring Options in OMEGAMON XE for Messaging" . The time will be 11 a.m., Eastern Daylight Time, 3:00 p.m. GMT.

The teleconference will discuss monitoring options to efficiently detect and identify root causes of your WebSphere MQ and Message Broker performance issues.

Here is a link to sign up for the event:
http://www.ibm.com/software/systemz/telecon/7oct/

z/OS 1.12 available shortly

z/OS Version 1 Release 12 will be ready for ordering as of Friday, September 24th.

z/OS V1.12 can provide automatic and real time capabilities for higher performance, as well as fewer system disruptions and response time impacts to z/OS and the business applications that rely on z/OS.

Interesting "new stuff" includes new VSAM Control Area (CA) Reclaim capability, a new z/OS Runtime Diagnostics function is designed to quickly look at the system message log and address space resources and can help you identify sources of possible problems, z/OS Predictive Failure Analysis (PFA) for managing SMF, performance improvements for many workloads, XML enhancements, networking improvements, and improved productivity with the new face of z/OS called the z/OS Management Facility (5655-S28).

Here is a link for more information on z/OS V1.12:

http://www-03.ibm.com/systems/z/os/zos/

A new photonic chip

While this does not have anything to do specific to z/OS or Tivoli, I thought it was an interesting read in Financial Times.

"A new photonic chip that works on light rather than electricity has been built by an international research team, paving the way for the production of ultra-fast quantum computers with capabilities far beyond today’s devices."

Sounds like an interesting breakthrough. Here's a link to the story:

http://www.ft.com/cms/s/0/8c0a68b0-c1bc-11df-9d90-00144feab49a.html

OMEGAMON XE For Storage V4.20 Interim Feature 3 enhancements

OMEGAMON XE For Storage V4.20 Interim Feature 3 has some interesting new enhancements. Some of these enhancements include the following:

DFSMShsm Common Recall Queue support
Display request info for all queued and active requests in the CRQPlex on a single workspace
Enable cancelling HSM requests from the CRQPlex Request workspace - even across systems
Provide Storage Groups and User DASD Groups space used stats in units of tracks and cylinders Multi-volume datasets now displayed as single entity in the Dataset Attribute Database reports Reports will now contain a column indicating whether a dataset is multi-volume or not
For a multi-volume datasets, space data will be summarized in a single row
Ability to identify TotalStorage array problems at the ranklevel
Situation alerts for DDM Throttling, Raid Degraded condition and RPM Exceptions
Support for issuing Storage Toolkit commands at a group level

If you have OMEGAMON Storage, here is a link for more info:
http://www-01.ibm.com/support/docview.wss?uid=swg24027743&myns=swgtiv&mynp=OCSS2JFP&mync=R

Getting started using ITMSUPER

I've done some earlier posts on using ITMSUPER, and some of the useful information that is contained within ITMSUPER. ITMSUPER is a very useful utility that is easy to use, and provides quite a bit of diagnostic information as to what is happening within your Tivoli monitoring infrastructure.

Here is a link to a brief "Getting started using ITMSUPER". This procedure includes a link to where you can go to get ITMSUPER.

Here's the link:
http://www-01.ibm.com/support/docview.wss?uid=swg21444266&myns=swgtiv&mynp=OCSSZ8F3&mync=R

Take advantage of snapshot history in OMEGAMON DB2

OMEGAMON DB2 has a very useful snapshot hsitory feature that is very granular and detailed. This particular history mechanism is different than the snapshot hsitory collected by the XE agent task for Tivoli Data Warehouse (TDW) history collection. The OMEGAMON DB2 snapshot hsitory can be collected on a frequent basis, provides considerable information, and is very useful for problem analysis. To access the OMEGAMON DB2 snapshot history, you need to enable and install the PE GUI interface (and yes you have access to this tool regardless of whether you run the PM or PE version of the product).

The example I have here shows how you can use the tool. Here I show two different views of DB2 active threads. The top one is from the current view, and the other is from several minutes prior. To change the history view you move the scroll bar I have highlighted. What's neat is you can easily move the scroll bar back and forth to get different point in time views of the system.

This feature is very useful, in particular for analyzing those problem scenarios where being able to go back in time and see the mix of applications running at the same time is key to understanding what is going on.

Upcoming webcast on NetView and zEnterprise

On September 30, 2010 at 11 AM, Eastern time, there will be a webcast titled "Tivoli NetView for z/OS in zEnterprise".

In this complimentary teleconference you can learn how IBM Tivoli NetView for z/OS addresses critical issues, including complexity, by providing the foundation for consolidating and integrating key service management processes in your zEnterprise environment. You’ll see how Tivoli’s NetView for z/OS-based integrated solutions can help you deliver value by improving the availability and resiliency of zEnterprise systems and applications, reduce the need for operator intervention, and fine-tune service delivery. With less unplanned downtime, there’s less impact on your business.

The speakers are Mark Edwards, Senior Product Manager, IBM Software Group and Larry Green, NetView for z/OS Architect, IBM Software Group.

Here is a link to sign up for the event:

http://www.ibm.com/software/os/systemz/telecon/30sep/index.html?S_TACT=100GV43M&S_CMP=5x5

Leveraging the Situation Console

The Situation Event Console will show the situations open in a given monitoring environment, and provide drill downs for details on the situation alert. By default, the Situation Console is provided for the entire enterprise on the product provided Enterprise workspace.

What's nice is you can implicitly filter and optimize the Situation Event Console for your specific requirements, and the types of alerts you need to see. In this example I made a change to the product provided DB2 Messages workspace. I split the top DB2 message window, and then did a click and drag from the tool bar, and dropped the Situation Event Console icon on the DB2 workspace I'm editing. The result is now I have a Situation Event Console filtered for just DB2 alerts. You can do the same thing, for other managed system types, as well. This technique is an easy way to tune out the noise, and target the information you are most interested in when it comes to situation alerts.

OMEGAMON DB2 Messages Workspace

OMEGAMON DB2 V4.1 added support for DB2 message logging and management to the Tivoli Portal as an interim feature. The DB2 message feature has some interesting and useful capabilities, such as highlighting application failure/abend messages and tracking Deadlock/timeout/escalation messages. The default workspace will show the last 10 messages, and will highlight typical problem messages. But, as with any Tivoli Portal workspace, you can easily customize the workspace to your specific requirements.

Another nice usage of the DB2 Messages workspace is the ability to create situations based upon DB2 messages. In the example I show here I created an alert based upon a DSNL027I message. Notice also that you can take advantage of the ability to highlight information, such as for the DSN3201I error message.

If you want to try out the DB2 Messages feature, but do not see any messages appearing in the workspace, check on the following command:

F cccccccc,F PESERVER,F db2ssid,DB2MSGMON=Y

where ccccccc is the OM DB2 collector task, DB2 ID would be the DB2 you want to collect messages from.

The above modify will enable message collection to occur, and you should be able to see data in the workspace.

About Policies

Policies are an interesting and powerful feature of the Tivoli Enterprise Portal. Common usages of policies include such things as enabling a situation to issue multiple commands when true, stopping and starting situations as needed, and using multiple checks and command options with a single command flow. Policies provide a way to expand the command capabilities of the Tivoli Portal.

There are some things to consider when using policies. First, be aware that situations that are embedded within the policy logic are 'copies' of the original situation. In other words, if you take a commonly used situation and embed it within a policy, that situation logic will be run twice, once for the situation itself, and once for the policy. That, in and of itself, may not be a problem. But, be aware that if you are using a higher cost situation in a policy, you will be using that higher cost situation twice. Second, situations usually run within the agent task, but policies run within the TEMS infrastructure. Third, similar to the interval concept of situations, policies also have an inherent loop execution logic.

To know if any policies are connected to a given managed system, you can right click on the managed system within the Tivoli Portal and select 'Manage policies' to see what policies have been deployed.

Upcoming webcast on OMEGAMON installation and troubleshooting

On September 14th there will be a webcast on "IBM Tivoli OMEGAMON® XE for z/OS from Installation to Troubleshooting – PART1". This webcast will cover the following:

Omegamon XE for zOS:Installation and Configuration
Omegamon XE for zOS:Usage
Omegamon XE for zOS:Troubleshooting

Here is a link for more information on the event:

http://www-01.ibm.com/support/docview.wss?uid=swg27019462&myns=swgtiv&mynp=OCSS2JNN&mync=R

More on OMEGAMON z/OS currency maintenance

While browsing through my Google reader I noticed an entry titiled "ABENDs after upgrading level of z/OS". The symptom is various ABENDs (i.e. S0C1, S0C4, U0012, U1213, etc.) in the TEMS or in any of the OMEGAMON agents after upgrading your level of z/OS.

The bottom line is when you upgrade your level of z/OS, you need to be sure to apply OMEGAMON Currency PTFs to support that new level of z/OS, AND (let's not forget the AND) OMEGAMON Currency PTFs for any level of z/OS you skipped over. If you skip a level of z/OS (i.e. upgrade from z/OS 1.9 to z/OS 1.11), you need to apply the OMEGAMON Currency PTFs for the level(s) you skipped as well as the level to which you upgraded.

Here is a link to the document, and the document in turn includes links to recommended maintenance levels for z/OS 1.10, 1.11, and 1.12.

http://www-01.ibm.com/support/docview.wss?uid=swg21439161&myns=swgtiv&mynp=OCSS2JNN&mync=R

Using ITMSUPER to understand the cost of situation processing

ITMSUPER is an excellent tool that can provide tremendous insight into what is happening in the IBM Tivoli monitoring infrastructure. ITMSUPER is available from the OPAL web site (see the link to OPAL on the right of this page). OPAL is a good source of handy tools and other goodies. ITMSUPER is one of the most useful.

There are many uses for ITMSUPER, analyzing situation processing is just one. In the example I show some of the typical output from ITMSUPER. I clicked on the line in the middle of the display "Cost of running situations". This display shows information on what situations are running within the given agent (TEMA) task. Note that the display also provides information on the situation interval, number of rows processed for the situation, and a relative cost of running the situation per hour. This is very good information to use to determine which situations are potentially more costly to run than others.

Upcoming webcast on z/OS storage management

On August 19th there will be a webcast on "IBM Tivoli System z Storage Management update: An integrated toolset for better insight, analysis and control".

This event will cover the IBM Tivoli storage management suite of solutions. In this session, examples to be discussed will include: how to pinpoint a critical address space not performing well and in real time and identify all the data sets and devices that the address space is using, reveal hidden errors in HSM control data sets that can result in data not being backed up and being unavailable when needed, maintenance of ICF catalogs to avoid costly downtime, and optimization of your environment with policy-based control over DASD allocation.

The webcast is a free event. Here is the URL to sign up:

http://www-01.ibm.com/software/os/systemz/telecon/19aug/index.html?S_TACT=100GV41M&S_CMP=5x5

OMEGAMON XE For IMS Transaction Reporting Facility overhead considerations

The Transaction Reporting Facility (TRF) component of OMEGAMON XE For IMS is used to create information needed for chargeback and IMS performance analysis. There can be overhead considerations when enabling Transaction Reporting Facility. The following APAR, OA33784, mentions overhead considerations when the DB2 collection option is enabled. Specifically, if the user is running BMPs, there will be additional TRF overhead for the DB2 collection portion, whether the BMP option is set to ON or OFF.

If you are running TRF you will want to take a look at this APAR:

http://www-01.ibm.com/support/docview.wss?uid=swg1OA33784&myns=swgtiv&mynp=OCSSXS8U&mync=R#more

OMEGAMON currency maintenance for z/OS 1.12

OMEGAMON currency support for z/OS 1.12 is being provided for both OMEGAMON Versions 410, 420 as well as later releases of OMEGAMON XE products on z/OS.

To have currency for z/OS 1.12, you will need to be fairly current on maintenance. Also, there will be maintenance that applies to common code shared across multiple OMEGAMON tools.

For a link to information on the recommended maintenance for z/OS 1.12:

http://www-01.ibm.com/support/docview.wss?uid=swg21429049&myns=swgtiv&mynp=OCSSRJ25&mynp=OCSS8RV9&mynp=OCSSRMRD&mynp=OCSS2JNN&mynp=OCSS2JFP&mynp=OCSS2JL7&mync=R

Also, here is a link to a forum if you have questions: https://www.ibm.com/technologyconnect/pip/listforums.wss?linkid=1j3000

Situations and their impact on the cost of monitoring

Situations can have an impact on the resource usage of the OMEGAMON agent (TEMA) tasks.

Referring back to an earlier post, I mentioned the notion of the more I do, the more it will likely cost. The more data I request and the more data I store and/or act on, the will result often times be a higher cost of collection, and potentially greater overhead. The more alerts, the more information I alert on, the more rows of information I potentially alert on, and the larger the number of managed systems I alert on, the result will potentially be a higher cost of alerting. This cost of alerting will often be seen in places such as the TEMA address space.

To easily see how situation processing is impacting a managed system, from the Tivoli Portal you can right click on a managed system and select 'Manage Situations' (see the example). The pop-up that you get will show what situations are distributed to the managed system, plus some other very interesting information about the situations.

There is some very interesting information that this pop-up shows, as well. One column shows the interval that the situation executes on. The tighter the interval, the more work the TEMA has to do to handle the situation. Notice also, that there are several different intervals for the situation. Many of them are running on a 30 second interval, others on 1 minute, others on different intervals. One thing to be aware of is situation optimization. If you have multiple situations referencing the same table of information, the Tivoli infrastructure has the ability to optimize the situation checks, by doing one check versus multiple. However, this will work only if the situations are on the same interval.

Another apsect of situation optimization, is that if a situation that has 'Take Action', it is not eligible for this optimization. If you have many situations with 'Take Action', this will potentially significantly reduce the potential benefit of this function. One suggestion is, if you have a component such as OMNIBus, to consider using the EIF interface, versus 'Take Action' to drive alert notification. Using the EIF option will not inhibit situation optimization.