Upcoming System z technology summits

Another round of System z technology summits will be happening in the next few weeks. The System z technology summit is a nice event with multiple tracks on System z, Database, Application development, and of course Systems Management. There is a lot of good technical content (meaning not just marketing fluff). Upcoming cities include Columbus, Costa Mesa, and San Francisco. I will be presenting "Leveraging The Tivoli Enterprise Portal" at the event in Columbus.

If you are interested, we would love to have you attend. Here are the dates, places, and links to sign up for the events:

Columbus, OH - May 11 - www.ibm.com/events/software/systemz/seminar/TechSum1
Costa Mesa, CA - June 16 - www.ibm.com/events/software/systemz/seminar/TechSum3
San Francisco, CA - May 13 - www.ibm.com/events/software/systemz/seminar/TechSum2

Article on OMEGAMON XE For Storage V4.20

Here's a link to a nice article in IBM Systems magazine on the capabilities and facilities of OMEGAMON XE For Storage V4.20. The article provides a good overview of the tool, and goes into what was added in V4.20. Here's a link to the article:

http://www.ibmsystemsmag.com/mainframe/marchapril10/focusonstorage/32105printp1.aspx

OMEGAMON z/OS DASD monitoring considerations

We've discussed OMEGAMON CICS and DB2 DASD collections considerations. Now let's take a look at OMEGAMON z/OS. OMEGAMON z/OS collects it's device and performance information, such as MSR times, from it's API to RMF. RMF collects the device level statistics, and OMEGAMON displays and analyzes it. DASD information in OMEGAMON z/OS may be seen in both the 3270 (classic and CUA) and in the Tivoli Portal GUI interface.

Recently I was asked by a customer about a message that started showing up on their Tivoli Portal event console. Here is an example of what they were seeing, a situation called "KM5_No_SYSPLEX_DASD_FILTER_WARN". What this situation highlighted was that monitoring DASD devices without a filter that eliminates some of the devices can lead to potential high CPU or storage problems within the monitoring infrastructure. The situation notifies the user that OMEGAMON z/OS does not collect shared DASD device data unless a DASD filter situation has been created, and is active.

So the next question may be, how do you enable the DASD filter situation? The procedure is pretty well documented in the OMEGAMON z/OS User Guide. Here is a link to the appropriate documentation pages:

http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/index.jsp?topic=/com.ibm.omegamon_xezos.doc/km5cfmst102.htm

The Users Guide also has some good recommedations on situation settings, such as collection intervals. I suggest you take the time to review theses options. Keep in mind that the more data you collect, and the more frequently you collect it, the more potential monitoring overhead.

I will talk more about OMEGAMON z/OS DASD monitoring in a subsequent post.

Cost of monitoring considerations for DASD

DASD is a major area of interest when we are talking about information needed for performance analysis on z/OS, relative to considering the cost of monitoring on z/OS. When you look at the OMEGAMON suite on z/OS consider the following: OMEGAMON z/OS monitors DASD, OMEGAMON Storage monitors DASD, OMEGAMON DB2 monitors DASD, OMEGAMON CICS monitors DASD, OMEGAMON IMS monitors DASD. See a potential pattern here? Each of the core OMEGAMON monitors provides some form of DASD or I/O analysis, and this is for very good reasons. Understanding I/O is an essential component of understanding the performance of the system and the workload. The trade-off comes with understanding the optimal method to monitor DASD relative to the cost.

As the prior paragraph implies, if you have the full suite of OMEGAMON tools, you have the potential to monitor the same string of DASD devices multiple times. So that brings us to the first recommendation: monitor DASD once, not multiple times. If you use OMEGAMON z/OS as your primary DASD analysis tool, then you can consider being more selective when enabling DASD analysis in OMEGAMON CICS or OMEGAMON DB2. In a prior post I mentioned how you can leave Object Analysis in OMEGAMON DB2 off by default, but turn on as needed. That is a good strategy since there are times where the Object Analysis data is useful, but you probably don't need to collect it on an ongoing basis.

Now let's consider OMEGAMON CICS. If you have OMEGAMON z/OS, then you don't really need to set DASD exceptions in OMEGAMON CICS, plus you may want to carefully consider enabling I/O clocks and counters for CICS task history (ONDV - online data viewing). When setting up task history parms in the KC2GLBxx module, consider your options related to I/O. DB2_CLOCKS_AND_COUNTERS, as well as VSAM, ADABASE, DLI, etc. are potentially expensive data collection options. If you don't really need it, then don't turn it on.

There are more considerations specific to OMEGAMON Storage and OMEGAMON z/OS, and I will cover this in a later post.

An upcoming webcast on z/OS security management

I wouldn't be the first person to mention the strength of security on z/OS as one of its inherent advantages. RACF is powerful, detailed, and flexible. But, to be honest, I'm not sure I would use the term "ease of use" when talking about RACF. That's where tools like IBM Tivoli zSecure come into play.

On April 22nd, at 11 AM ET, there will be a free webcast on "Cost-effective mainframe security compliance with zSecure suite v1.11". The webcast will cover how you can reduce costs with simplified mainframe and RACF security administration, reduce exposures with automated security policy controls that enforce privileged user monitoring and other best practices, and manage risks with closed loop monitoring and remediation to reduce security threats. The speakers are Jamie Pease, IT Specialist, System z Security, and Glinda Cummings, Worldwide Tivoli Senior Product Manager .

If you are interested in simplifying z/OS security management in your shop, this might be a topic of interest. And the price of the webcast is right, it's free. Here's a link to sign up for the webcast:

ibm.com/software/systemz/telecon/22apr

Optimizing the cost of monitoring on z/OS

One of the things I commonly work on with my customers is optimizing the cost of monitoring. When you think about it, there is an inherent balancing act between the cost of monitoring, and the analytic value of the information collected and presented by the monitoring suite. If you ever read the classic 1966 Robert Heinlein novel "The Moon Is A Harsh Mistress" (one of my favorite books when I was a kid), the saying was "Tanstaafl", roughly meaning "there ain't no free lunch". Monitoring and performance management methods may adhere to this saying, as well.

I often get asked questions like "what is the monitoring overhead for the tool?", or "what is the cost if I enable a given function of the tool?". The answer is the eternal running joke of all technology vendors, "it depends". The answer, while often being a bit of a dodge, is actually a truism. The cost and overhead of the tool is often in direct correspondence to how the user chooses to employ and use the tool.

Now that I've done the "it depends" statement, there are some general rules of thumb that may be applied. Having a long standing DB2 and relational database background, there are certain analogies I sometimes draw between database optimization, and monitoring optimization. One of the key ones is, the more data I request and the more data I store and/or act on, the will result often times be a higher cost of collection, and potentially greater overhead. If I'm coding a DB2 Select call that results in a large tablespace scan than pulls back a large multi-million row result set, that will usually run longer and cost more than a singleton select that does index access and only pulls out a single row.

You can apply the same logic to monitoring. From a realtime perspective, if I'm monitoring the MSR (millisecond response time) for thousands of DASD volumes on an ongoing basis, that will be more expensive than if I just look at a single volume in detail, as needed. From a history perspective, the more history data I gather, and the more data I store as a result, the result may be a higher cost of history collection. But let's not forget alerting. The more alerts, and the more information I alert on, and the larger the number of managed systems I alert on, the result will potentially be a higher cost of alerting.

What I plan to do over the next few weeks is do a series of postings on this balancing act. In essence to help you answer the question: "what is the cost of monitoring, versus the diagnostic value of the data?". I will be covering all the core OMEGAMON tools on z/OS, and we will look at real time collection, historical collection, and alert management.

OMEGAMON IMS V4.20 provides new situation options

One of the nice enhancements in OMEGAMON IMS V4.20 was finally getting Bottleneck Analysis information in the TEP. This is useful for a variety of reasons. Bottleneck Analysis is a key analytic component of the tool, and provides the user with information on where the workload is spending its time, therefore giving you an indication of where you may want to direct your tuning efforts.

Plus, having Bottlenck Analysis data in the TEP, means the data may be used for situation creation. Here is an example of how to create a situation using Bottleneck Analysis data. By creating a situation like this you can look for workloads with high percentile wait times (such as waits for CPU, waits for locks, or waits for other resources), and include that information in the situation logic, making the siutation more meaningful and flexible.

A handy link to find Tivoli documentation

I often get requests for where to find links for certain documentation, such as OMEGAMON doc. Here is one link that pulls together the doc links for a whole bunch of IBM Tivoli products. I have this link on the right side on the blog page under "Useful Links".

http://www.ibm.com/developerworks/wikis/display/tivolidoccentral/Home

An easy way to generate OMEGAMON DB2 report JCL

OMEGAMON DB2 (the original Candle product) and IBM DB2 PM/PE were merged together into a single set of solutions a few releases ago. The current solution is now called OMEGAMON XE for DB2 PM/PE, to reflect that the tools have been merged.

When the merger occurred, the historical reporting mechanisms of DB2 PM/PE became the core historical functions of OMEGAMON DB2. If you were a new user to the solution, that meant you had to learn a new reporting language, and create new reporting JCL. But, there is an easy way to create OMEGAMON DB2 reporting JCL. If you invoke the ISPF interface that comes with OMEGAMON DB2, you have an option to generate reporting JCL using the ISPF interface.

The example shown here shows how you can invoke this function. You then specify what reports you want, and hit F5 to generate the JCL. You can then edit and submit the resultant JCL.

This is a convenient and easy way to get started with the history functions of OMEGAMON DB2, and quickly create reports using the tool.

Recommended maintenance levels for OMEGAMON

The following link is to a page which pulls together a lot of useful information on recommended maintenance levels for the various OMEGAMON monitoring tools (both V4.10 and V4.20). There are also links on this page to interim fixes, and other interesting stuff.

http://www-01.ibm.com/support/docview.wss?uid=swg21290883&myns=swgtiv&mynp=OCSSQH9J&mynp=OCSSZJZF&mynp=OCSSRLD6&mynp=OCSSZ8F3&mynp=OCSSSQ3B&mynp=OCSSUSPS&mynp=OCSSUSPA&mynp=OCSSXS8U&mynp=OCSS2JNN&mynp=OCSS2JFP&mynp=OCSS2JL7&mynp=OCSSLSDR&mync=R

Documentation on event forwarding in the TEP

In some prior posts I have talked about forwarding events from the Tivoli Portal (TEP) to OMNIbus. Here is a link to a nice document that covers the configuration of the event forwarding mechanisms:

http://www-01.ibm.com/support/docview.wss?uid=swg21330541&myns=swgtiv&mynp=OCSSQH9J&mynp=OCSSRLD6&mynp=OCSSSQ3B&mynp=OCSSUSPS&mynp=OCSSUSPA&mynp=OCSSXS8U&mynp=OCSS2JNN&mynp=OCSS2JMJ&mynp=OCSS2JFP&mynp=OCSS2JL7&mynp=OCSSLSDR&mync=R

Historical baselines in the TEP

ITM6.2.2 introduced several visual baseline functions to the Tivoli Enterprise Portal. The list of baseline functions includes the following:

Model Situation - create a situation using historical data and statistical functions to “model” and select optimal threshold values

Statistical Baseline - enables the results of statistical functions to be drawn as lines in the chart to visually determine what is normal in an environment

Monitored Baseline - Situation thresholds can be visualized in a bar, area, or plot chart. Allows a user to track in real-time how a metric is performing relative to its threshold.

Historical Baseline - Work with detailed (not summarized) historical data from the data warehouse.

The example shown here shows a statistical baseline as applied to an OMEGAMON IMS repsonse time chart. Here we see, in the lower left corner of the workspace, an area chart of IMS response time, and then I added a statistical baseline of average reposnse time in the form of a line across the area chart. The line makes it easier to see when response time exceeds the average.

Upcoming System z Storage Directions and Best Practices

IBM will be doing a series of seminars on "System z Storage Directions and Best Practices". The seminar will cover a briefing on the latest announcements in mainframe storage and then will discuss the financial benefits of using IBM Tivoli software to manage stored data. The seminars are free and will take place at the following dates/locations (plus I've provided a link for each location to sign up):

April 6th -- Minneapolis, IBM Offices, 650 3rd Avenue South
https://www-950.ibm.com/events/wwe/grp/grp004.nsf/v16_agenda?openform&seminar=ZZZA3YES&locale=en_US

April 7th -- Chicago, Hyatt Building, IBM Offices – Sixth Floor, 71 S Wacker Dr
https://www-950.ibm.com/events/wwe/grp/grp004.nsf/v16_agenda?openform&seminar=4Q2ABUES&locale=en_US

April 8th -- Hazelwood (St Louis), IBM Offices, 325 James S McDonnell Blvd, Building 300
https://www-950.ibm.com/events/wwe/grp/grp004.nsf/v16_agenda?openform&seminar=CEBG3VES&locale=en_US

Upcoming webcast on OMEGAMON CICS V4.20

There will be a webcast in April on "What's New in OMEGAMON XE for CICS on z/OS V4.2.0". The presentation will cover key new features and functions in the tool including: customer enhancements, performance enhancements, exploitation of the new features of CICS Transaction Server v4.1, and repackaging to include full monitoring capabilities of CICS Transaction Gateway z/OS as part of the core OMEGAMON CICS monitor.

The presenter is Richard Burford, and he really knows his stuff in the area of CICS. The webcast will happen at 11 ET on April 15th. Here is a link to sign up for the event:

http://www-01.ibm.com/software/os/systemz/telecon/15apr/

Introduction To System z Class

IBM will be offering an "Introduction To System z Overview" class in Chicago in May. I've looked over the class agenda, and it looks like a very good overview of the technical capabilities of z. The class is a 3 day event, May 25th through the 27th, at the Hyatt Center in downtown Chicago. It's intended primarily for "newbies" to z/OS. And the price is right. It's free!!!

Here's the topics to be covered:

Introducing the Mainframe Environment
o System z Architecture and Hardware
o z/OS Overview
o Systems Management
o Security Server
o TSO/E and ISPF
o UNIX System Services
o Working with Datasets
o Using JCL and SDSF
o Web-based Workloads
o Bulk and batch workloads
o ESB workloads
o Consolidated workloads (z/Linux and z/VM)
o Application Programming
o Transaction Management Systems
o Database Management Systems
o WebSphere Brand

For more information & where to register, please contact:
Ahsan Saiyed at 312-529-1602 or aksaiyed@us.ibm.com
Patrick Knott at 314-252-4069 or gpknott@us.ibm.com

First Come, First Serve enrollment.

Changes to OMEGAMON DB2 Thread Cancel

There have been some changes recently to OMEGAMON DB2 in terms of how it handles authorization to execute a CANCEL THREAD command. Here's the story.

PK90400 enhanced OMEGAMON DB2 so that when the product is configured for external security, the userid of the logged on user is passed to DB2 to determine the DB2 authority to issue a CANCEL THREAD command. But, when instead OMEGAMON DB2 is configured for internal security the OMEGAMON task id is passed to DB2 for the authority to cancel a thread. Prior to PK90400 the CANCEL THREAD command was always issued under the authority of the OMEGAMON task.

There is now an APAR to provide a user configurable option to allow OMEGAMON to always issue the CANCEL THREAD under the authority of the OMEGAMON task, even if configured for external security.

Crystal clear, huh? It is an important thing to be aware of, becuse the authority to cancel DB2 threads is probably something you want to have properly secured.

Here is a link for more info:

http://www-01.ibm.com/support/docview.wss?uid=swg1PM07797&myns=swgtiv&mynp=OCSSUSPS&mynp=OCSSUSPA&mync=R

New article in IBM System z Advisor

I just published an article in the IBM System z Advisor on "Leveraging OMEGAMON XE and the Tivoli Enterprise Portal to create Management By Exception Views". This is a more detailed discussion of some of the posts I've made earlier in this blog on how to use the TEP to create what I call Management By Exception workspaces.

Here is a link to the article:

http://www-01.ibm.com/software/tivoli/systemz-advisor/2010-03/leveraging-omegamon-xe.html

Upcoming OMEGAMON webcast

I will be doing a webcast on "What’s new and exciting In OMEGAMON XE for IMS V4.2.0" this coming March 25th at 11 AM ET. I will be spending time on the new enhancements to the tool, and also looking at some of the new capabilities you get with ITM 6.22, and how to exploit them in OMEGAMON.

It's a freebie, and here is the URL to sign up for the event:

http://www-01.ibm.com/software/os/systemz/telecon/25mar/

Virtual Pulse

If you missed out on going to the Pulse conference in Vegas, you can still take advantage of the Pulse Virtual Online event. This event will give you a chance to check out some of the same material that was covered at Pulse.

Here is a link for the Pulse Virtual event:

http://vshow.on24.com/clients/ibm/pulse2010

RLIM enhancements to OMEGAMON XE For CICS V4.20

I've posted earlier on the new features of OMEGAMON XE for CICS V4.20 (now GA by the way). One nice enhancement I didn't mention is you now have the ability to use RLIM (the resource limting facility) to automatically start an application trace. This means that if you have an outlier transaction running in a CICS region, RLIM can invoke the trace and hopefully capture at least a portion of the problem tran. To take advantage of this option you need to set your RLIM options carefully, but it has potential for certain customers. I will do a nother post on RLIM in more detail later this month.