Problems with historical data not showing up in the TEP?

This is an interesting Technote that I read recently. Apparently some Tivoli Portal users who were collecting history were seeing gaps in history collection data, or history data missing altogether. The Technote describes the eyecatchers to look for if this issue is occurring, and then provides a workaround/resolution.

The net is if you are missing data, and you see a long time (in the range of multiple minutes or more) between these messages in RKPDLOG of the z/OS TEMS or TEMA (i.e. the agent task on z/OS), then you may want to check things out. Here are the messages (an init started followed by an init complete message):

yyyy/mm/dd hh:mm:ss KPDINDS: Initialization started for DataStore file DSN:&rhilev.&rte.lowlev

yyyy/mm/dd hh:mm:ss KPDINDS: Initialization complete for DataStore file DSN:&rhilev.&rte.lowlev

The workaround? Go into the KxxPCTL (xx meaning the product code for the agent type) for the task in question, and add the parm BATCHINIT like in the following example:

MAINTPRC PROC=KPDPROC1 GENPROC BATCHINIT

The BATCHINIT parm indicates that initialization of a datastore file should be done by the batch job instead of being done within the PDS processing of the TEMA or TEMS started task.

Interested in DB2 V10?

If you are interested in DB2 V10, here is a webcast on what's in DB2 V10 and customer experiences with DB2 V10. The event will have information on such things as, temporal tables, how hash access can speed up queries against large tables, new SQL features, how access path determination is now smarter, and how concurrency can be improved without sacrificing integrity.

This webcast is being put on by International DB2 User Group (IDUG). The event is on November 2nd at 11AM Eastern time. Here's a link to register for the webcast:

http://www.bulldogsolutions.net/IDUG/IDU1001/frmRegistration.aspx?bdls=29505

New release of the zSecure security suite

On October 4, IBM announced a new release of the zSecure solutions (V1.13) to assist with compliance monitoring and auditing, audit usage and configurations, enforce policy compliance, help administer mainframe server security, and monitor for threats. zSecure can help to centralize and consolidate security administration, and to leverage the mainframe as the enterprise security hub.

The new release features the following line items which also enhance the security intelligence of CICS, IMS, and DB2 applications:

Automates security analysis of CICS and IMS transactions and programs
Automated analysis of SAF classes being used by DB2, CICS, or IMS
Enhances Access Monitor and allows you to improve data consolidation
Allows annotating userid displays with data from external human resource files
Globalization enhancements to support international language support and auditing
Allows addition of your own sensitivity classification, audit concern, and priority
Supports currency with z/OS V1R13, ACF2 R14 and R15, CICS V4R2, and Top Secret R12, R14, and R15
Extends integration with Communications Server and provides various interface improvements

If you are interested in zSecure here is a URL to check out:
http://www-01.ibm.com/software/tivoli/products/zsecure

Here's a link to a white paper on "Consolidating security across platforms with IBM System z":
http://www.servicemanagementcenter.com/main/pages/IBMRBMS/SMRC/ShowCollateral.aspx?oid=80003

Here's a link to a Red Book on "z/OS Mainframe Security and Audit Management using IBM Tivoli zSecure" :
http://www.redbooks.ibm.com/Redbooks.nsf/WebComment?OpenForm&ParentUNID=6A07891F5A43E3DC8525741000507EAA

Check out the IBM System Availability Roadshow coming in October

IBM will be putting on a series of events in various cities, the "IBM System Availability Roadshow". It's an all day event and the agenda looks pretty solid (i.e. it's not a bunch of marketing fluff). Here's the agenda:

System Availability Overview
Explore the significant role that System z availability plays in
meeting your business objectives.

Creating End-to-End System Availability
Learn through key value scenarios how Tivoli integrated
solutions – including NetView for z/OS, OMEGAMON, and
System Automation – enable you to maintain optimal service
availability and performance.

Automation Deep Dive and Key Integration
Understand how Tivoli automation solutions improve the
availability of your system, network, and applications; thereby
reducing operator errors and maximizing resource efficiencies.

Network Management Deep Dive and Key Integration
Gain insights into how NetView for z/OS and OMEGAMON
solutions enable you to manage and optimize the availability and
performance of your TCP/IP and SNA networks.

Best Practices - Tailoring the solution for your needs
Utilize a problem solving approach to demonstrating how key
visualization, control, and automation capabilities of NetView for
z/OS based solutions can be tailored to meet the ever changing
demands of your enterprise.

The event will be in the following cities:
Tues, Oct. 4 Southfield 18000 W 9 Mile Rd, Rm 1505 Southfield, MI
Thurs, Oct. 6 Chicago 71 S. Wacker Drive, 6th Floor Chicago, IL
Tues, Oct 11 Markham 3600 Steeles Ave. East, D109 Markham, ON
Thurs, Oct 13 New York 590 Madison Avenue, Rm 1218 New York City
Tues, Oct 18 Hartford One Financial Plaza, 12th Floor Hartford, CT
Wed, Oct 19 Raleigh 3100 Smoketree Court, Rm 9002 Raleigh, NC
Thurs, Oct 20 Washington, DC 600 14th St, NW, Rm 356 Washington, DC
Tues, Oct 25 Atlanta Hillside 4111 Northside Parkway, 03F23 Atlanta, GA
Thurs, Oct 27 Dallas 13800 Diplomat Dr, Rm 232 Dallas, TX
Tues, Nov 1 San Ramon 4000 Executive Pkwy, Rm 340 San Ramon, CA
Thurs, Nov 3 Costa Mesa 600 Anton Blvd, Rm 209 Costa Mesa, CA

If you are interested, email Joanie Gines jgines@us.ibm.com to enroll.

An example of using the OMEGAMON z/OS CPU Looping Index indicator

You still run into shops where they have jobs going into loops, and it may be necessary to identify and address these tasks. OMEGAMON For z/OS V4.20 added an interesting number called the CPU Loop Index. The CPU Loop Index is a number that is calculated at the address space level that shows the relative likelihood that a given task on z/OS may be in a CPU loop. The notion is the more CPU heavy a task is, without any other indications of activity, such as I/O, the greater the likelihood that the task may be in a loop. The loop index is calculated up to 100. The closer to 100, the higher the possibility of a loop. Now it's not a guarantee that the task is in a loop, but it at least gives you an indication that the task is work taking a look at.

Here's an example of how I'm using this measure with a customer to highlight possible looping tasks. The example I have here is a customized availability view TEP workspace. This workspace is pulling in information from a variety of sources, including OMEGAMON, Tivoli Workload Scheduler, and System Automation to highlight potential issues, problem applications, and long running jobs.

To add the CPU Loop Index to this display, we made a copy of the standard OMEGAMON query for this table, and added a filter to it to only show jobs with an index greater than 50. This is recommended for performance reasons, since you will usually have hundreds of tasks on a z box. Why render all that data you don't need? We then apply a filter to the workspace to just show tasks above a certain level (such as 90 in this example).

It's an interesting number, feel free to take a look at it.

Upcoming webcast on OMEGAMON and the Tivoli Enterprise Portal

On September 29th I will be doing a webcast, "Top 10 Problem Solving Scenarios Using OMEGAMON and Tivoli Enterprise Portal". In this presentation I will cover how OMEGAMON and the Tivoli Enterprise Portal can be used to identify, isolate, and solve typical z/OS performance and availability management issues.

The session will include examples of how to get the most out of OMEGAMON and the Tivoli Enterprise Portal, and discuss the integration of OMEGAMON with other Tivoli solutions. I will also be covering some things I've been blogging about lately, such as ITM 6.2.3 and the Tivoli Performance Analyzer.

The event is September 29th at 11AM. The price to attend is right (it's free). Please feel free to stop by.

http://www.ibm.com/software/systemz/telecon/sep29

Summary of what's in ITM 6.2.3

As I mentioned a few days ago, ITM 6.2.3 is now available. There are quite a few interesting enhancements in ITM 6.2.3. I've mentioned a couple things already, including self-describing agents and Tivoli Performance Analyzer. Here's a summary of the more interesting stuff in ITM 6.2.3:

§Application Support Mismatch

–Show new or updated application support event (Note – not yet supported on z/OS yet)

§Tivoli Performance Analyzer now a base component of IBM Tivoli Monitoring

–Adds predictive capability to Tivoli Monitoring

–Monitor resource consumption trends, anticipate future performance issues

§Situation event console and Common event console enhancements

§Manually started situation enhancements

–

§Historical data compression before upload to TDW

–Reduce network traffic, data collected from distributed monitoring agents is now compressed in memory at the collection location

§New and enhanced CLI tacmd commands

§Tivoli Application Dependency Discovery Manager policies

–Create policies that interact with your Tivoli Application Dependency Discovery Manager (TADDM) environment

–Policies can forward events such as, new, changed, or destroyed resources to your Tivoli Application Dependency Discovery Manager server

Some interesting stuff, in particular the interface with TADDM. Imagine the notion of being able to integrate configuration and change management with monitoring? After all, what's the biggest cause of unplanned outages? How about changes.

§

Check out Destination z

"Destination z" is a community consisting of Systems z users, educators, business partners and IBMers. Destination z is intended to spread the word on the latest on System z technology, trends and resources.

There are lots of good links, and sources of information. Membership in Destination z is open to System z customers, IBMers, IBM business partners and qualifying academic institutions (educators and students).

To join go to: www.destinationz.org/join

What is Tivoli Performance Analyzer?

Earlier this week I mentioned that ITM 6.2.3 would be bringing in new feature/function. Tivoli Performance Analyzer in the past has been an add-on component to the ITM infrastructure. With ITM 6.2.3, Tivoli Performance Analyzer becomes part of the base tool.

What is Tivoli Performance Analyzer? At its core, it's intended to add predictive capability to Tivoli Monitoring so you can monitor resource consumption trends, anticipate future performance issues, and avoid or resolve problems more quickly. For example, you can use Tivoli Performance Analyzer to predict application bottlenecks and create alerts for potential service threats.

So, how does it work? Tivoli Performance Analyzer works by analyzing information in the Tivoli Data Warehouse (TDW) and providing an analysis of trends gathered from that data. The user may specify different analysis scenarios, based on what history data is being collected by the user, and the requirements of the user. You have various trending options, such as linear trending or arithmetic trending, and you can use it to calculate such things as time to hit a given threshold.

Tivoli Performance Analyzer works in unison with the TDW, and provides some interesting new analytic capabilities for ITM monitoring solutions (including OMEGAMON). Here's a manual number for more information: SC27-4004-00.

IBM Tivoli Monitoring (aka ITM) 6.2.3 version is now available

It's been a couple weeks now, but ITM 6.2.3 is now available. Don't get ITM 6.2.3 confused with ITM 6.2.2 fixpack 3. ITM 6.2.3 is a new version of the IBM Tivoli Monitoring infrastructure.

What's new in ITM 6.2.3? Quite a few things of interest. Probably the most intriguing and potentially the most powerful is a feature known as self-describing agents. Tired of having to worry so much about application CDs, CAT and ATR files, and looking at ITMSUPER output to identify mismatches in support levels for various agents? Self describing agents makes that process less of a chore. What does it mean? Agents will communicate their application support requirements to the infrastructure dynamically. The infrastructure becomes more flexible and dynamic as a result.

There's a lot more in ITM 6.2.3 that I will be covering over the next few days. To get started on ITM 6.2.3, here's a link:

https://www.ibm.com/developerworks/mydeveloperworks/blogs/0587adbc-8477-431f-8c68-9226adea11ed/entry/ibm_tivoli_monitoring_6_2_3_version_is_now_ega3?lang=en

Interested in capacity planning and System z? Check out this webcast.

If you are interested in System z and capacity planning, there is a webcast next week on September 15th on the topic. Attend "Optimize IT hardware and software expenditures with Tivoli capacity planning and IBM business analytics" to learn about Tivoli “what if” analysis, the Tivoli open architecture for gathering and analyzing data, and application performance modeling and rich graphical reporting.

The webcast will cover the following topics:

• Collecting historical log data on IT utilization to use in performance reporting and usage accounting
• Forecasting and modeling data relationships at the application level
• Using simulation to conduct “what if” analyses that help you avoid unnecessary costs
• Reading, normalizing and storing data for service level optimization
• Delivering business intelligence to be used for predictive analytics

The event is September 15th at 11 AM ET. To attend here is a link:

http://www-01.ibm.com/software/sw-events/teleconference/T805130X92184D07.html

Use ITCAM to identify mainframe as well as distributed bottlenecks

ITCAM (IT Composite Application Monitoring) provides a detailed and dynamic analysis of application transaction flow, and shows end to end application response time, along with aiding in the identification of transaction bottlenecks. What's powerful about ITCAM is it has the ability to 'stitch' together information from a variety of sources to determine the application flow, and show performance information. This end to end perspective could conceivably flow between mainframe and distributed, and encompass various middleware components in between.

The example screenshot shows a distributed client that connects to the mainframe via IMS Connect, and then invokes various IMS transactions. When you look at the example you can see that IMS Connect is the potential bottleneck, and the IMS Connect portion of the response time is highlighted in red on the bottom portion of the screen. This is a fairly straightforward example of how this function operates.

ITCAM is flexible, powerful, and allows you to see where your complex composite applications are spending their time.

Logs logs, where are my logs?

Tivoli distributed and OMEGAMON mainframe kicks out lots of logs and even more messages. These logs can be useful when it comes to problem analysis, and determining why a component may not be functioning as desired.

The logs that are available, and the format of the logs may vary by platform or component. If you want a listing of all the various available logs, here is a link to a handy little document that summarizes where the various logs reside:

https://www-304.ibm.com/support/docview.wss?mynp=OCSSZ8F3&mync=R&uid=swg21502429&myns=swgtiv

Information on z/OS 1.13 OMEGAMON currency maintenance

z/OS 1.13 is coming out, so the question is what, if any, maintenance may be required for OMEGAMON in a z/OS 1.13 environment? The good news is OMEGAMON currency support for z/OS 1.13 is provided for Version 410, 420 and later releases of OMEGAMON XE z/OS products.

Here's a link to a document that lists what, if any, OMEGAMON maintenance is required. Here's the link:

https://www-304.ibm.com/support/docview.wss?mynp=OCSSUSPS&mync=R&uid=swg21512105&myns=swgtiv

By the way, if you are interested in z/OS 1.12 maintenance, here's a link for that, as well:

https://www-304.ibm.com/support/docview.wss?uid=swg21429049

OMEGAMON for z/OS Management Suite V4.2.0 Announced

IBM has announced the OMEGAMON z/OS Management Suite V4.2.0. What exactly is the z/OS Management Suite? Think of it as an integrated offering that includes monitoring for the z/OS operating system, network monitoring, and expanded DASD storage monitoring, all within a single solution.

OMEGAMON XE for z/OS by itself does not really provide much in the way of network information. OMEGAMON XE for z/OS provides DASD, cache, and control unit information, but the available information is largely focused from the perspective of 3390 type devices, etc.

What OMEGAMON z/OS Management Suite V4.2 provides is a way to expand the reach of your core z/OS monitoring to include robust detailed network analysis and more detailed storage information within a single software offering. Technically you could purchase and install OMEGAMON XE for Mainframe Networks and OMEGAMON Storage, in addition to OMEGAMON z/OS. What OMEGAMON z/OS Management Suite V4.2 provides is a way to have all this integrated capability in a single solution.

For more information here is a link:

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=an&subtype=ca&htmlfid=897/ENUS211-325&appname=isource&language=enus#h2-abstrx

Interested in cloud?

The IT business has always been full of buzz words and phrases. "Cloud" is certainly one of the latest examples. If you want to learn more about cloud, IBM is putting a series of events that will go into quite a bit of detail of how to get started with highly scalable clouds. The event will look at such things as platform choice for consolidating new cloud workloads and applications, and containing the cost of administering and managing cloud.

If you want to learn more about cloud, the event will be happening at several locations

Fort Worth, Texas September 14, 2011
Markham, Ontario September 20, 2011
Waltham, Massachusetts October 4, 2011
New York, New York October 12, 2011
Chicago, Illinois October 18, 2011
San Francisco, California October 27, 2011
Washington D.C. November 1, 2011

If you are interested, here's a URL to sign up:

ibm.com/software/systemz/seminar/cloud

Data collection considerations for OMEGAMON Storage

There is an interesting technote that discusses collection overhead considerations when doing dataset level monitoring using OMEGAMON Storage. The symptoms may be high CPU in the KCNDLxxx address space, and you may see the following message:

KDFS023A CANDLE DATASET I/O KDFSMIBF ROUTINE IEXP FAILED 0000000C 00000000

What this indicates is that the dataset collection process is overwhelmed with data and is unable to catch up.

This goes back to posts I had made about a year ago relevant to optimal monitoring strategies. In general, the more data you request, the more it will cost to collect and display/manage that data. So consider carefully what you ask for, you may get more than you bargained for.

In this technote the author references the "OMEGAMON XE For Storage Tuning Guide".

http://publib.boulder.ibm.com/infocenter/tivihelp/v15r1/topic/com.ibm.omegamon.stor.doc/SC23-9704-01.pdf

The technote also suggests such things as collecting data every so often (example every 20 I/Os, SAMPCT=20), versus every I/O. Or, collect data based upon an exception criteria, such as a high MSR time for a device.

With so many active UCBs in many shops and potentially so many datasets, it makes sense to have a strategy for collection, versus just turning it on for everything.

Here's a link to the technote:

https://www-304.ibm.com/support/docview.wss?mynp=OCSS2JFP&mync=R&uid=swg21508611&myns=swgtiv

Presentations at Share

At the Share conference in Orlando I did four presentations as part of the proceedings. Here are my topics with links to the material. Each of these presentations contains at least some updates.

Here are the topics and the links to them. Feel free to download:


"Understanding The Interaction Of z/OS Workload Manager And DB2"

http://share.confex.com/share/117/webprogram/Session9395.html


"Monitoring: Become More Proactive Via Effective Historical Analysis"

http://share.confex.com/share/117/webprogram/Session9633.html


"Optimal Alert Management Strategies for System z and Beyond"

http://share.confex.com/share/117/webprogram/Session9776.html


"IMS Performance - Getting The Most Out Of Your Monitoring Technology: Isolating And Solving Common Issues"

http://share.confex.com/share/117/webprogram/Session9808.html

Share presentation on NetView Canzlog

Canzlog is a consolidated log management function that was added as part of NetView V6.1. Canzlog has been popular with quite a few customers as a method of managing multiple log streams in a consolidated manner. It's a nice facility, and worth taking a look at.

Dale Carter of Bank of America) and Tom Howe of IBM did a presentation on "Canzlog in NetView for z/OS 6.1: User Experience".

Here's a link to the presentation:

http://share.confex.com/share/117/webprogram/Session9914.html

Share presentation on the DB2 Catalog

Willie Favero of IBM did an interesting presentation at Share in Orlando on what's new and changed in the DB2 Catalog, "The DB2 for z/OS Catalog Gets a Makeover ". DB2 V10 certainly introduced quite a few changes, and this topic goes through what has happened with the Catalog, what was changed, what was added, and what it means for the DB2 sysprog.

Here's a link for the presentation:

http://share.confex.com/share/117/webprogram/Session9539.html