OMEGAMON Messaging command interface

As in the prior example of OMEGAMON IMS V5.1, the new release of OMEGAMON Messaging provides an easy to use command facility integrated into the enhanced 3270 user interface.  Here's an example:

In the above example we see how you can issue commands, such as MQ channel commands seen here. 

OMEGAMON IMS V5.1 expands command support

In prior releases of OMEGAMON IMS the Type 2 IMS commands were not supported.  Now with OMEGAMON IMS V5.1 the tool finally provides support for Type 2 IMS commands.  Here's an example:

Powerful new 3270 displays with OMEGAMON Storage V5.1

OMEGAMON Storage V5.1 includes support for the new enhanced 3270 user interface.  The implementation of enhanced UI for OMEGAMON Storage is robust, and includes quite a few powerful new displays. 

One example of a very useful display that has been translated from the Tivoli Portal to enhanced 3270 UI is the Dataset Atttributes screen.  The Dataset Attributes Database is a feature of OMEGAMON Storage that helps you manage data sets. The database maintains attributes related to data set space, DCB, and catalog issues.  The database feature collects data regarding all data sets on all volumes in the z/OS environment (except for volumes that you have excluded) so that you can do analysis from a single point of control.  You can accomplish such tasks as identifying exception conditions regarding data sets throughout the environment, seeing an installation-wide view of data set space utilization, exceptional conditions, and summary statistics, and identifying resources that require attention, such as data sets that have excessive unused space, extents, or CA/CI splits.
  Here is an example of the Dataset Attributes screen in the enhanced 3270 interface.  Note how the display aggregates quite a bit of information together.  

OMEGAMON Messaging V7.1 provides many powerful new displays

It's exciting now that OMEGAMON Messaging V7.1 provides a robust 3270 interface.  With V7.1 there are many powerful new displays inclduing the Health overview panel. 

The Health Overview panel gives the overall health values for queue manager, queues and channels, and the worst health status on the display will be sorted to the top of list.  Among the various values used in determining health include Queue Manager status (Is it active?  Are the channel initiator and command servers active? Are there any connections?),  Queue performance indicators (Are there queues with high depth? Do XMIT queues have messages? Are there messages on the DLQ? Are there get inhibited or put inhibited queues?), and Channel status (Is the percent (of max) current channels or active channels too high?  Are there current channels not in running state? Are there in-doubt channels?)

There's a lot of information on the display, and it's a good place to start for problem analysis.  Here's an example of the screen:
 

OMEGAMON IMS V5.1 adds support for the enhanced 3270 user interface

With the release of OMEGAMON IMS V5.1, we now have another OMEGAMON in the suite that has support for the enhanced 3270 user interface.  The big benefits of the enhanced 3270 user are consistency and integration.  With the enhanced user interface you can logon to one UI and access any of the OMEGAMONs from a central point.

Here's an example of what the OMEGAMON IMS emhanced user interface looks like. 

Just like the DB2 monitor, you get a default panel that can show both IMSplex level information, and the individual IMS subsystems.  You can use the "/" navigation to drill in for detail, in the similar manner to how the other OMEGAMONs operate.

New versions of OMEGAMON IMS, Storage and Messaging announced

Yesterday IBM announced new versions of OMEGAMON IMS, OMEGAMON Storage, and OMEGAMON for Messaging.  These new versions continue the process that was begun earlier this year of expanding the integration and consolidation of the OMEGAMON 3270 interface capablities.  And, as always, and new version of OMEGAMON is always an exciting event.

IBM Tivoli OMEGAMON XE for IMS on z/OS V5.1, IBM Tivoli OMEGAMON XE for Storage on z/OS V5.1, and IBM Tivoli OMEGAMON XE for Messaging on z/OS V7.1 use a new architecture that simplifies installation (PARMGEN) and includes integrated enhanced 3270 monitors (the new enhanced 3270 ui). These products integrate related data to help subject matter experts  solve problems encountered on a day-to-day basis.

 

New capabilities of IBM Tivoli OMEGAMON XE for IMS on z/OS V5.1, IBM Tivoli OMEGAMON XE for Storage on z/OS V5.1, and IBM Tivoli OMEGAMON XE for  Messaging on z/OS V7.1 are designed to:

• Improve problem resolution efficiency by requiring fewer steps to isolate root  cause performance impact in real time, and, therefore, providing greater availability. 
• Improve visibility, control, and automation with a new more comprehensive 3270-based user interface capable of viewing the entire enterprise-wide environment from a single 3270 screen. 
• Reduce the time required for installation, configuration, and maintenance by leveraging enhanced IBM Tivoli Monitoring functions and a new PARMGEN configuration tool.
• Provide an enterprise end-to-end view of performance monitoring and availability on a common technology for both IBM System z® and open systems.

I will be blogging extensively on the capabilities of the new versions in the next few days.

For more information here is a link:

http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?infotype=an&subtype=ca&appname=gpateam&supplier=897&letternum=ENUS212-308

Using Tivoli Data Warehouse for trending analysis

Because it is snapshot type data, Tivoli Data Warehouse (TDW) data works well for trending and analysis.  TDW data is not necessarily equivalent to SMF data, which tends to be more interval or event driven in nature.  TDW is not equivalent to CICS SMF 110's or DB2 Accountig trace data.  But, if you want to be able to do things like trend resource usage (such as CPU, DASD, or memory) over time,  TDW can work well in that context.   Also, when doing trending analysis, the TDW summarization process can add some interesting information to the data.

Here's an example of what I'm talking about.  In this example, I'm trending z/OS CPU over time using CPU TDW snapshot data summarized by hour.  Note that in addition to average CPU usage you also get a Min and Max CPU usage for the same time interval, and here I show how easy it is to chart that information on a single graph.

Learn about what's new in zEnterprise

Today there will be a virtual event to introduce the next generation of zEnterprise, the world’s fastest, most scalable and secure enterprise system with the ability to integrate resources for operational analytics, trusted resilience, and efficiency at an enterprise scale. 

The virtual doors open at 10:45  Eastern Time.  Here's a link to attend:
http://bit.ly/NRtwF6

An example of analyzing issues with Tivoli Data Warehouse

I had an interesting project this week to get Tivoli Data Warehouse (TDW) operational for a customer.  With the assistance of an IBM colleague (thank you Andrew), we got TDW fully configured and (seemingly) connected.  The Warehouse proxy and Summarization/Pruning agents were started.  The Warehouse proxy log showed that it was connecting to the TEMS.  The Warehouse Proxy configuration workspace showed that everything was green and connected.  Real time monitoring was working great.  We could get 24 hours worth of history, but nothing more.  So why were we still not seeing any data being written to the TDW?

We looked at the Warehouse proxy log on the TDW box.  There were no obvious error messages.  But there also were none of the eyecatchers we noted in earlier posts (such as that TDW database objects were being created).  So if the Warehouse Proxy had no errors, what was the issue?

In this shop, the agents were mainframe TEMAs.  The next step was to look at the RKLVLOGs of the agents running on z/OS.  When I looked through the logs I noticed messages, such as the following, that would occur every hour.  Since we had set the warehouse interval for one hour for this collection, this seemed like more than just a coincidence.

+4EBF9D4E.0002 ERROR MESSAGE: "RPC Error"

(4EBF9D4E.0003-13B4:khdxdacl.cpp,577,"routeExportRequest") Export for object  (table POOLS appl KIP) failed in createRouteRequest, Status = 8.
(4EBF9D63.0000-13B4:kdcc1sr.c,460,"rpc__sar") Connection failure: "ip.pipe:#xx.xxx.x.xxx:63358", 1C010001:1DE00045, 21, 100(3), FFFF/4119,

The conclusion was that this message seemed to indicate that the TEMA was attempting to connect to the warehouse infrastructure to send the data to the TDW, but was getting a connection error.  And this error was happening every hour. 

The problem?  In this case, there was an internal firewall that was blocking off the port we needed (in this example 63358 - see above).  Although real time monitoring was working, and the TDW configuration panel showed green, we still needed this port to be able to send data to the TDW.  Once the port was opened, the TDW worked perfectly.

The moral of the story?  When analyzing these types of issues it is important to look at things from multiple perspectives, look at all the logs, and don't forget about things like firewalls and other security challenges.

Upcoming OMEGAMON webcasts in September

There are a couple webcast events covering OMEGAMON in September.

"Increase System z storage visibility with OMEGAMON Monitoring"  will cover what's new and exciting in the area of OMEGAMON Storage and System z.  The redesigned OMEGAMON XE for Storage V5.1 allows subject matter experts to resolve storage issues using fewer screens and keystrokes. OMEGAMON along with IBM’s Storage Management Suite can provide end-to-end visibility, control and automation for good storage management.  The speaker is Kevin Hosozawa, Product Manager, OMEGAMON for Storage, IBM Software Group . 

The webcast is September 13, 2012, 11 a.m., EDT.  Here's  a link to sign up:
http://ibm.co/N30pbq

Later in September there will be webcast on OMEGAMON IMS. 

"Improve IMS monitoring with new OMEGAMON V5."  Get the details about what’s new in OMEGAMON for IMS V5.1, including easier region navigation, reduced MIPS usage and the capability to issue IMS commands from OMEGAMON. Learn how it reduces potential delays or outages by reporting coupling facility structure statistics and other critical IMS attributes

The speaker is Mike Goodman, OMEGAMON for IMS Product Manager, IBM Software Group. 

The webcast is September 27, 2012, 11 a.m., EDT.  Here's a link to sign up:
http://ibm.co/NveTXc

More OMEGAMON V5.1 test drive events coming your way

Want a chance to take the new OMEGAMON V5.1 for a test drive?  You can try it out for yourself on a live z/OS environment.  There are more test drive events coming your way.  Here's a list:

Chicago  -  Sept 11th
IBM Chicago TEC
71 South Wacker Drive, 6th Floor  

Springfield, IL   -  Sept 25th
IBM Springfield TEC
3201 West White Oaks Drive, Suite 204

Minneapolis,  MN   -   Oct  18th
IBM Minneapolis TEC
650 3rd Avenue South

Omaha, NE   -   Nov  7th
IBM Omaha Office
1111 N. 102nd Court, Suite 231

REGISTER Now! Please send an email to IBM Representative Cliff Koch at

cjkoch@us.ibm.com or call (314)-409-2859. 

My presentations at Share

At the Share confernce last week in Anaheim I did a couple presentations. 

One topic was somewhat in keeping with one of the themes of the conference, Big Data.  In "Predictive Analytics and IT Service Management"  I cover the concepts of predictive analysis, consider common systems management challenges, and look at how predictive analysis concepts may be applied to address IT Service Management needs.  Here's a link:
https://share.confex.com/share/119/webprogram/Session11479.html


Another session that I did covered the topic of   "Understanding The Impact Of The Network On z/OS Performance".  This was an updated verison of a presentation I did at Share in Atlanta this past spring.  In the presentation I discuss how to create a more integrated, end to end monitoring strategy that includes relevant network information.  I cover understanding the impact of the network on z/OS performance, how to analyze typical z/OS workloads and network performance, analyzing performance using available tools (such as OMEGAMON), and recommended approaches and management strategies.  Here's a link:
https://share.confex.com/share/119/webprogram/Session11900.html

Some very interesting Tivoli monitoring sessions

There were quite a few good Tivoli related monitoring and management sessions at Share last week in Anaheim.  Some of the best ones were done by Mike Bonett of IBM. 

"Using NetView for z/OS for Enterprise-Wide Event Management and Automation" was a very interesting session that covered numerous ways that you exploit and use NetView in your enterprise.  Here's a link:
https://share.confex.com/share/119/webprogram/Session11905.html

"The IBM Tivoli Monitoring Infrastructure on System z and zEnterprise" is one of the most comprehensive and concise reviews I've seen of the Tivoli infrastructure.  If you want a good explanation of all the Tivoil terminology, this is the best one I've seen.  Here's a link:
https://share.confex.com/share/119/webprogram/Session11524.html

"Forecasting Performance Metrics using the IBM Tivoli Performance Analyzer"  is a good overview of the capabilities of this very interesting feature of the Tivoli Portal. Here's a link:
https://share.confex.com/share/119/webprogram/Session11523.html

"Getting Started with the Unified Resource Manager (zManager) APIs for zEnterprise Monitoring and Discovery"  looks at the support for zManager now available in the Tivoli Portal.  If you are not familiar with this support, it is very much worth looking at this presentation.  Here's a link:
https://share.confex.com/share/119/webprogram/Session11630.html

New Share presentation on OMEGAMON optimization

It's Share conference week in Anaheim, California (home of Mickey and Minnie).  I'll be providing some links on interested presentations I attended over the course of the week.

One presentation I attended yesterday was done by the always interesting and informative Don Zeunert (aka Dr Z).  Don did a presentation on "Tuning Tips To Lower System z Costs with OMEGAMON Monitoring".   This was a very informative presentation that covered quite a few tuning tips for OMEGAMON.

Here's a link to download the presentation:
https://share.confex.com/share/119/webprogram/Session11791.html

Monitoring Summarization and Pruning activity

We've looked at specifying history collection, and as part of that process defining summarization and pruning.  Summarization and pruning should happen on a regular basis as defined in the setup and configuration for the summarization and pruning agent.  Here's an example of where you would specify the time of day that summarization and pruing takes place.

Once the summarization and pruning is set up, you can then monitor the summarization and pruning activity on an ongoing basis using a product provided workspace (see example below).  The statistics workspace under summarization and pruing shows interesting information, such as tables being sumamrized, and if there were any tables that saw errors during summarization/pruing.

It's recommended that if you are collecting TDW history, and are using the summarization and pruning facility, that you check this workspace periodically to make sure you do not have any issues.

Upcoming webcast on the new OMEGAMON DB2 V5.11

There is a new webcast event upcoming on the new release of OMEGAMON DB2,  OMEGAMON V5.11.  This webcast will cover the capabilities of the tool including:  advanced problem determination using focused scenarios designed by customers, combining OMEGAMON DB2 information with CICS and z/OS in a new enhanced 3270 workspace, improved efficiency with fewer screen interactions to find root cause performance impact in real time,  and additional end-to-end response time measurement capability.

The speaker is Steve Fafard, Product Manager, OMEGAMON for DB2.  The event will be August 9th, 2012, 11 a.m., EDT. 

The price is right.  The webcast is a free event.  Here's a link to sign up:

http://ibm.co/Nt3HIq

Tivoli Data Warehouse - Summarization And Pruning considerations

We' ve covered quite a bit on the Tivoli Data Warehouse (TDW) and what happens when you initialize history collection for a given table. 

Once you have had history collecting for an interval of time, you may want to look at using summarization and pruning to control the quantity of data being retained, and to also take advantage of the ability to use summarized data for trending purposes. 

Below is an example of the configuration options for summarization and pruning.  As part of the earlier exercise we enabled TDW history collection for Common Storage.  Now we are going to enable summarization and pruning for this data. 

In the above example, we are enabling pruning on detailed data by specifying that we will keep 30 days of detail.  Detail beyond that point will be removed by the Summarization and pruning agent process.   Also, the detail data will be summarized, both on an hourly and on a daily basis.  We will be keeping 90 days of hourly data, and one year of daily data. 

By specifying summarization and pruning, we can control the quantity of data being retained within the TDW.  This can be very important, especially for high volume tables of data or tables being collected for a large number of managed systems. 

Another consideration to be aware of is that when you specify summarization, the summarization agent will create another set of tables.  In the above example, the agent will have DB2 to create a table for hourly data and another table to hold daily data. 

This is an area of confusion for some uers. I have encountered some shops who thought they were going to save space right away by turning on summarization.  However, when they enabled the function, the TDW database ended up requiring more space.  Why?  Because the summarization agent had to create the additional tables to store the summarized data.

Some things to be aware of when installing OMEGAMON DB2 V5.11

In addition to introducing support for the new enhanced 3270 user interface, there are some infrastructure considerations you should be aware of with OMEGAMON DB2 V5.11.  OMEGAMON DB2 V5.11, in a smiliar fashion to OMEGAMON z/OS and CICS V5.1, introduces support for self describing agents (SDA).  SDA support eliminates the need for the application support media when defining the agent to the Tivoli monitoring infrastructure. 

Another thing to be aware of is that the OMEGAMON DB2 agent (i.e. the TEMA) has been re-architected in V5.11.  Because of this, there are some additional steps you may need to be aware of when deploying OMEGAMON DB2 V5.11.  It is important that you review the the documentation in the V5.11 PSP bucket, and look at relevant readme information.  Here's a link to the PSP information:

http://www-304.ibm.com/support/docview.wss?uid=isg1_5655W37_HKDB511

If you overlook this step, you may be missing some relevant information, in particular data sharing related information.

OMEGAMON DB2 V5.11 features the new enhanced 3270 user interface

OMEGAMON DB2 V5.11 was recently released.  There were several interesting enhancements and additions to the tool.  One of the more interesting new features is support for the new enhanced 3270 user interface.  With the addition of OMEGAMON DB2 to the fold, this means that three core OMEGAMONs (z/OS, CICS, DB2) now support the enhanced 3270 ui. 

One of the nice aspects of the enhanced ui is the ability to do your monitoring using an integrated interface that enables you to monitor from a single point of view.  Here is an example of the default KOBSTART panel, now showing DB2 information on the display.

From this central screen, as the example shows, you can easily navigate to z/OS, CICS or DB2. 

From the main panel (KOBSTART), you can then drill down into DB2 specific detailed views, and drill in on commonly used displays, such as DB2 thread views.  Here is an example of how you would drill in on DB2 threads. 

From this you you can drill in on detail for each thread executing on the system, and do analysis of what the threads are doing and where they are spedning their time.

More TDW under the covers

Now that it appears data on z/OS Common Storage utilization is flowing to the TDW, be aware that there is information readily available to track on an ongoing basis the amount of data flowing to the TDW, and to show indications of potential issues with TDW history collection.

In the Tivoli Portal there are workspaces that show the status of the Warehouse Proxy, and also track history row processing counts and history collection error counts.  The example below shows the Warehouse Proxy statistics workspace. 

The rows statistics chart on the workspace shows the number of rows sent to the Warehosue Proxy and the number of rows inserted by the Warehouse Proxy to the TDW database.  Note there is a marked difference in this example between the number of rows sent and the number of rows inserted.  This indicates a potential issue where data is being sent to the Warehouse Proxy, but due to some issue the data is not making it into the TDW database.  Note also in the Failures/Disconnections chart there is a count showing failures with the TDW.

Now that we see there is a potential issue with data making its way into the TDW database, how do you get more information on what the problem may be?  First, you can look at the log for the Warehouse Proxy process.  Also, you can look at the Warehouselog table in the TDW database (this is assuming you have it enabled - be aware of an earlier blog post I made on this not being on by default in ITM 6.23).

I the following example I show information selected from the Warehouselog table in the TDW.  The Warehouselog table shows row statistics and error message information (if applicable) for each attempted insert to the TDW. 

In the example we see that the Common Storage table has 16 rows received and 16 rows inserted.  However, the Address Space CPU Utilization table seems to have an issue with 1 row received and one row skipped, and also an error message.  Analysis of the error message would give some indication as to why the TDW is having an issue.

As you can see there is quite a bit of information available to track the status and activity of TDW, and it is worth checking this information on an ongoing basis to ensure history collection is proceeding normally.