About me

As always, Thanks for visiting.
Vikram at Louis Lake, Alberta, Canada

This Is My Story:

I happen to travel in North America from my work. I took time off from the work and visited Lake Louise which is around 200 km from Calgary, Alberta, Canada. It is a beautiful lake. The water is so crystal clear that I wanted to take bath but it was freezing cold. The glacier at the end of Lake is so beautiful that picture does not tell its serenity. One has to be there to actually feel it.

  • Life spent so far 70%
  • Retirement destination 22%
  • Satisfaction in Life 90%

Super Computer of Karma

I have not yet understood the Laws of Karma. Trying to gain understanding of it. “Karmo Ki Gati Ko Koi Nahi Jaan Sakta”.

Karmas that bind us

We all know about bad Karmas. But, seldom we think that good Karmas are also binding. It is as bad as Bad Karmas. When I heard about this, I was not able to grasp it.

Naiskarma

Naiskarma Siddhi delivers one from this world. I am trying to learn how to practice Naiskarma – It is extremely difficult.

When we get praise

When someone praises me, I feel happy. It is a clear indication that Naiskarma Siddhi is far away from me. Did you hear PTSD? The soldiers will not suffer PTSD, if the Karams they did were for the Nation and not for themselves.

Recent Blog Posts

Learn – What is happening, tips and techniques in DB2

DB2 pureScale instance creation hangs after GPFS

While creating DB2 pureScale instance, it appears that the node becomes unresponsive under RHEL 7.2. If you reboot the node and look at the /var/log/messages, you may notice these several messages:

kernel:BUG: soft lockup - CPU#1 stuck for 23s! [mmfsd:3280]

The mmfsd is GPFS (aka IBM Spectrum Scale) file system daemon and somehow it looks that this is the cause of the this CPU soft look.

The other symptom of soft CPU lockup is the high queue seen in the vmstat output. Please look at the first column ‘r’ under procs. The value of ‘r’ would be very high in this case.

It looks that Supervisor Mode Access Prevention (SMAP) feature of Intel Xeon V4 processor (Broadwell) and Linux kernel 3.7 or later causes mmfsd to not have access to some memory space. The SMAP feature in Intel Broadwell family of CPU (including Intel Core i7 6820 HQ) has the protection enabled which disallows access from kernel-space memory to user-space memory, a feature aimed at making it harder to exploit software bugs. Now, GPFS is a kernel level access and this feature is disallowing GPFS access of kernel-space memory with a result that soft lockup of CPU occurs and that leads to system appearing hung-up.

This causes the node to appear to hang but it is actually soft CPU lockup issue as seen with the above command. The soft CPU lockup also causes the high queue – with a result that the system becomes non-responsive.

The RHEL 7.2 kernel has the support for SMAP feature by default. If your cpu has this feature or not, you can check the output from cat /proc/cpuinfo | grep smap and if you see smap in the flags section, you have this Supervisor Mode Access Prevention (SMAP) feature enabled.

GPFS has fixed this issue in v4.2.1.1 but the version that comes with DB2 11.1 FP 1 is v4.1.1.9. If you are using later version of DB2, you can find out the version of GPFS that will be installed by looking at file spec in the folder <db2softwaredir>/server_t/db2 directory.

You can disable smap feature in Linux kernel by adding kernel parameter nosamp as shown below (for RHEL 7.2).

  1. Locate grub.cfg  –> It can be in different places depending upon legacy or EFI boot. In my case, this was in /boot/grub2/grub.cfg
  2. Edit grub.cfg and find line associated with the system image such as line containing vmlinuz and add “nosamp” parameter at the end.

Alternatively, you can use edit /etc/default/grub and add nosamp parameter at the end of line containing GRUB_CMDLINE_LINUX and then run grub2-mkconfig -o /boot/grub2/grub.cfg and reboot the system.

You can also run the grubby command to add this kernel parameter.

# grubby --update-kernel=ALL --args=nosmap

 

 

 

 

DB2 pureScale Health Check

More and more customers are now asking for some way to check the health of the DB2 pureScale system. Let’s just focus on few early diagnostics before we jump to the deep exploration at the SQL statement level. This is just akin going to primary health care physician to catch symptoms early.

The DB2 pureScale heart is Cluster Caching Facility (CF) and if this slows down, it will have effect on everything.

If we just focus early on the symptoms exhibited by CF, it is equivalent to finding the cure for the disease.

In my experience working with hundreds customers all over North America, almost 99.9% of the folks jump right to the SQL tuning. Their is nothing wrong in it but I consider it to be treating the symptom and than the disease. A particular SQL could as well be the real issue and I do not deny that but it is last on my to-do list to tune a system. Let’s get on track to check the health of pureScale system.

I am just copying few SQLs here that I have used in many situations to derive some conclusions on the health of the CF.

CF Health Check

If CF becomes the bottleneck, everything slows down. This is on my priority list to check its health. As per Steve Rees, a performance expert Guru, I have settled on the following 3 SQLs to examine CF health.

SLS and WARM Rate

SELECT CF_CMD_NAME, 
       DEC(CAST(SUM(TOTAL_CF_WAIT_TIME_MICRO) AS FLOAT) / 
       CAST(SUM(TOTAL_CF_REQUESTS) AS FLOAT),5,0) AS CF_CMD_RATE 
FROM   TABLE (SYSPROC.MON_GET_CF_WAIT_TIME(-2)) 
WHERE  ID = (select ID from sysibmadm.db2_cf where STATE = 'PRIMARY') 
AND    CF_CMD_NAME IN ('SetLockState', 'WriteAndRegisterMultiple') 
GROUP BY CF_CMD_NAME 
HAVING SUM(TOTAL_CF_WAIT_TIME_MICRO) > 0 
AND    SUM(TOTAL_CF_REQUESTS) > 0
;

In above SQL, we are just taking 2 CF commands SetLockState and WriteAndRegisterMultiple and finding out the average rate in micro seconds. The SetLockState (SLS) is a small packet and it should bea quick turnaround and gives a baseline for all other messages. The other message WriteAndRegisterMultiple (WARM) is a larger packet and will include XI time. High WARM times can indicate both heavier network activity and more page invalidation, and measuring WARM times periodically can reveal when statement times are being adversely affected. (Courtesy: Toby Haynes of IBM Lab in Toronto).

Overall CF Wait Rate

The following SQL is the average rate over all CF commands. This is another overall indicator but it has to be compared against a baseline that you collected in a healthy system.

-- AVG_CF_WT_MICRO is an indication if CF is the bottleneck
-- This is an average over all CF calls
-- Best way to judge good or bad number - Look for a change
-- from what is normal for your system

SELECT INT(SUM(RECLAIM_WAIT_TIME)) RECLAIMTIME_MILLI,
 INT(SUM(CF_WAITS)) AS NUM_CF_WAITS,
 INT(SUM(CF_WAIT_TIME)) CF_WAIT_MILLI, 
 CAST ((CASE WHEN SUM(CF_WAITS) = 0 THEN NULL 
 ELSE (INT(1000 * DEC(SUM(CAST(CF_WAIT_TIME AS FLOAT))/
 SUM(CAST(CF_WAITS AS FLOAT)),5,4)))
 END) AS DECIMAL(10,4)) AVG_CF_WT_MICRO 
FROM TABLE(SYSPROC.MON_GET_WORKLOAD('',-2)) AS t
;

However, the output from above two SQLs should be seen along with the page negotiation rate. The page negotiation refers to a page getting tossed around due to request made by another member. If the page negotiation rate is much higher, it will lead to performance degradation as more and more time is spent in shuffling / tossing around the page between members through CF. Use the following SQL to determine this rate.

Page Negotiation Rate

-- For example: If Member 'A' acquires a page P and modifies a row on it.
-- 'A' holds an exclusive page lock on page until 'A' commits
-- Member 'B' wants to modify a different row on the same page.
-- 'B' does not have to wait until 'A' commits
-- CF will negotiate the page back from 'A' on 'B's behalf.
-- Provides better concurrency - Good but excessive can cause
-- contention, low CPU usage, reduced throughput.

-- Recoomendations to reduce excessive page reclaim
-- What is excessive is debatable and sort it in desc order 
-- for all tables and pick top 5 tables
-- If excessive page reclaim then do these
-- Consider smaller page size
-- For small HOT tables with frequent updates, increase PCTFREE
-- PCTFREE will spread rows over more pages
-- Side effect - More space consumption (but this is small table anyway)
 
SELECT MEMBER, 
       SUBSTR(TABSCHEMA,1,12) AS SCHEMA, 
       SUBSTR(TABNAME,1,16) AS NAME,
       SUBSTR(OBJTYPE,1,16) AS TYPE,
      (PAGE_RECLAIMS_X + PAGE_RECLAIMS_S) AS PAGE_RECLAIMS, 
       RECLAIM_WAIT_TIME
FROM TABLE( MON_GET_PAGE_ACCESS_INFO(NULL,NULL, NULL) ) AS WAITMETRICS
ORDER BY SCHEMA, NAME
;

The output from the above 3 SQLs can be collected at 15 minute interval plot the time series graph and observe the trend. The increase in SLS and WARM rate by a factor of 2 to 3 with reference to a baseline will indicate that CF network is becoming the bottleneck.

Adding more adapters at the CF will help resolve this bottleneck.

Group Buffer Pool Full Condition

Use the following SQL to determine group buffer pool full condition. This condition will cause a stall condition in which synchronous I/O will run to flush the pages through members to make room in the Group Buffer Pool. This will lead to a burst of the I/O activity. Increasing the CF memory may delay the burst activity.

-- GBP Full Condition
-- Good value: < 5 per 10000 transactions
-- If higher, GBP is small
-- The castout engines might not be keeping up
-- SOFTMAX is set too high

WITH GBPC AS ( 
 SELECT 10000.0 * SUM(GBP.NUM_GBP_FULL) / 
        SUM(COMMIT_SQL_STMTS) AS GBP_FULL_CONDITION 
 FROM TABLE(MON_GET_GROUP_BUFFERPOOL(-2)) as GBP, SYSIBMADM.SNAPDB
) SELECT CASE WHEN GBP_FULL_CONDITION < 5.0 THEN 'GOOD VALUE' 
 ELSE 'GBP FULL CONDITION' END AS RESULT,
 CASE WHEN GBP_FULL_CONDITION < 5.0 THEN 'NO GBP FULL CONDITION' 
 ELSE 'INCREASE CF_GBP_SZ OR DECREASE SOFTMAX OR INCREASE NUM_IOSERVERS' 
 END AS RECOMMENDATION
 FROM GBPC 
;

CF Swap Usage

We may not be able to avoid few hundred pages of swap but excessive swapping will adversely affect the performance of the system as a whole. Keep an eye of the CF swap usage. If you are not dipping into swap space, you have kept one problem at bay.

WITH TYPES AS (SELECT NAME FROM SYSIBMADM.ENV_CF_SYS_RESOURCES 
               GROUP BY NAME)
SELECT A.ID, 
 SUBSTR(MIN(DECODE(T.NAME, 'HOST_NAME', A.VALUE)),1,36) HOST_NAME,
 INT(MIN(DECODE(T.NAME, 'MEMORY_TOTAL', A.VALUE)) -
 MIN(DECODE(T.NAME, 'MEMORY_FREE', A.VALUE))) MEMORY_IN_USE ,
 INT(MIN(DECODE(T.NAME, 'MEMORY_SWAP_TOTAL', A.VALUE)) -
 MIN(DECODE(T.NAME, 'MEMORY_SWAP_FREE', A.VALUE))) SWAP_IN_USE
FROM SYSIBMADM.ENV_CF_SYS_RESOURCES A, TYPES T
WHERE A.NAME = T.NAME
GROUP BY A.ID
;

Member Swap Usage

WITH TYPES AS (SELECT NAME FROM SYSIBMADM.ENV_SYS_RESOURCES 
 GROUP BY NAME)
SELECT SUBSTR(MIN(DECODE(T.NAME, 'HOST_NAME', A.VALUE)),1,36) HOST_NAME,
 INT(MIN(DECODE(T.NAME, 'MEMORY_TOTAL', A.VALUE)) -
 MIN(DECODE(T.NAME, 'MEMORY_FREE', A.VALUE))) MEMORY_IN_USE ,
 INT(MIN(DECODE(T.NAME, 'MEMORY_SWAP_TOTAL', A.VALUE)) -
 MIN(DECODE(T.NAME, 'MEMORY_SWAP_FREE', A.VALUE))) SWAP_IN_USE
FROM SYSIBMADM.ENV_SYS_RESOURCES A, TYPES T
WHERE A.NAME = T.NAME
;

SAN System Performance

The SAN system performance measurement is not accurately possible using DB2 but we can measure the rate at which pages are flushed to the disk from the buffer pool and from the DB2 logs. This is a crude indication of the performance of the I/O subsystem.

The castout is the term used on z/OS DB2 sysplex, which is similar to page cleaning in DB2 LUW. Calculate number of writes per transaction and calculate time per write. If this time is increasing over 10 ms, it is an indication that I/O subsystem is not able to keep-up.

-- Calculate number of writes / transactions (CASTOUTS_PER_TRANSACTION)
-- Calculate time per write (CASTOUT_TIME_MILLI_PER_TRANSACTION)
-- 
-- Bursty write activity may be a sign of SOFTMAX being high
-- Long Write Time (CASTOUT_TIME_MILLI_PER_TRANSACTION) is an
-- indication that I/O subsystem may not be able to keep up
-- 
-- Castout activity is influenced by
-- SOFTMAX - Lower value means faster group crash receovery but more
-- aggressive cleaning
-- Consider setting SOFTMAX higher than equivalent EE system

SELECT CASE WHEN SUM(W.TOTAL_APP_COMMITS) < 100 
       THEN NULL ELSE
 CAST( FLOAT(SUM(B.POOL_DATA_WRITES+B.POOL_INDEX_WRITES))
 / SUM(W.TOTAL_APP_COMMITS) AS DECIMAL(6,1)) END 
 AS "CASTOUTS_PER_TRANSACTION",
 CASE WHEN SUM(B.POOL_DATA_WRITES+B.POOL_INDEX_WRITES) < 1000 
    THEN NULL ELSE
 CAST( FLOAT(SUM(B.POOL_WRITE_TIME))
 / SUM(B.POOL_DATA_WRITES+B.POOL_INDEX_WRITES) AS DECIMAL(5,1)) END 
 AS "CASTOUT_TIME_MILLI_PER_TRANSACTION"
FROM TABLE(MON_GET_WORKLOAD(NULL,NULL)) AS W, 
TABLE(MON_GET_BUFFERPOOL(NULL,NULL)) AS B
;

If the value of CASTOUTS_PER_TRANSACTION is less than 10 and CASTOUT_TIME_MILLI_PER_TRANSACTION is less than 1 ms, the I/O subsystem is keeping well and indicates that it is a modern flash based SAN storage.

Size of Database

select member, 
 decimal(sum(double(tbsp_used_pages) * tbsp_page_size ) / 
 1024 / 1024, 10, 2 ) as db_mb_used 
from table( mon_get_tablespace(null, -2)) 
group by member
;

Detect Log Full Condition

SELECT MEMBER,
       TOTAL_LOG_AVAILABLE / 1048576 AS LOG_AVAILABLE_MB,
       TOTAL_LOG_USED / 1048576 AS LOG_USED_MB,
       CAST (((CASE WHEN (TOTAL_LOG_AVAILABLE + TOTAL_LOG_USED) = 0
            OR (TOTAL_LOG_AVAILABLE + TOTAL_LOG_USED) IS NULL
            OR TOTAL_LOG_AVAILABLE = -1 THEN NULL
            ELSE ((CAST ((TOTAL_LOG_USED) AS DOUBLE) / CAST (
               (TOTAL_LOG_AVAILABLE + TOTAL_LOG_USED) AS DOUBLE))) * 100
       END)) AS DECIMAL (5,2)) AS USED_PCT,
       APPLID_HOLDING_OLDEST_XACT
FROM TABLE (MON_GET_TRANSACTION_LOG(-2))
ORDER BY USED_PCT DESC;
From above, find out the % of log space used and most importantly the application id that is holding a log file active, which may lead to a log full condition.

Detect Log Hog / Drag Application

select a.application_handle,
       a.workload_occurrence_state as status,
       substr(a.session_auth_id,1, 10) as authid,
       substr(c.application_name, 1, 10) as applname,
       int(a.uow_log_space_used/1024/1024) as logusedM,
       timestampdiff(4, char(current timestamp
          - b.agent_state_last_update_time)) as idleformin
from table(mon_get_unit_of_work(NULL,-2)) as a,
     table(mon_get_agent(NULL,NULL,NULL,-2)) as b,
     table(mon_get_connection(NULL, -2)) as c
where a.application_handle = b.application_handle
  and a.coord_member = b.member
  and a.coord_member = a.member
  and b.agent_type = 'COORDINATOR'
  and a.uow_stop_time is null
  and a.application_handle = c.application_handle
  and a.coord_member = c.member
  and b.event_type = 'WAIT'
  and b.event_object = 'REQUEST'
  and b.event_state = 'IDLE';
Use above SQL to find out the application that has been idle or waiting for a long time and this can hold a log file active.

How to detect log full condition

I have seen working with hundreds of customers a chronic LOG FULL condition and the shortcut people use is to increase the LOGPRIMARY and LOGSECOND. This solution works well for a single unit of work that can not fit into the total log space available.

So, first let’s calculate the total log space available:

LOG SIZE = (LOGPRIMARY + LOGSECOND) * 4 * LOGFILSIZ

For example, if my LOGPRIMARY=2, LOGSECOND=1 and LOGFILSIZ=4096, then my LOG SIZE = (2+1) * 4 * 4096 = 49, 152 KB = 48 MB.

If you look at the physical log files:

$ ls -l *.LOG
-rw------- 1 db2psc db2iadm 16785408 Mar 16 14:38 S0000000.LOG
-rw------- 1 db2psc db2iadm 16785408 Mar 16 14:21 S0000001.LOG
-rw------- 1 db2psc db2iadm 16785408 Mar 16 14:23 S0000002.LOG

If you look at the size of each log file, it is more by 8 KB for each file than what our actual calculations show. This is the overhead of the log space.

The LOG FULL conditions can appear most for the following two conditions:

  • The size of a single unit of work is greater than the actual size of the log files. In this case, if we are doing INSERTS and if each unit of work exceeds 48 MB, we will run out of the log space. This issue can be happily resolved by increasing all or a combination of database configurations parameters such as: LOGPRIMARY, LOGSECOND and LOGFILSIZ. This appears to solve the LOG FULL condition.
  • The second most common cause of the LOG FULL is not from the single unit of work exceeding all available log space. But, it is from the fact that a single transaction has started the unit of work and it has not yet finished with a result that the active log file is still open. You continue to do the work and when it is turn of this log file to be archived (archive logging) or use it again (circular logging), you will get LOG FULL condition. If you are using LOAD utility, the LOAD record gets logged and that log file will remain open for the duration of the LOAD command. During this condition, if a need arises to either archive the log or use the same file again, you will get LOG FULL condition.

You can use the following SQL to determine the connection (doing the work on behalf of the application) that is holding the oldest transaction that has not yet been committed. For this type of LOG FULL condition, it is best to kill the connection and let the work keep going.

SELECT MEMBER,
       TOTAL_LOG_AVAILABLE / 1048576 AS LOG_AVAILABLE_MB,
       TOTAL_LOG_USED / 1048576 AS LOG_USED_MB,
       CAST (((CASE WHEN (TOTAL_LOG_AVAILABLE + TOTAL_LOG_USED) = 0
            OR (TOTAL_LOG_AVAILABLE + TOTAL_LOG_USED) IS NULL
            OR TOTAL_LOG_AVAILABLE = -1 THEN NULL
            ELSE ((CAST ((TOTAL_LOG_USED) AS DOUBLE) / CAST (
               (TOTAL_LOG_AVAILABLE + TOTAL_LOG_USED) AS DOUBLE))) * 100 
       END)) AS DECIMAL (5,2)) AS USED_PCT,
       APPLID_HOLDING_OLDEST_XACT
FROM TABLE (MON_GET_TRANSACTION_LOG(-2))
ORDER BY USED_PCT DESC;

You may see a sample output as shown below:

MEMBER LOG_AVAILABLE_MB     LOG_USED_MB          USED_PCT APPLID_HOLDING_OLDEST_XACT
------ -------------------- -------------------- -------- --------------------------
     0                    0                    46   98.37                        185

If you notice that the LOG used is more than 98% and the connection holding the log file hostage is 185. Find out who that person is and get it fixed to release the active log file to remove the log full condition.

If you want to go further and see how long this transaction has been on hold, run the following SQL.

select a.application_handle, 
       a.workload_occurrence_state as status, 
       substr(a.session_auth_id,1, 10) as authid,
       substr(c.application_name, 1, 10) as applname, 
       int(a.uow_log_space_used/1024/1024) as logusedM, 
       timestampdiff(4, char(current timestamp 
          - b.agent_state_last_update_time)) as idleformin 
from table(mon_get_unit_of_work(NULL,-2)) as a, 
     table(mon_get_agent(NULL,NULL,NULL,-2)) as b, 
     table(mon_get_connection(NULL, -2)) as c 
where a.application_handle = b.application_handle 
  and a.coord_member = b.member
  and a.coord_member = a.member 
  and b.agent_type = 'COORDINATOR'
  and a.uow_stop_time is null
  and a.application_handle = c.application_handle 
  and a.coord_member = c.member 
  and b.event_type = 'WAIT' 
  and b.event_object = 'REQUEST' 
  and b.event_state = 'IDLE';

The sample output is shown as below:

APPLICATION_HANDLE   STATUS                           AUTHID     APPLNAME      LOGUSEDM  IDLEFORMIN 
-------------------- -------------------------------- ---------- ---------- ----------- -----------
                 185 UOWWAIT                          DB2PSC     db2bp                0          72

If you notice that connection 185 has not used much log space (less than 1 MB) and it has been waiting since last 72 minutes. This is drag on the system and this has the potential of bringing the system down to its knees.

But, please note the SQL LOG FULL condition for condition number 2 as explained above depends upon the log file that is active and that can happen for variety of reasons. One of the other common cuase that I have seen is the bad programming in which ROLLBACK is not specified on an exception and commit never happened for that unit of the work. This situation will lead to LOG FULL condition if that transaction is not resolved.

Kernel upgrade and impact on pureScale

In Linux environment, it is a common practice to upgrade the kernel for security patches, vulnerability fixes etc. And if you are running a DB2 pureScale environment, it is likely that you are going to break the GPFS GPL as soon as you upgrade the kernel and this will result into DB2 pureScale GPFS getting broken. If by any chance, you were running a production environment – you invited a big trouble for yourself.

The reason being – When you create a DB2 pureScale instance, it will compile GPL and those libraries are stored in /lib/modules/`uname -r`/extra folder. When kernel gets upgraded, the uname -r value changes and GPFS can no longer locate GPFS libraries in the new /lib/modules/`uname -r`/extra since this no longer exists. So what is the procedure one should follow to safely upgrade the Linux kernel and also get the appropriate GPFS GPL for that kernel.

You need to ask one question from your SA before they attempt to build a new kernel for whatever reason. Do you have a compatible GPFS for the Linux kernel that you are upgrading? Who can reply this? You can go to this IBM link https://www.ibm.com/support/knowledgecenter/SSFKCN/com.ibm.cluster.gpfs.doc/gpfs_faqs/gpfsclustersfaq.html and search for latest supported Linux Distributions and it will have supported Linux kernel version. If you are on latest and greatest Kernel version and it may so happen that you may not see the kernel version that you are looking at. How do you go forward? Here are the steps.

Get a test environment with same Kernel version as you have in your pureScale environment and it is not necessary that you have to have pureScale in this test environment. Install GPFS RPMs from your DB2 software directory. The sample commands are:

# cd /root/download
# tar xvfz gpfs-4.1.tar.gz 
# tar xvfz gpfs-4.1.1.11.tar.gz

# cd 4.1
# rpm -ivh gpfs.base-4.1.0-0.x86_64.rpm
# rpm -ivh gpfs.gskit-8.0.50-16.x86_64.rpm
# rpm -ivh gpfs.msg.en_US-4.1.0-0.noarch.rpm
# rpm -ivh gpfs.gpl-4.1.0-0.noarch.rpm

# cd ../4.1.1.11

# rpm -Uvh gpfs.base-4.1.1-11.x86_64.update.rpm
# rpm -Uvh gpfs.gpl-4.1.1-11.noarch.rpm
# rpm -Uvh gpfs.gskit-8.0.50-47.x86_64.rpm
# rpm -Uvh gpfs.msg.en_US-4.1.1-11.noarch.rpm

# rpm -qa | grep -i gpfs
gpfs.msg.en_US-4.1.1-11.noarch
gpfs.gpl-4.1.1-11.noarch
gpfs.base-4.1.1-11.x86_64
gpfs.gskit-8.0.50-47.x86_64

OR

# rpm -ivh 4.1/*.rpm
# rpm -Uvh 4.1.1.11/*.rpm

After installation, run command /usr/lpp/mmfs/bin/mmbuildgpl to build the GPFS build directory. The GPFS kernel libraries will be under /lib/modules/`uname -r`/extra.

Now do the kernel upgrade and reboot the machine and again run /usr/lpp/mmfs/bin/mmbuildgpl to build the GPFS build directory. If it is successful, you are good to go as new GPFS libraries are in new /lib/modules/`uname -r`/extra

But if for some reason – mmbuildgpl fails and it will if the Kernel that you upgraded has not been validated for the GPFS. What are the choices?

Check from the above IBM link if your new kernel is supported or not. If not, open a PMR and request for a special build of GPFS for the kernel that you have to upgrade for and install / upgrade the new RPMs and then repeat the /usr/lpp/mmfs/bin/mmbuildgpl and you are good to go if this succeeds,

If the kernel is supported but your DB2 software does not have the newer version then you should look for the DB2 fix pack that has the GPFS version that supports the new kernel. You can do a google search on DB2 Software Compatibility Matrix Report and this will take you to an IBM link and you will be able to find out the DB2 fix pack that has the desired GPFS version. Get the fix pack and go to server/db2/linuxamd64/gpfs folder and run the commands:

# ./db2ckgpfs -v media
# ./db2ckgpfs -v install

To know the GPFS version in media and the installed version. The base and fp directories will have the base and GPFS fix packs and run appropriate rpm commands to either upgrade or install it.

Your other option is to search IBM Fix Central and look for the desired GPFS fixpacks and then install / upgrade it. But, this is not my preferred path. I would always like to get the GPFS software from the db2 software directory.

GPFS Disk Cleanup

During pureScale instance creation, you might run into instance creation failed condition. Under some circumstances, it is possible that db2icrt does not perform a proper clean-up for GPFS and it may fail next time complaining that the disk is already a GPFS disk. When NSDs are created, GPFS writes the inode table at several places for recovery purposes and by simply using dd command to wipe out first few KB at starting of the disk may not be sufficient at times and GPFS signatures are still there.

What is the easiest way to clean / remove GPFS signatures from the disk?

You can use the following commands to first create a GPFS cluster, start it and then create NSD and then delete it. This will remove the disk cleanly.

#!/bin/bash
DOMAIN=zinox.com
NODE=db2test01
PEERDOMAIN=gpfs
DISK=/dev/sdc

mmcrcluster -N $NODE:manager-quorum -p $NODE \ 
     -R /usr/bin/scp -r /usr/bin/ssh -C $PEERDOMAIN -U $DOMAIN
mmchlicense server --accept -N $NODE
mmstartup -a
mmgetstate -a
sleep 10
cat << EOF | tee disk.txt
%nsd:
 device=$DISK
 nsd=nsd01
 usage=dataAndMetadata
EOF

mmcrnsd -F disk.txt -v no
mmdelnsd -F disk.txt
mmshutdown
mmdelnode $NODE
mmlscluster

The most important commands are mmcrnsd -F disk.txt -v no using -v switch to no and then mmdelnsd -F disk.txt.

Don't Be Shy. Get In Touch.

Any thing related to DB2 or Karmas, send me a note.

Contact Me