The GPFS logs are in /var/adm/ras and RSCT logs are in /var/ct/<db2domainname>log/mc directory.
If you are lazy like me, create an alias in your profile to change the directory
alias rlog='cd /var/ct/db2domain*/log/mc'
The directory structure looks like:
node02:/var/ct/db2domain_20120625010135/log/mc # l total 28 drwxr-x--- 7 root root 4096 Jun 25 01:01 ./ drwxr-xr-x 5 root root 4096 Jun 25 01:01 ../ drwxr-x--- 2 root root 4096 Jun 25 01:01 IBM.ConfigRM/ drwxr-xr-x 2 root root 4096 Jun 25 01:01 IBM.GblResRM/ drwxr-xr-x 2 root root 4096 Jun 25 19:37 IBM.RecoveryRM/ drwxr-xr-x 2 root root 4096 Jun 25 01:01 IBM.StorageRM/ drwxr-xr-x 2 root root 4096 Jun 25 01:01 IBM.TestRM/
The important one is the IBM.RecoveryRM directory through which you can determine how node recovery is happening like db2 clean up, stop and start.
node02:/var/ct/db2domain_20120625010135/log/mc/IBM.RecoveryRM # ls -l -rw------- 1 root root 1048576 Jun 25 19:36 trace.7.sp -rw------- 1 root root 1048576 Jun 25 19:33 trace.8.sp -rw------- 1 root root 524288 Jun 25 23:34 trace_summary.4.sp
Use rpttr coomand to format the trace as the files are in binary format.
node02:/var/ct/db2domain_20120625010135/log/mc/IBM.RecoveryRM # rpttr -o dtic trace.8.sp | more Filename: trace.8.sp Program Name: /usr/sbin/rsct/bin/IBM.RecoveryRMd Properties: Little Endian, 32-bit mode Platform: Linux/Intel FileVersion: 5 Node Number: 1 Process Id: 8025 Machine Id: 0x00000000 Node ID: 0x31d53600595dfbb1 Trace Library Build Level: rjops002a 06/25/12 19:33:27.512329 T(4101557104) _RCD RIBME-HIST: BINDER ...Collocation from db2_db2psc_1-rs/Fixed/IBM.App lication/node03 to db2mnt-db2sd_20120625010238-rs/Fixed/IBM.Application/node03 06/25/12 19:33:27.512376 T(4101557104) _RCD RIBME-HIST: BINDER ...limit 0 <= usage 138 + test 0... 06/25/12 19:33:27.512385 T(4101557104) _RCD RIBME-HIST: BINDER ...relationship 5 to db2mnt-db2data2-rs/
If you are again lazy like me, you can take a shortcut and just use the samlog command to see the last 10 minutes of the output.
node02:/var/ct/db2domain_20120625010135/log/mc/IBM.RecoveryRM # samlog | more samlog called at 2012-06-25 23:38:52 on node02 with options node02 0.00 IBM.RecoveryRM trace_summary, IBM.GblResRM trace_summary node04 -0.06 IBM.RecoveryRM trace_summary, IBM.GblResRM trace_summary node03 -0.02 IBM.RecoveryRM trace_summary, IBM.GblResRM trace_summary ---------------------------------------------------------------------- 2012-06-25 01:01:45.790020 R(node02) T(4113562480) _RCD CHARM::processMembership() Master daemon node name - node 02, NodeNum = 1 2012-06-25 01:01:45.790020 R(node02) T(4113562480) _RCD CHARM::processMembership() Master daemon node name - node 02, NodeNum = 1 2012-06-25 01:01:45.790417 R(node02) T(4112997232) _RCD CHARM::verifyConfig Entered 2012-06-25 01:01:45.790417 R(node02) T(4112997232) _RCD CHARM::verifyConfig Entered 2012-06-25 01:01:48.697788 R(node02) T(4112997232) _RCD CHARM::verifyConfig Leaving 2012-06-25 01:01:48.697788 R(node02) T(4112997232) _RCD CHARM::verifyConfig Leaving 2012-06-25 01:01:48.766405 R(node02) T(4112087920) _RCD Decision engine started 2012-06-25 01:01:48.766405 R(node02) T(4112087920) _RCD Decision engine started 2012-06-25 01:01:49.793908 R(node02) T(4112087920) _RCD Cluster now has quorum 2012-06-25 01:01:49.793908 R(node02) T(4112087920) _RCD Cluster now has quorum 2012-06-25 01:01:49.794672 R(node02) T(4112087920) _RCD ResBase::inUse(): Resource node02 created 2012-06-25 01:01:49.794672 R(node02) T(4112087920) _RCD ResBase::inUse(): Resource node02 created 2012-06-25 01:01:49.795256 R(node02) T(4112087920) _RCD CHARM_RIBM log [Node node02/Node/IBM.PeerNode is usable]
If you have patience, go through the traces and you will know how recovery actually happens.