Hang Issue due to SMAP

While creating DB2 pureScale instance, it appears that the node becomes unresponsive under RHEL 7.2. If you reboot the node and look at the /var/log/messages, you may notice these several messages:

kernel:BUG: soft lockup - CPU#1 stuck for 23s! [mmfsd:3280]

The mmfsd is GPFS (aka IBM Spectrum Scale) file system daemon and somehow it looks that this is the cause of the this CPU soft look.

The other symptom of soft CPU lockup is the high queue seen in the vmstat output. Please look at the first column ‘r’ under procs. The value of ‘r’ would be very high in this case.

It looks that Supervisor Mode Access Prevention (SMAP) feature of Intel Xeon V4 processor (Broadwell) and Linux kernel 3.7 or later causes mmfsd to not have access to some memory space. The SMAP feature in Intel Broadwell family of CPU (including Intel Core i7 6820 HQ) has the protection enabled which disallows access from kernel-space memory to user-space memory, a feature aimed at making it harder to exploit software bugs. Now, GPFS is a kernel level access and this feature is disallowing GPFS access of kernel-space memory with a result that soft lockup of CPU occurs and that leads to system appearing hung-up.

This causes the node to appear to hang but it is actually soft CPU lockup issue as seen with the above command. The soft CPU lockup also causes the high queue – with a result that the system becomes non-responsive.

The RHEL 7.2 kernel has the support for SMAP feature by default. If your cpu has this feature or not, you can check the output from cat /proc/cpuinfo | grep smap and if you see smap in the flags section, you have this Supervisor Mode Access Prevention (SMAP) feature enabled.

GPFS has fixed this issue in v4.2.1.1 but the version that comes with DB2 11.1 FP 1 is v4.1.1.9. If you are using later version of DB2, you can find out the version of GPFS that will be installed by looking at file spec in the folder <db2softwaredir>/server_t/db2 directory.

You can disable smap feature in Linux kernel by adding kernel parameter nosamp as shown below (for RHEL 7.2).

1. Locate grub.cfg  –> It can be in different places depending upon legacy or EFI boot. In my case, this was in /boot/grub2/grub.cfg
2. Edit grub.cfg and find line associated with the system image such as line containing vmlinuz and add “nosmap” parameter at the end.

Alternatively, you can use edit /etc/default/grub and add nosmap parameter at the end of line containing GRUB_CMDLINE_LINUX and then run grub2-mkconfig -o /boot/grub2/grub.cfg and reboot the system.

You can also run the grubby command to add this kernel parameter.

# grubby --update-kernel=ALL --args=nosmap

Hang Issue due to blk_cloned_rq_check_limits

Their is another issue related with GPFS and the technote is : http://www-01.ibm.com/support/docview.wss?uid=ssg1S1009622

One of our colleague Kajen ran into this: During instance creation on RHEL 7.3 (db2icrt), I ran into problem where the disk specified for instance shared dev paths fails and the db2icrt hangs for ever. You will see the messages in /var/log/messages file regarding the failure and the message on the technote: kernel: blk_cloned_rq_check_limits: over max size limit. you can workaround the problem by adding udev rules as per the link given above.

The workaround to the above problem as per IBM technote is to write a udev rule.

If the GPFS file system block size is 4MB, and the storage is from IBM, you need to create a new rule in /etc/udev/rules.d/54-custom.rules with the following content:

ACTION=="add|change", SUBSYSTEM=="block", ATTRS{vendor}=="IBM*", RUN+="/bin/sh -c '/bin/echo 4096 > /sys/block/%k/queue/max_sectors_kb'"

You need to change the block size in the ‘echo’ command to match the file system block size. You also need to provide the correct storage vendor in the “ATTRS{vendor}” statement.