AIX KDB: "no real storage"

When using AIX kdb to analysis kernel dump, the following error may happen:

[kdb_read_mem] no real storage @ FFFFFFFFFFF98B0

The reason is that kdb is trying to access virtual memory which was swapped to disk. AIX kernel core dump contains data in physical memory only. So for data structure in swap partition or swap file, kdb cannot address it correctly. That's why such error happened. The solution is to pin all kernel data structure and text segments so that they never are swapped to disk. If you encouter the error, that means you are not good luck and some kernel data strcuture were not pinned. No solution.

Good news for  AIX version 7.1, it is a default setting and this problem won't happen.

AIX Kernel Dump Analysis

1. Uncompress core dump from customer

#ulimit -f unlimited //Avoid too large file error on AIX
#ulimit -Hf unlimited

#gunzip SR_99999999_99997.7TD.000.snap.pax.Z
#pax -rvf SR_99999999_99997.7TD.000.snap.pax -x pax

#cd dump
#dmpuncompress -p dump.BZ
#uncompress unix.Z

2. Load kdb

#kdb dump unix

dump mapped from @ 700000000000000 to @ 7000001475f12c5

Preserving 1921497 bytes of symbol table [unix]
Component Names:
 1)  minidump [2 entries]
 2)  dmp_minimal [10 entries]
 3)  proc [1636 entries]
 4)  thrd [5152 entries]
 5)  mtrc [97 entries]
 6)  lfs [6 entries]
 7)  bos [6 entries]
[kdb_dump_llseek] Invalid file offset
WARNING: Version mismatch between unix file and command kdb
           START              END <name>
0000000000001000 0000000004070000 start+000FD8
F00000002FF47600 F00000002FFDF940 __ublock+000000
000000002FF22FF4 000000002FF22FF8 environ+000000
000000002FF22FF8 000000002FF22FFC errno+000000
F1000F0A00000000 F1000F0A10000000 pvproc+000000
F1000F0A10000000 F1000F0A18000000 pvthread+000000
Dump analysis on CHRP_SMP_PCI POWER_PC POWER_6 machine with 32 available CPU(s)  (64-bit registers)
Processing symbol table...
.......................done
read vscsi_scsi_ptrs OK, ptr = 0x0
[kdb_read_mem] no real storage @ F1000100120D9000

(0) status
(0) stat

3. Common used files in kernel snap
       

errpt(kernel log): general/errpt.out
LVM configuration: lvm/*.snap
Process/Memory/Drivers: kernel/kernel.snap

         
4. Refer the following post for general introduction of kdb

http://www.ibm.com/developerworks/aix/library/au-kdbsteps/index.html

SVN+SSH Access without password

Amazon EC2 instances allows SSH login without password. If you setup SVN server on Amazon EC2 instance,  you can easily use SVN+SSH access your source code server without password as well. The following is guideline.

Assume svn repository was created as normal:

$sudo svnadmin create /var/lib/svnroot
$sudo chown -R ubuntu /var/lib/svnroot
$sudo chmod -R g+rws svnroot

Then at client side:

$export SVN_EDITOR=vi
$export SVN_SSH="ssh -i /path/to/your.pem -l ubuntu"
$svn import myproject svn+ssh://ec2instancename.compute-1.amazonaws.com/var/lib/svnroot/myproject

SCST Build Error: arch/x86/Makefile_32.cpu: No such file or directory

Try to build SCST module on Ubuntu 11(Linux 3.0) and got the following error:

cd scst && make all
make[1]: Entering directory `/home/hui/scst/scst'
cd src && make all
make[2]: Entering directory `/home/hui/scst/scst/src'
make -C /lib/modules/3.0.22-scstcustom/build SUBDIRS=/home/hui/scst/scst/src BUILD_DEV=m
make[3]: Entering directory `/usr/src/linux-headers-3.0.22-scstcustom'
/usr/src/linux-headers-3.0.22-scstcustom/arch/x86/Makefile:39: /usr/src/linux-headers-3.0.22-scstcustom/arch/x86/Makefile_32.cpu: No such file or directory
make[3]: *** No rule to make target `/usr/src/linux-headers-3.0.22-scstcustom/arch/x86/Makefile_32.cpu'.  Stop.
make[3]: Leaving directory `/usr/src/linux-headers-3.0.22-scstcustom'
make[2]: *** [all] Error 2
make[2]: Leaving directory `/home/hui/scst/scst/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/hui/scst/scst'
make: *** [scst] Error 2

It was caused by incorrect linux-headers. Try to recreate headers for arch/x86 and the problem was fixed:

Go to Linux source directory:

#cd /usr/src/linux-3.0.0/arch
#tar zcvf x86.tgz x86

Put x86.tgz to /usr/src/linux-headers-3.0.22-scstcustom/arch and untar the x86 files

#cd /usr/src/linux-headers-3.0.22-scstcustom/arch/
#tar xvf x86.tgz

locale: Cannot set LC_CTYPE to default locale: No such file or directory

Ubuntu 11 errors:

perl: warning: Please check that your locale settings:
    LANGUAGE = (unset),
    LC_ALL = (unset),
    LC_CTYPE = "UTF-8",
    LANG = "en_US.UTF-8"
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

After executing the following commands, the error was fixed.

$sudo locale-gen en_US
$sudo dpkg-reconfigure locales

Compile SCST on ubuntu 11

1) Get Compile tools

$sudo apt-get install subversion
$sudo apt-get kernel-package libncurses5-dev fakeroot wget bzip2

2) Get Linux source

$cd ~
$sudo svn co https://scst.svn.sourceforge.net/svnroot/scst/trunk scst
$cd /usr/src
$sudo apt-get source linux-image-$(uname -r)
$cd /usr/src/linux-3.0.0
$sudo cp /home/hui/scst/iscsi-scst/kernel/patches/put_page_callback-3.0.patch .
$sudo cp /home/hui/scst/scst/kernel/scst_exec_req_fifo-3.0.patch .
$sudo patch -p1 <put_page_callback-3.0.patch
$sudo patch -p1 <scst_exec_req_fifo-3.0.patch
$sudo cp /boot/config-3.0.0-12-generic .config

4) Build kernel

$sudo make-kpkg clean
$sudo fakeroot make-kpkg --initrd --append-to-version=-scstcustom kernel_image kernel_headers

5) Install kernel

$sudo dpkg -i linux-headers-3.0.22-scstcustom_3.0.22-scstcustom-10.00.Custom_i386.deb
$sudo dpkg -i linux-image-3.0.22-scstcustom_3.0.22-scstcustom-10.00.Custom_i386.deb

6) Reboot the host and switch to scstcustom  then build scst

$cd ~/scst
$sudo make scst scst_install iscsi iscsi_install scstadm scstadm_install

That's all. You will be able to find scst modules in /lib/modules/`uname -r`/