This section describes the steps you need to perform to debug remote processes running on the Evaluation Board Server ¹. The procedures described in Section 16.6.1 through Section 16.6.3 describe the simplest procedure for using Ladebug with the Evaluation Board Server. Since Ladebug can be started at any monitor breakpoint (not only when the program is at its initial entry point), many variations are possible.

16.6.1 Building an Executable File

To build the executable file for remote debugging:

1. Compile your source files using the -g option to save symbolic information.
2. Link the source files with the -N and -Tx options, where x is the load address for the executable file.
3. Use the cstrip utility to strip the coff header from the executable file. Keep the unstripped executable file.

To load the executable file and start the Ladebug server:

1. Set up the host Tru64 UNIX Alpha machine as the bootp server for the Evaluation Board Server (for example, the EB64).
2. Start the server.
3. Use the bootadr command on the monitor to set the boot address to the load address that was used when linking the executable file.
4. Use the netload command to load the stripped executable file across the ethernet.
5. Set a breakpoint at the entry point to the program with the stop command.
6. Start the program with the go command.
7. When the program reaches the breakpoint, enter the LADBX command to start the Ladebug server.

After starting the server, to start the Ladebug debugger on the host system:

1. Enter the following command:

   ladebug -rn <EB64> -pid 0 <exe>

   
   In this command, <exe> is the file name of the unstripped executable file and <EB64> is the internet name or address of the Evaluation Board Server.

2. After you finish debugging or want to return to the monitor's local command interface, enter the quit command. The quit command closes Ladebug and returns the monitor to its command prompt. The state of the program you are debugging is not affected by this action.

The following conditions may arise when remote debugging using the Evaluation Board Server:

• If the program under server control disables interrupts for long periods of time, Ladebug may lose communication with the Evaluation Board Server.
• The Evaluation Board Server does not support the loading of programs under Ladebug control. As such, a program must be loaded by the monitor and then connected to Ladebug, as described in Section 16.6.2.

Ladebug supports kernel debugging, which is a task normally performed by systems engineers or system administrators. A systems engineer might debug a kernel space program, which is built as part of the kernel and which references kernel data structures. A systems administrator might debug a kernel when a process is hung, or kernel parameters need to be examined or modified, or the operating system hangs, panics, or crashes. Kernel debugging aids in analyzing crash dumps.

Security

You may need to be the superuser (root login) to examine either the running system or crash dumps. Whether or not you need to be the superuser depends on the directory and file protections for the files you attempt to examine.

Compiling a Kernel for Debugging

Compilation of a kernel should be done without full optimization and without stripping the kernel of its symbol table information. Otherwise, your ability to debug the kernel is greatly reduced.

By default, compilation does not strip the symbol table information. By default, optimization is only partial. If you do not change these defaults, there should not be a problem.

Adding or Deleting Symbol Table Information

From within the debugger, you can selectively add or delete symbol table information for a kernel image, with the addstb or delstb commands. These commands can be useful because symbol table information can impact debugger performance and take up considerable disk space. The syntax is as follows:

addstb kernel_image

delstb kernel_image

Patching a Disk File

From within the debugger, you can use the patch command to correct bad data or instructions in an executable disk file. The text, initialized data, or read-only data areas can be patched. The bss segment cannot be patched because it does not exist in disk files.

The syntax is as follows:

patch expression1 = expression2

For example,

(ladebug) patch @foo = 20

Setting the Thread Context

The debugger variable $tid contains the thread identifier of the current thread. The $tid value is updated implicitly by the debugger when program execution stops or completes.

You can modify the current thread context by setting $tid to a valid thread identifier.

When there is no process or program, $tidis set to 0.

The debugger variable $tid is the same as $curthread except that $tid is used for kernel debugging.

Summary and Additional Information

The remainder of this chapter briefly describes the use of Ladebug to

• Perform local kernel debugging
• Analyze crash dumps
• Perform remote kernel debugging with the kdebug debugger

You can find additional information on kernel debugging in

• The kdbx(8) reference page
• The kdebug(8) reference page
• The DEC OSF/1 Kernel Debugging manual

The kernel debugging functionality supported by Ladebug is very similar to the functionality described in the above-listed dbx sources, with the substitution of the term ladebug for the term dbx.

17.1 Local Kernel Debugging

When you have a problem with a process, you can debug the running kernel or examine the values assigned to system parameters. (It is generally recommended that you avoid modifying the value of the parameters, which can cause problems with the kernel.)

Invoke the debugger with the following command:

# ladebug -k /vmunix /dev/mem

The -k flag maps virtual to physical addresses to enable local kernel debugging. The /vmunix and /dev/mem parameters cause the debugger to operate on the running kernel.

Now you can use Ladebug commands to display the current process identification numbers (process IDs), and trace the execution of processes. The following example shows the use of the command kps (which is the alias of the show process command) to display the process IDs:

(ladebug) kps

PID COMM 00000 kernel idle 00001 init 00014 kloadsrv 00016 update . . .

The Ladebug commands cont, next, rerun, run, setting registers, step, and stop are not available when you do local kernel debugging. (Stopping the kernel would also stop the debugger.)

If you want to examine the stack of, for example, the kloadsrv daemon, you set the $pid symbol to its process ID (14) and enter the where command, as in the following example (the spacing in the example has been altered to fit the page):

(ladebug) set $pid = 14 (ladebug) where

> 0 thread_block() ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/kern/sched_prim.c":1623, 0xfffffc000043d77c] 1 mpsleep(0xffffffff92586f00, 0x11a, 0xfffffc0000279cf4, 0x0, 0x0) ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/bsd/kern_synch.c":411, 0xfffffc000040adc0] 2 sosleep(0xffffffff92586f00, 0x1, 0xfffffc000000011a, 0x0, 0xffffffff81274210) ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/bsd/uipc_socket2.c":654, 0xfffffc0000254ff8] 3 sosbwait(0xffffffff92586f60, 0xffffffff92586f00, 0x0, 0xffffffff92586f00, 0x10180) ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/bsd/uipc_socket2.c":630, 0xfffffc0000254f64] 4 soreceive(0x0, 0xffffffff9a64f658, 0xffffffff9a64f680, 0x8000004300000000, 0x0) ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/bsd/uipc_socket.c":1297, 0xfffffc0000253338] 5 recvit(0xfffffc0000456fe8, 0xffffffff9a64f718, 0x14000c6d8, 0xffffffff9a64f8b8, 0xfffffc000043d724) ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/bsd/uipc_syscalls.c":1002, 0xfffffc00002574f0] 6 recvfrom(0xffffffff81274210, 0xffffffff9a64f8c8, 0xffffffff9a64f8b8, 0xffffffff9a64f8c8, 0xfffffc0000457570) ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/bsd/uipc_syscalls.c":860, 0xfffffc000025712c] 7 orecvfrom(0xffffffff9a64f8b8, 0xffffffff9a64f8c8, 0xfffffc0000457570, 0x1, 0xfffffc0000456fe8) ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/bsd/uipc_syscalls.c":825, 0xfffffc000025708c] 8 syscall(0x120024078, 0xffffffffffffffff, 0xffffffffffffffff, 0x21, 0x7d) ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/arch/alpha/syscall_trap.c":515, 0xfffffc0000456fe4] 9 _Xsyscall(0x8, 0x12001acb8, 0x14000eed0, 0x4, 0x1400109d0) ["/usr/sde/osf1/build/goldos.nightly1/src/kernel/arch/alpha/locore.s":1046, 0xfffffc00004486e4]

Examining the stack trace may reveal the problem. Then you can modify parameters, restart daemons, or take other corrective actions.

The kdbx Interface

The kdbx interface is a crash analysis and kernel debugging tool. It serves as a front end to the Ladebug debugger. The kdbx interface is extensible, customizable, and insensitive to changes to offsets and sizes of fields in structures. The only dependencies on kernel header files are for bit definitions in flag fields.

The kdbx interface has facilities for interpreting various symbols and kernel data structures. It can format and display these symbols and data structures in the following ways:

• In a predefined form as specified in the source code modules that currently accompany the kdbx interface
• As defined in user-written source code modules according to a standardized format for the contents of the kdbx modules

The Ladebug commands (except signals such as Ctrl/P) are available when you use the kdbx interface. (Many of these commands have aliases that match dbx commands, for the convenience of users who are accustomed to debugging kernels with the dbx debugger.) In general, kdbx assumes hexadecimal addresses for commands that perform input and output.

The sections that follow explain using kdbx to debug kernel programs.

Beginning a kdbx Session

Using the kdbx interface, you can examine either the running kernel or dump files created by the savecore utility. In either case, you examine an object file and a core file. For running systems, these files are usually /vmunix and /dev/mem, respectively. The savecore utility saves dump files it creates in the directory specified by the /sbin/init.d/savecore script. By default, the savecore utility saves dump files in the /var/adm/crash directory.

To examine a running system, enter the kdbx command with the following parameters:

# kdbx -k /vmunix /dev/mem

When you begin a debugging session, kdbx reads and executes the commands in the system initialization file /var/kdbx/system.kdbxrc. The initialization file contains setup commands and alias definitions. (For a list of kdbx aliases, see the kdbx(8) reference page.) You can further customize the kdbx environment by adding commands and aliases to one of the following initialization files:

• /var/kdbx/site.kdbxrc, which contains customized commands and alias definitions for a particular system
• ~/.kdbxrc, which contains customized commands and alias definitions for a specific user
• ./.kdbxrc, which contains customized commands and alias definitions for a specific project (this file must reside in the current working directory when kdbx is invoked)

The kdbx Interface Commands

The kdbx interface provides the following commands:

alias [name ] [command-string ]

• Sets or displays aliases. If you omit all arguments, alias displays all aliases. If you specify the variable name, alias displays the alias for name, if one exists. If you specify name and command-string, alias establishes name as an alias for command-string.

context proc | user

• Sets context to the user's aliases or the extension's aliases. This command is used only by the extensions.

coredata start_address end_address

• Dumps, in hexadecimal, the contents of the core file starting at start_address and ending before end_address.

ladebug command-string

• Passes the variable command-string to Ladebug. Specifying Ladebug is optional; if the command is not recognized by kdbx, it is passed automatically.

help [-long ] [args ]

• Prints help text.

proc [flags ] [extension ] [arguments ]

• Executes an extension and gives it control of the kdbx session until it quits. The variable extension specifies the named extension file and passes arguments to it as specified by the variable arguments. Valid flags are as follows:
  -debug
  • Causes input to and output from the extension to be displayed on the screen.
  
  -pipe in_pipe out_pipe
  • Used in conjunction with the dbx debugger for debugging extensions.
  
  -print_output
  • Causes the output of the extension to be sent to the invoker of the extension without interpretation as kdbx commands.
  
  -redirect_output
  • Used by extensions that execute other extensions to receive the output from the called extensions. Otherwise, the user receives the output.
  
  -tty
  • Causes kdbx to communicate with the subprocess through a terminal line instead of pipes. If you specify the -pipe flag, proc ignores it.

• Displays string on the terminal. If this command is used by an extension, the terminal receives no output.

• Exits the kdbx interface.

• Reads and interprets files as kdbx commands in the context of the current aliases. If the you specify the -x flag, the debugger displays commands as they are executed.

• Removes the alias, if any, from name.

Using kdbx Extensions

In addition to its commands, the kdbx interface provides extensions. You execute extensions using the kdbx command proc. For example, to execute the arp extension, you enter the following command:

kdbx> proc arp

You can create your own kdbx extensions.

For more information on the extensions, see the DEC OSF/1 Kernel Debugging manual.

17.2 Crash Dump Analysis

If your system panics or crashes, you can often find the cause by using either Ladebug or kdbx to analyze a crash dump.

The operating system can crash in the following ways:

• Hardware trap --- A hardware problem often results in the kernel trap() function being invoked.
• Software panic --- A software panic, resulting from a software failure, calls the kernel panic() function.
• Hung system --- When the system hangs, you can force the creation of dump files.

If the system crashes because of a hardware fault or an unrecoverable software state, a dump function is invoked. The dump function copies the core memory into the primary default swap disk area as specified by the /etc/fstab file structure table and the /sbin/swapdefault file. At system reboot time, the information is copied into a file, called a crash dump file.

You can analyze the crash dump file to determine what caused the crash. For example, if a hardware trap occurred, you can examine variables, such as savedefp, the program counter (pc), and the stack pointer (sp), to help you determine why the crash occurred. If a software panic caused the crash, you can use the Ladebug debugger to examine the crash dump and the uerf utility to examine the error log. Using these tools, you can determine what function called the panic() routine.

Crash dump files, such as vmunix.n and vmcore.n, usually reside in the /var/adm/crash directory. The version number (n in vmunix.n and vmcore.n ) must match for the two files.

For example, you might use the following command to examine dump files:

# ladebug -k vmunix.1 vmcore.1

Examining the Exception Frame

When you debug your code by working with a crash dump file, you can examine the exception frame using Ladebug. The variable savedefp contains the location of the exception frame. (No exception frames are created when you force a system to dump.) Refer to the header file /usr/include/machine/reg.h to determine where registers are stored in the exception frame. The following example shows an exception frame:

(ladebug) print savedefp/33X

ffffffff9618d940: 0000000000000000 fffffc000046f888 ffffffff9618d950: ffffffff86329ed0 0000000079cd612f . . . ffffffff9618da30: 0000000000901402 0000000000001001 ffffffff9618da40: 0000000000002000

Extracting the Character Message Buffer

You can use Ladebug to extract the preserved message buffer from a running system or dump files to display system messages logged by the kernel. For example:

(ladebug) print *pmsgbuf

struct { msg_magic = 405601 msg_bufx = 1181 msg_bufr = 1181 msg_bufc = "Alpha boot: memory from 0x68a000 to 0x6000000 DEC OSF/1 T1.2-2 (Rev. 5); Thu Dec 03 11:20:36 EST 1992 physical memory = 94.00 megabytes. available memory = 83.63 megabytes. using 360 buffers containing 2.81 megabytes of memory tc0 at nexus scc0 at tc0 slot 7 asc0 at tc0 slot 6 rz1 at asc0 bus 0 target 1 lun 0 (DEC RZ25 (C) DEC 0700) rz2 at asc0 bus 0 target 2 lun 0 (DEC RZ25 (C) DEC 0700) rz3 at asc0 bus 0 target 3 lun 0 (DEC RZ26 (C) DEC T384) rz4 at asc0 bus 0 target 4 lun 0 (DEC RRD42 (C) DEC 4.5d) tz5 at asc0 bus 0 target 5 lun 0 (DEC TLZ06 (C)DEC 0374) asc1 at tc0 slot 6 fb0 at tc0 slot 8 1280X1024 ln0: DEC LANCE Module Name: PMAD-BA ln0 at tc0 slot 7 ln0: DEC LANCE Ethernet Interface, hardware address: 08:00:2b:2c:f6:9f DEC3000 - M500 system Firmware revision: 1.1 PALcode: OSF version 1.14 lvm0: configured. lvm1: configured. setconf: bootdevice_parser translated 'SCSI 0 6 0 0 300 0 FLAMG-IO' to 'rz3' " } (ladebug)

The crashdc Utility

The crashdc utility collects critical data from operating system crash dump files or from a running kernel. You can use the data it collects to analyze the cause of a system crash. The crashdc utility uses existing system tools and utilities to extract information from crash dumps. The information garnered from crash dump files or from the running kernel includes the hardware and software configuration, current processes, the panic string (if any), and swap information.

The crashdc utility is invoked each time the system is booted. If it finds a current crash dump, crashdc creates a data collection file with the same numerical file name extension as the crash dump.

You can also invoke crashdc manually. The syntax of the command for invoking the data collection script is as follows:

/bin/crashdc vmunix. n /vmcore. n

See the DEC OSF/1 Kernel Debugging manual for an example of the output from the crashdc command.

Managing Crash Dump File Creation

To ensure that you are able to analyze crash dump files following a system crash, you must understand the crash dump file creation process. This process requires that you reserve space on the system for crash dump files. The amount of space you save depends upon your system configuration and the type of crash dump file you want the system to create.

Saving Dumps to a File System

When the system reboots, it attempts to save a crash dump from the crash dump partition to a file system. The savecore utility (/sbin/savecore), which is invoked during system startup before the dump partition is accessed, checks to see if the system crashed or was rebooted. If the system crashed within the last three days, the savecore utility performs the following tasks as the system reboots:

• Checks to see if a dump has been made within the last three days and that there is enough space to save it.
• Saves the dump file and kernel image into a specified directory. The default files for the kernel image and the dump file are vmunix.n and vmcore.n, respectively.
  The variable n gives the number of the crash. The number of the crash is recorded in the bounds file. After the first crash, the bounds file is created in the crash dump directory and the value one is stored in it. That value is incremented for each succeeding crash.
• Logs a reboot message using the facility LOG_CRIT, which logs critical conditions. For more information, refer to the syslog(3) reference page.
• Logs the panic string in both the ASCII and binary error log files, if the system crashed as a result of a panic.
• Attempts to save the kernel syslog message buffer from the dump files. The msgbuf.err entry in /etc/syslog.conf file specifies the file name and location for the msgbuf dump file. The default /etc/syslog.conf file specification is as follows:

  ---

  Previous

  Next

  Contents

  Index