Rolling Upgrade

A rolling upgrade is a software upgrade of a cluster that is performed while the cluster is in operation. Patching your system is one type of upgrade that can be performed using this procedure. The term “Rolling Patch” is sometimes used to describe the patching process using the Rolling Upgrade procedure. In general, the terms Rolling Patch and Rolling Upgrade are synonymous in this chapter.

In a Rolling Upgrade, one member at a time is upgraded and returned to operation while the cluster transparently maintains a mixed-version environment for the base operating system, cluster, and Worldwide Language Support (WLS) software. Clients accessing services are not aware that a rolling upgrade is in progress.

A rolling upgrade consists of an ordered series of steps, called stages. The commands that control a rolling upgrade enforce this order.

When performing a rolling upgrade, the same procedure is used for patching your system as for upgrading to a new operating system or TruCluster version. The principal difference is that for a rolling patch you use the dupatch utility and for a rolling upgrade you use the installupdate utility during the install stage.

This chapter provides the same information as the Rolling Upgrade chapter of the Cluster Installation manual. It is provided here as a convenience so you can review your patching options in one manual.




	Note: If you have not yet created your cluster, we recommend that you patch your system first. See “Patching a System Prior to Creating a Cluster” for this time-saving procedure.

The first part of this chapter contains instructions for performing a rolling upgrade, for displaying the status of a rolling upgrade, and for undoing one or more stages of a rolling upgrade. Those interested in how a rolling upgrade works can find the details in “Rolling Upgrade Commands” and the sections that follow it.

This chapter discusses the following topics:

Tasks, and combinations of tasks, you can perform during a single rolling upgrade (“Rolling Upgrade Supported Tasks”)
Tasks you cannot perform during a rolling upgrade (“Unsupported Tasks”)
How to perform a rolling upgrade (“Rolling Upgrade Procedure”)
How to display the status of a rolling upgrade (“Displaying the Status of a Rolling Upgrade”)
Iimportant information you need to be aware of if you remove or reinstall patches during a rolling upgrade (“Removing Patches Installed During a Rolling Upgrade”)
Displaying the status of a rolling upgrade (“Displaying the Status of a Rolling Upgrade”)
How to undo the stages of a rolling upgrade (“Undoing a Stage”)
The commands used during a rolling upgrade (“Rolling Upgrade Commands”)
Rolling upgrade stages (“Rolling Upgrade Stages”)
Two mechanisms that support rolling upgrades: tagged files (“Tagged Files”) and version switches (“Version Switch”)
Rolling upgrade and layered products (“Rolling Upgrade and Layered Products”)
Rolling upgrade and RIS (“Rolling Upgrade and RIS”)

Figure 4-1 “Rolling Upgrade Flow Chart” provides a simplified flow chart of the tasks and stages that are part of a rolling upgrade initiated on a Version 5.1B cluster:

Figure 4-1 Rolling Upgrade Flow Chart

Rolling Upgrade Supported Tasks

The tasks that you can perform during a rolling upgrade depend on which versions of the base operating system and cluster software are currently running on the cluster. The main focus of this chapter is to describe the behavior of a rolling upgrade that starts on a TruCluster software Version 5.1B cluster. However, because you may read this chapter in preparation for a rolling upgrade from TruCluster software Version 5.1A to Version 5.1B, we point out rolling upgrade differences between the two versions.

The following list describes the basic tasks you can perform within a rolling upgrade:

Upgrade the cluster's Tru64 UNIX base operating system and TruCluster software software. You perform this type of rolling upgrade to upgrade from the installed version to the next version.

When performing a rolling upgrade of the base operating system and cluster software, you can roll only from one version to the next version. You cannot skip versions.




	Note: A rolling upgrade updates the file systems and disks that the cluster currently uses. The roll does not update the disk or disks that contain the Tru64 UNIX operating system used to create the cluster (the operating system on which you ran clu_create). Although you can boot the original operating system in an emergency when the cluster is down, remember that the differences between the current cluster and the original operating system increase with each cluster update.

Patch the cluster's current versions of the Tru64 UNIX base operating system and TruCluster software software.
Install a New Hardware Delivery (NHD) kit (the cluster must be running TruCluster software Version 5.1A or later).

Rolling in a patch kit or an NHD kit uses the same procedure as rolling in a new release of the base operating system and cluster software. The difference is which commands you run during the install stage:

To upgrade the base operating system and cluster software, run installupdate in the install stage.
To roll in a patch kit, run dupatch in the install stage. You can invoke dupatch multiple times in the install stage to roll in multiple patch kits.
If you want to perform a no-roll patch of the cluster, do not run the clu_upgrade command. Instead run the dupatch command from a cluster member running in multi-user mode.
No-roll patching applies patches quickly and reduces the number of reboots required. It patches the cluster in one operation. However, it requires a reboot of the whole cluster to complete the operation, so the cluster is unavailable for a period.
To install an NHD kit, run nhd_install in the install stage.

Throughout this chapter, the term rolling upgrade refers to the overall procedure used to roll one or more software kits into a cluster.

As shown in Figure 4-1 “Rolling Upgrade Flow Chart”, you can perform more than one task during a rolling upgrade.

If the cluster is running Version 5.1A or Version 5.1B, a rolling upgrade can include the task combinations listed in Table 4-1 “Rolling Upgrade Tasks Supported by Version 5.1A and Version 5.1B”:

Table 4-1 Rolling Upgrade Tasks Supported by Version 5.1A and Version 5.1B

An update installation from Version 5.1A to Version 5.1B

An update installation from Version 5.1B to the next release

A patch of Version 5.1A

A patch of Version 5.1B

The installation of a New Hardware Delivery (NHD) kit onto a Version 5.1A cluster

The installation of an NHD kit onto a Version 5.1B cluster

An update installation from Version 5.1A to Version 5.1B of the base operating system and cluster software, followed by a patch of Version 5.1B

An update installation from Version 5.1B to the next release of the base operating system and cluster software followed by a patch of the next release^[1]

An NHD installation onto a Version 5.1A cluster followed by a patch of Version 5.1A

An NHD installation onto a Version 5.1B cluster followed by a patch of Version 5.1B

An update installation from Version 5.1A to Version 5.1B followed by the installation of an NHD kit for Version 5.1B

An update installation from Version 5.1B to the next release of the base operating system and cluster software followed by the installation of an NHD kit for that next release^[2]

An update installation from Version 5.1A to Version 5.1B, followed by the installation of an NHD kit for Version 5.1B, followed by a patch of Version 5.1B

An update installation from Version 5.1B to the next release, followed by the installation of an NHD kit for the next release, followed by a patch of the next release^[2]

^[1]Within one rolling upgrade, you can combine an upgrade of the base operating system and cluster software with a patch of the new software. This means that during the install stage, you can run installupdate on the first member followed by dupatch to patch the newly installed software. When you roll the remaining members they automatically get both the new software and the patches.

However, you cannot patch the current software and then upgrade the base operating system and cluster software within one rolling upgrade. This operation requires two rolling upgrades.

^[2]Allowed only if you have already installed an NHD kit on the Version 5.1A or Version 5.1B cluster.

Unsupported Tasks

The following list describes tasks that you cannot perform or that we recommend you do not attempt during a rolling upgrade:

Do not remove or modify files in the /var/adm/update directory. The files in this directory are critical to the roll. Removing them can cause a rolling upgrade to fail.
During the install stage, you cannot run a dupatch command followed by an installupdate command. To patch the current software before you perform a rolling upgrade, you must perform two complete rolling upgrade operations: one to patch the current software, and one to perform the update installation.
You cannot bypass versions when performing a rolling upgrade of the base operating system and cluster software. You can only roll from one version to the next version.
Do not use the /usr/sbin/setld command to add or delete any of the following subsets:
- Base Operating System subsets (those with the prefix OSF).
- TruCluster Server subsets (those with the prefix TCR).
- Worldwide Language Support (WLS) subsets (those with the prefix IOS).
Adding or deleting these subsets during a roll creates inconsistencies in the tagged files.
Do not install a layered product during the roll.
Unless a layered product's documentation specifically states that you can install a newer version of the product on the first rolled member, and that the layered product knows what actions to take in a mixed-version cluster, we strongly recommend that you do not install either a new layered product or a new version of a currently installed layered product during a rolling upgrade.
For more information about layered products and rolling upgrades, see “Rolling Upgrade and Layered Products”.

Rolling Upgrade Procedure

In the procedure in this section, unless otherwise stated, run commands in multi-user mode. Each step that corresponds to a stage refers to the section that describes that stage in detail. We recommend that you read the detailed description of stages in “Rolling Upgrade Stages” before performing the rolling upgrade procedure.

Some stages of a rolling upgrade take longer to complete than others. Table 4-2 “Time Estimates for Rolling Upgrade Stages” lists the approximate time it takes to complete each stage.

Table 4-2 Time Estimates for Rolling Upgrade Stages

Stage	Duration
Preparation	Not under program control.
Setup	45 - 120 minutes.^[1]
Preinstall	15 - 30 minutes.^[1]
Install	The same amount of time it takes to run installupdate, dupatch, nhd_install, or a supported combination of these commands on a single system.
Postinstall	Less than 1 minute.
Roll (per member)	Patch: less than 5 minutes. Update installation: about the same amount of time it takes to add a member.^[2]
Switch	Less than 1 minute.
Clean	30 - 90 minutes.^[1]
^[1]These stages create, verify, or remove the tagged files required for a rolling upgrade. The time that it takes to run one of these stages depends on the speed of the member executing the command, the speed of the storage, and whether the member executing the command is the CFS server for the root (`/`), `/usr`, and `/var` file systems. Consider relocating these file systems to the member where you will run the clu_upgrade command. ^[2]After rolling the lead member, use parallel rolls to roll multiple members simultaneously and shorten the time it takes to roll a cluster.

You can use the following procedure to upgrade a TruCluster software Version 5.1A cluster to Version 5.1B, and to upgrade a cluster that is already at Version 5.1B.

Prepare the cluster for the rolling upgrade (“Preparation Stage”):
1. Choose one cluster member to be the lead member (the first member to roll). (The examples in this procedure use a member whose memberid is 2 as the lead member. The example member's host name is provolone.)
2. Back up the cluster.
3. If you will perform an update installation during the install stage, remove any blocking layered products, listed in Table 4-6 “Blocking Layered Products”, that are installed on the cluster.
4. To determine whether the cluster is ready for an upgrade, run the clu_upgrade -v check setup lead_memberid command on any cluster member. For example:
  # clu_upgrade -v check setup 2
  If a file system needs more free space, use AdvFS utilities such as addvol to add volumes to domains as needed. For disk space requirements, see “Preparation Stage”. For information on managing AdvFS domains, see the Tru64 UNIX AdvFS Administration manual.
5. Verify that each system's firmware will support the new software. Update firmware as needed before starting the rolling upgrade.

Perform the setup stage (“Setup Stage”).




	Notes: If your current cluster is at Version 5.1A or later and if you plan to upgrade the base operating system and cluster software during the install stage, mount the device or directory that contains the new TruCluster software kit before running clu_upgrade setup. The setup command will copy the kit to the `/var/adm/update/TruClusterKit` directory. If your current cluster is at Version 5.1A or later and if you plan to install an NHD kit during the install stage, mount the device or directory that contains the new NHD kit before running clu_upgrade setup. The setup command will copy the kit to the `/var/adm/update/NHDKit` directory.

On any member, run the clu_upgrade setup lead_memberid command. For example:

# clu_upgrade setup 2

“Setup Stage” shows the menu displayed by the clu_upgrade command.

When the setup stage is completed, clu_upgrade prompts you to reboot all cluster members except the lead member.

One at a time, reboot all cluster members except the lead member. Do not start the preinstall stage until these members are either rebooted or halted.
Perform the preinstall stage (“Preinstall Stage”).
On the lead member, run the following command:
# clu_upgrade preinstall
If your current cluster is at Version 5.1A or later, the preinstall command gives you the option of verifying or not verifying the existence of the tagged files created during the setup stage.
- If you have just completed the setup stage and have done nothing to cause the deletion any of the tagged files, you can skip this test.
- If you completed the setup stage a while ago and are not sure what to do, let preinstall test the correctness of the tagged files.

Perform the install stage (“Install Stage”).




	Note: During the install stage you load the new software on the lead member, in effect rolling that member. When you perform the roll stage, this new software is propagated to the remaining members of the cluster. The clu_upgrade command does not load software during the install stage. The loading of software is controlled by the commands you run: installupdate, dupatch, or nhd_install. See Table 4-1 “Rolling Upgrade Tasks Supported by Version 5.1A and Version 5.1B” for the list of rolling upgrade tasks and combination of tasks supported for Version 5.1A and Version 5.1B.

See Chapter 3 “Patch Installation and Removal Instructions” for instructions on installing a patch kit using the dupatch command.
See the Tru64 UNIX Installation Guide for detailed information on using the installupdate command.
See the Tru64 UNIX New Hardware Delivery Release Notes and Installation Instructions that came with your NHD kit for detailed information on using the nhd_install command.

If the software you are installing requires that its installation command be run from single-user mode, halt the system and boot the system to single-user mode:

# shutdown -h now
>>> boot -fl s




	Note: Halting and booting the system ensures that it provides the minimal set of services to the cluster and that the running cluster has a minimal reliance on the member running in single-user mode. In particular, halting the member satisfies services that require the cluster member to have a status of DOWN before completing a service failover. If you do not first halt the cluster member, services will probably not fail over as expected.

When the system reaches single-user mode, run the following commands:

# init s
# bcheckrc
# lmf reset

Run the installupdate, dupatch, or nhd_install command.
To roll in multiple patch kits, you can invoke dupatch multiple times in a single install stage. Be aware that doing so may make it difficult to isolate problems should any arise after the patch process is completed and the cluster is in use.
You cannot run a dupatch command followed by an installupdate command. To patch the current software before you perform a rolling upgrade, you must perform two complete rolling upgrade operations: one to patch the current software, and one to perform the update installation.

(Optional) After the lead member performs its final reboot with its new custom kernel, you can perform the following manual tests before you roll any additional members:
1. Verify that the newly rolled lead member can serve the shared root (/) file system.
  1. Use the cfsmgr command to determine which cluster member is currently serving the root file system. For example:
    # cfsmgr -v -a server / Domain or filesystem name = / Server Name = polishham Server Status : OK
  2. Relocate the root (/) file system to the lead member. For example:
    # cfsmgr -h polishham -r -a SERVER=provolone /
2. Verify that the lead member can serve applications to clients. Make sure that the lead member can serve all important applications that the cluster makes available to its clients.
  You decide how and what to test. We suggest that you thoroughly exercise critical applications and satisfy yourself that the lead member can serve these applications to clients before continuing the roll. For example:
  - Manually relocate CAA services to the lead member. For example, to relocate the application resource named cluster_lockd to lead member provolone:
    # caa_relocate cluster_lockd -c provolone
  - Temporarily modify the default cluster alias selection priority attribute, selp, to force the lead member to serve all client requests directed to that alias. For example:
    # cluamgr -a alias=DEFAULTALIAS,selp=100
    The lead member is now the end recipient for all connection requests and packets addressed to the default cluster alias.
    From another member or from an outside client, use services such as telnet and ftp to verify that the lead member can handle alias traffic. Test client access to all important services that the cluster provides.
    When you are satisfied, reset the alias attributes on the lead member to their original values.
Perform the postinstall stage (“Postinstall Stage”).
On the lead member, run:
# clu_upgrade postinstall

Perform the roll stage (“Roll Stage”).

Roll the members of the cluster that have not already rolled.^[1]

You can roll multiple members simultaneously (parallel roll), subject to the restriction that the number of members not being rolled (plus the quorum disk, if one is configured) is sufficient to maintain cluster quorum.

To roll a member, do the following:

Halt the member system and boot it to single-user mode. For example:
# shutdown -h now >>> boot -fl s
When the system reaches single-user mode, run the following commands:
# init s # bcheckrc # lmf reset
Roll the member:
# clu_upgrade roll
If you are performing parallel rolls, use the -f option with the clu_upgrade roll command. This option causes the member to automatically reboot without first prompting for permission:
# clu_upgrade -f roll
The roll command verifies that rolling the member will not result in a loss of quorum. If a loss of quorum will result, then the roll of the member does not occur and an error message is displayed. You can roll the member later, after one of the currently rolling members has rejoined the cluster and its quorum vote is available.
If the roll proceeds, the member is prepared for a reboot. If you used the -f option, no prompt is displayed; the reboot occurs automatically. If you did not use the -f option, clu_upgrade displays a prompt that asks whether you want to reboot at this time. Unless you want to examine something specific before you reboot, enter yes. (If you enter yes, it may take approximately half a minute before the actual reboot occurs.)
Perform parallel rolls to minimize the time needed to complete the roll stage. For example, on an eight-member cluster with a quorum disk, after rolling the lead member, you can roll four members in parallel.
1. Begin the roll stage on a member. (The lead member was rolled during the install stage. You do not perform the roll stage on the lead member.)
2. When you see a message similar to the following, begin the roll stage on the next member:
  *** Info *** You may now begin the roll of another cluster member.
  If you see a message that begins like the following, it is probably caused by the number of currently rolling members that contribute member votes.
  *** Info *** The current quorum conditions indicate that beginning a roll of another member at this time may result in the loss of quorum.
  In this case, you have the following options:
  - You can wait until a member completes the roll stage before you begin to roll the next member.
  - If there is an unrolled member that does not contribute member votes, you can begin the roll stage on it.
Continue to roll members until all members of the cluster have rolled. Before starting each roll stage, wait until you see the message that it is all right to do so.
When you roll the last member, you will see a message similar to the following:
*** Info *** This is the last member requiring a roll.




	Note: The roll actually takes place during the reboot. The clu_upgrade roll command sets up the it(8) scripts that will be run during the reboot. When you reboot, the it scripts roll the member, build a customized kernel, and then reboot again so the member will be running on its new customized kernel. When the member boots its new customized kernel, it has completed its roll and is no longer running on tagged files.

Perform the switch stage (“Switch Stage”).
After all members have rolled, run the switch command on any member.
# clu_upgrade switch
One at a time, reboot each member of the cluster.
Perform the clean stage (“Clean Stage”).
Run the following command on any member to remove the tagged (.Old..) files from the cluster and complete the upgrade.
# clu_upgrade clean

Removing Patches Installed During a Rolling Upgrade

The following sections provide important information you need to be aware of if you remove or reinstall patches during a rolling upgrade.

Undoing a CSP or ERP When Paused at the Postinstall Stage

When applying an ERP or CSP to a TruCluster system it is good practice to stop at the Postinstall stage and perform testing of the patch prior to rolling the patch on all the cluster members. This can help reduce further impacts to the cluster. Installing the ERP/CSP using the Rolling Patch method is fairly straightforward. The removal of the patch is where questions arise.

This note provides a summary of the steps and commands used to perform this task on a two-member TruCluster system.

This procedure assumes that the ERP/CSP was installed using the Rolling Patch upgrade method (using clu_upgrade) and not the No Roll method. This procedure does not apply to ERPs or CSPs that where installed using the NoRoll method or to situations where the ERP/CSP was installed on a single member TruCluster system.

Prior to installing patches, always make sure to maintain current backups of the following file systems for a TruCluster system, along with disklabels for associated disks. Also note system configuration information using the sys_check -all command.

/ = cluster_root#root Shared by all members
/usr = cluster_usr#usr Shared by all members
/var = cluster_var#varShared by all members
/cluster/members/member1/boot_partition = root1_domain#root
Member1 Member specific root file system.
/cluster/members/member1/boot_partition = root2_domain#root
Member2 Member specific root file system
Any additional Member specific root file systems

Summary of Steps:

While logged on to Server 1 (Lead Member) perform the undo of the Postinstall stage.
# clu_upgrade undo postinstall
Shut down SERVER1 (Lead Member) then boot single-user mode to run the dupatch patch removal program. Do not shut down to single-user level. Instead, shut down the system and boot to single-user mode:
# shutdown -h now P00>> boot -fl s Entering Single-User Mode
Perform the following commands on SERVER1:
# init s # bcheckrc # lmf reset

Perform the dupatch delete procedure. For systems running Patch Kit 4 and earlier, SERVER1 (Lead Member) must be at single-user mode for this procedure. With systems running Patch Kit 5 and later, you can run the procedure in multi-user mode. Change directories to the location of the CSP patch and run dupatch.

# cd /usr/patch_kit

Run dupatch on SERVER1 and select Patch Kit Deletion from the Main Menu:

#  ./dupatch

  Main Menu:
  ---------

    1)  Patch Kit Installation
    2)  Patch Kit Deletion
    3)  Patch Kit Documentation
    
    4)  Patch Tracking
    5)  Patch Baseline Analysis/Adjustment
    
    h)  Help on Command Line Interface
    
    q)  Quit

Enter your choice: 2

Ignore any Special Instructions displayed at this time; they do not apply to ERPs or CSPs. You should, however, refer to the ERP/CSP Patch Release Notes for any special instructions specific to the ERP/CSP being removed.

After the Special Instructions are presented, the following menu is displayed:
1) Patches for Tru64 UNIX V5.1B 2) Patches for TruCluster Server V5.1B
If the CSP/ERP is a base operating system patch select 1, if the patch is specific to TruCluster, select 2.
Enter you name and comment when prompted.
Following the display of a long listing of operating system or TruCluster patches (depending on selection made in last menu) identify the patch from the list and select it.
The patch will then be deleted and, if necessary, a new kernel will be built.
When prompted to reboot answer yes; if not prompted, manually reboot system.

With the dupatch deletion completed and the Lead Member rebooted, perform undo of the Install Stage as follows:
# shutdown -h now Halted P00>>
On SERVER2 (Member2 or any Non-Lead Member) run enter the clu_upgrade undo install command. It warns to use dupatch, which has already completed, so answer yes. This restores tagged files and may take a few minutes (for example, 20 minutes on an ES45 using EVA5000 SAN storage):
# clu_upgrade undo install
Boot the Lead Member into multi-user mode and perform undo of the Preinstall Stage.
# clu_upgrade undo preinstall
Undo Setup Stage. To do this first you have to disable tagged files on other members (Member 2 (SERVER2) and so on).
1. Check the status of the clu_upgrade process and if non-lead member is running on Tagged Files, perform the following command to disabled tagged files. The non-lead members should be running on tagged files at this time.
  # clu_upgrade tagged disable 2
  :
  This command is run on Member2 (SERVER2) and the member ID 2 is passed as a parameter for this member. Any additional members need to be disabled also.
2. Reboot the member that the Tagged Files where disabled on to complete the process; in this example, Member 2 (SERVER2).
3. Run the undo Setup Stage on the lead member (SERVER1) to complete the deletion of the patch. This task takes approximately 10 minutes to complete.
  # clu_upgrade undo setup

The cluster is now backed out from the clu_upgrade patch installation.

Caution on Removing Version Switched Patches

When removing version switched patches on a cluster, do not remove version switched patches that were successfully installed in a previous rolling upgrade.

This situation can occur because more than one patch subset may contain the same version switched patch. Although both the new and old patches can be removed during a roll, only the most recently installed, newer version switched patch can be properly removed.

The older version switched patch can only be properly removed according to the documented procedure associated with that patch. This usually requires running some program before beginning the rolling upgrade to remove the patch.

If you accidentally remove the older version switched patch, the rolling upgrade will most likely fail on the switch stage. To correct this situation, you will have to undo the upgrade by undoing all the stages up to and including the "install" stage. You will then need to reinstall the original version switched patch from the original patch kit that contained it.

Steps Prior to the Switch Stage

You can remove a patch kit you installed during the rolling upgrade at any time prior to issuing the clu_upgrade switch command by returning to the install stage, rerunning dupatch, and selecting the Patch Deletion item in the Main Menu. See “Removing Patches” for information about removing patches with dupatch.

The procedure is as follows:

Uninstall the patch kit as described in “Removing Patches”.
Run the clu_upgrade undo install command.
Note that although you do not have to run the clu_upgrade install command when installing a patch kit or an NHD kit, you must run the clu_upgrade undo install command if you want to remove those kits and undo the install stage. After you run the clu_upgrade undo install, you can continue undoing stages as described in “Undoing a Stage”.

Steps for After the Switch Stage

To remove patches after you have issued the clu_upgrade switch command, you will have to complete the current rolling upgrade procedure and then rerun the procedure from the beginning (starting with the setup stage).

When you run the install stage, you must bring down your system to single-user mode as described in steps 1 through 6 of “Installing Patches from Single-User Mode”. When you rerun dupatch (step 7), select the Patch Deletion item in the Main Menu. See “Removing Patches” for information about removing patches with dupatch.

If the patch uses the version switch, you can still remove the patch, even after you have issued the clu_upgrade switch command. Do this as follows:

Complete the current rolling upgrade procedure.
Undo the patch that uses the version switch by following the instructions in the release note for that patch. Note that the last step to undo the patch will require a shutdown of the entire cluster.
Rerun the rolling upgrade procedure from the beginning (starting with the setup stage). When you rerun dupatch, select the Patch Deletion item in the Main Menu.

Use the grep command to learn which patches use the version switch. For example, in the C shell:

# grep -l PATCH_REQUIRES_VERSION_SWITCH=\"Y\" /usr/.smdb./*PAT*.ctrl

For information about version switches, see “Version Switch”.

Note:

If you rerun the rolling upgrade procedure to remove patches, the prompts you receive during the setup stage will be different from those issued during the initial rolling upgrade. Those prompts will look as follows:

Do you want to continue to upgrade the cluster? [yes]: Return

What type of upgrade will be performed?

1) Rolling upgrade using the installupdate command
2) Rolling patch using the dupatch command
3) Both a rolling upgrade and a rolling patch
4) Exit cluster software upgrade

Enter your choice: 2

Displaying the Status of a Rolling Upgrade

The clu_upgrade command provides the following options for displaying the status of a rolling upgrade. You can run status commands at any time.

To display the overall status of a rolling upgrade: clu_upgrade -v or clu_upgrade -v status.
To determine whether you can run a stage: clu_upgrade check [stage]. If you do not specify a stage, clu_upgrade tests whether the next stage can be run.
To determine whether a stage has started or completed: clu_upgrade started stage or clu_upgrade completed stage.
To determine whether a member has rolled: clu_upgrade check roll memberid.
To verify whether tagged files have been created for a layered product: clu_upgrade tagged check [prod_code [prod_code ...]]. If you do not specify a product code, clu_upgrade inspects all tagged files in the cluster.




	Notes: During a roll, there might be two versions of the clu_upgrade command in the cluster — an older version used by members that have not yet rolled, and a newer version (if included in the update distribution or patch kit). The information that is displayed by the status command might differ depending on whether the command is run on a member that has rolled. Therefore, if you run the status command on two members, do not be surprised if the format of the displayed output is not the same. If you run clu_upgrade status after running installupdate, clu_upgrade will display a message indicating that the install stage is complete. However, the install stage is not really complete until you run the clu_upgrade postinstall command.

Undoing a Stage

The clu_upgrade undo command provides the ability to undo a rolling upgrade that has not completed the switch stage. You can undo any stage except the switch stage and the clean stage. You must undo stages in order; for example, if you decide to undo a rolling upgrade after completing the preinstall stage, you undo the preinstall stage and then undo the setup stage.




	Note: Before undoing any stage, we recommend that you read the relevant version of the Cluster Release Notes to determine whether there are restrictions related to the undoing of any stage.

To undo a stage, use the undo command with the stage that you want to undo. The clu_upgrade command determines whether the specified stage is a valid stage to undo. Table 4-3 “Undoing a Stage” outlines the requirements for undoing a stage:

Table 4-3 Undoing a Stage

Stage to Undo	Command	Comments
Setup	clu_upgrade undo setup	You must run this command on the lead member. In addition, no members can be running on tagged files when you undo the setup stage. Before you undo the setup stage, use the clu_upgrade -v status command to determine which members are running on tagged files. Then use the clu_upgrade tagged disable `memberid` command to disable tagged files on those members. (See “Tagged Files” for information about tagged files and the commands used to manipulate them.) When no members are running on tagged files, run the clu_upgrade undo setup command on the lead member.
Preinstall	clu_upgrade undo preinstall	You must run this command on the lead member.
Install	clu_upgrade undo install	You can run this command on any member except the lead member. Halt the lead member. Then run the clu_upgrade undo install command on any member that has access to the halted lead member's boot disk. When the command completes, boot the lead member. If you installed a patch kit or individual patches during the install stage, you must first run dupatch to uninstall the patch kit before running the clu_upgrade undo install command. “Removing Patches Installed During a Rolling Upgrade” describes the steps for removing a patch kit during a rolling upgrade.
Postinstall	clu_upgrade undo postinstall	You must run this command on the lead member.
Roll	clu_upgrade undo roll `memberid`	You can run this command on any member except the member whose roll stage will be undone. Halt the member whose roll stage is being undone. Then run the clu_upgrade undo roll `memberid` command on any other member that has access to the halted member's boot disk. When the command completes, boot the halted member. The member will now be using tagged files.

Rolling Upgrade Commands

The clu_upgrade command, described in clu_upgrade(8), controls the overall flow of a rolling upgrade and ensures that the stages are run in order. During the install stage, you run one or more of installupdate, dupatch, or nhd_install to load and install software. These commands are rolling upgrade aware; they are modified to understand which actions they are allowed to take during the install and roll stages of a rolling upgrade.

When you start a rolling upgrade, the cluster is running the software from the previous release. For the first part of any rolling upgrade, you are running the clu_upgrade command that is already installed on the cluster. If a new version is installed during the rolling upgrade, there may be minor differences in the on-screen display and behavior between the two versions of the command.

The following two tables show at which stages during a rolling upgrade new versions of upgrade commands, if shipped with the kits being installed, become available during a rolling upgrade:^[2]

Table 4-4 “Stages and clu_upgrade Versions When Performing a Rolling Upgrade from Version 5.1A” maps commands to stages for a rolling upgrade from Version 5.1A to Version 5.1B, a patch kit, or an NHD kit; or to Version 5.1B of the base operating system and cluster software followed by a patch of the new software within the same rolling upgrade.
Table 4-5 “Stages and clu_upgrade Versions When Performing a Rolling Upgrade from Version 5.1B” maps commands to stages for a rolling upgrade from Version 5.1B to the next release of the operating system and cluster software, a Version 5.1B patch kit, or an NHD kit; or to the next release of the base operating system and cluster software followed by a patch of the new software within the same rolling upgrade.

Table 4-4 Stages and clu_upgrade Versions When Performing a Rolling Upgrade from Version 5.1A

Stage	Version 5.1A	Next Release^[1]	Comments
Preparation	X		The currently installed (old) version of clu_upgrade is always run in this stage.
Setup	X		The currently installed (old) version of clu_upgrade is always run in this stage. If performing an update installation, the new version of the clu_upgrade is extracted from the TruCluster software kit and installed at `/usr/sbin/clu_upgrade`, replacing the old version. Because this replacement is done before tagged files are created, all members will use the new clu_upgrade throughout the remainder of the rolling upgrade.
Preinstall		X	If the rolling upgrade includes an update installation, all members use the new version of clu_upgrade installed during the setup stage. (Otherwise, members continue to run the current version of clu_upgrade.)
Install		X	If the rolling upgrade includes an update installation, all members use the version of clu_upgrade installed during the setup stage. During the update installation, a new version of installupdate replaces the old one. A patch kit always installs the latest version of dupatch. If performing a patch, and if the patch kit includes a new version of clu_upgrade, the new version is installed and will be used by all cluster members starting with the postinstall stage.
Postinstall		X	If a new version of clu_upgrade was installed in either the setup stage or the install stage, all members use the new version.
Roll		X	If a new version of clu_upgrade was installed in either the setup stage or the install stage, all members use the new version.
Switch		X	If a new version of clu_upgrade was installed in either the setup stage or the install stage, all members use the new version.
Clean		X	If a new version of clu_upgrade was installed in either the setup stage or the install stage, all members use the new version.
^[1]Version 5.1B of Tru64 UNIX and TruCluster software, a patch kit for Version 5.1A, or the installation of an NHD kit on Version 5.1A.

Table 4-5 Stages and clu_upgrade Versions When Performing a Rolling Upgrade from Version 5.1B

Stage	Version 5.1B	Next Release^[1]	Comments
Preparation	X		The currently installed (old) version of clu_upgrade is always run in this stage.
Setup	X		The currently installed (old) version of clu_upgrade is always run in this stage. If performing an update installation, the new version of the clu_upgrade is extracted from the TruCluster software kit and installed at `/usr/sbin/clu_upgrade`, replacing the old version. Because this replacement is done before tagged files are created, all members will use the new clu_upgrade throughout the remainder of the rolling upgrade.
Preinstall		X	If the rolling upgrade includes an update installation, all members use the new version of clu_upgrade installed during the setup stage. (Otherwise, members continue to run the current version of clu_upgrade.)
Install		X	If the rolling upgrade includes an update installation, all members use the version of clu_upgrade installed during the setup stage. During the update installation, a new version of installupdate replaces the old one. A patch kit always installs the latest version of dupatch. If performing a patch, and if the patch kit includes a new version of clu_upgrade, the new version is installed and will be used by all cluster members starting with the postinstall stage.
Postinstall		X	If a new version of clu_upgrade was installed in either the setup stage or the install stage, all members use the new version.
Roll		X	If a new version of clu_upgrade was installed in either the setup stage or the install stage, all members use the new version.
Switch		X	If a new version of clu_upgrade was installed in either the setup stage or the install stage, all members use the new version.
Clean		X	If a new version of clu_upgrade was installed in either the setup stage or the install stage, all members use the new version.
^[1]The next release of Tru64 UNIX and TruCluster software, a patch kit for Version 5.1B, or the installation of an NHD kit on Version 5.1B.

Rolling Upgrade Stages

The following sections describe each of the rolling upgrade stages.




	Note: These sections only describe the stages. Use the procedure in “Rolling Upgrade Procedure” to perform a rolling upgrade.

Preparation stage (“Preparation Stage”)
Setup stage (“Setup Stage”)
Preinstall stage (“Preinstall Stage”)
Install stage (“Install Stage”)
Postinstall stage (“Postinstall Stage”)
Roll stage (“Roll Stage”)
Switch stage (“Switch Stage”)
Clean stage (“Clean Stage”)

Preparation Stage

Command	Where Run	Run Level
clu_upgrade -v check setup `lead_memberid`	any member	multi-user mode

During the preparation stage, you back up all important cluster data and verify that the cluster is ready for a roll. Before beginning a rolling upgrade, do the following:

Choose one member of the cluster as the first member to roll. This member, known as the lead member, must have direct access to the root (/), /usr, /var, and, if used, i18n file systems.
Make sure that the lead member can run any critical applications. You can test these applications after you update this member during the install stage, but before you roll any other members. If a problem occurs, you can try to resolve it on this member before you continue. If you cannot resolve a problem, you can undo the rolling upgrade and return the cluster to its pre-roll state. (“Undoing a Stage” describes how to undo rolling upgrade stages.)

Back up the clusterwide root (/), /usr, and /var file systems, including all member-specific files in these file systems. If the cluster has a separate i18n file system, back up that file system. In addition, back up any other file systems that contain critical user or application data.




	Note: If you perform an incremental or full backup of the cluster during a rolling upgrade, make sure to perform the backup on a member that is not running on tagged files. If you back up from a member that is using tagged files, you will only back up the contents of the `.Old..` files. Because the lead member never uses tagged files, you can back up the cluster from the lead member (or any other member that has rolled) during a rolling upgrade. Most sites have automated backup procedures. If you know that an automatic backup will take place while the cluster is in the middle of a rolling upgrade, make sure that backups are done on the lead member or on a member that has rolled.

If you plan to run the installupdate command in the install stage, remove any blocking layered products listed in Table 4-6 “Blocking Layered Products” that are inst