Exadata cell update

Exadata Cell Storage

updateX2-2 quarter rack

image version from 11.2.3.3.0 to 11.2.3.3.1

Infrastructure & Governance

Date : 2015. 1.30 작성자 : 김 종 인

■ Cell node update

- patchmgr 란 ? ( = DB node 의 dbnodeupdate.sh)exadata 의 cell node 의 new release 를 적용하기 위한 모든 수동작업을 대체하고 작업전 check 및 적용과정을 자동으로 수행해주는 기능을 한다 .cell node image version / Infiniband Firmware 를 최신으로 Update 한다 .

- patchmgr 버전 ( 별도로 다운받을 경우 11.2.3.x 와 12.1.x 버전용 따로 존재하므로 확인후 다운받아야 한다 . 문서 888828.1 항상확인 )

- 2015 년 1 월 29 일 기준 최신버전

11.2.3.3.1 - Patch 19166601 - Storage server and InfiniBand switch software

12.1.2.1.0 - Patch 17885582 - Storage server and InfiniBand switch software Patch 19681939 - Exadata 12.1.1.1.1 patchmgr plug-ins

해당 패치압축을 풀면 patchmgr 가 내장되어 있음 .

- 작업후 Linux Kernel version Updata 됨

Image 11.2.3.3.0 ☞ 2.6.39-400.126.1 (RHEL 5.9)Image 11.2.3.3.1 ☞ 2.6.39-400.128.17 (RHEL 5.10)Image 12.1.2.1.0 ☞ 2.6.39-400.243.1 (RHEL 6.6)

https://support.oracle.com/epmos/faces/ui/patch/PatchDetail.jspx?parent=DOCUMENT&sourceId=888828.1&patchId=19166601




작업순서

1. cell 의 ip 정보등 확인 작업

[EXA1]root@exa1:/root# dcli -g cell_group -l root /opt/oracle.cellos/ipconf -verifygtceladm01: Verifying of Exadata configuration file /opt/oracle.cellos/cell.confgtceladm01: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checksgtceladm02: Verifying of Exadata configuration file /opt/oracle.cellos/cell.confgtceladm02: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checksgtceladm03: Verifying of Exadata configuration file /opt/oracle.cellos/cell.confgtceladm03: Done. Configuration file /opt/oracle.cellos/cell.conf passed all verification checks

2. 해당 Patch 다운로드 후 압축해제

-[EXA1]root@exa1:/root# unzip p19166601_112331_Linux-x86-64.zip-Archive: p19166601_112331_Linux-x86-64.zip- creating: patch_11.2.3.3.1.140708/- inflating: patch_11.2.3.3.1.140708/sundcs_36p_repository_2.1.3_4.pkg - inflating: patch_11.2.3.3.1.140708/upgradeIBSwitch.sh - creating: patch_11.2.3.3.1.140708/linux.db.rpms/- inflating: patch_11.2.3.3.1.140708/linux.db.rpms/perl-XML-Parser-2.34-6.1.2.2.1.x86_64.rpm - creating: patch_11.2.3.3.1.140708/plugins/- inflating: patch_11.2.3.3.1.140708/plugins/010-check_17854520.sh ....................


작업순서

3. [EXA1]root@exa1:/root# crsctl stop crs

4. [EXA1]root@exa1:/root# dcli -g /root/cell_group -l root 'service celld stop‘5. [EXA1]root@exa1:/root# dcli -g /root/cell_group -l root 'service celld status‘

6. [EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -reset_force (patchmgr -h 시 -reset_force 옵션은 없음 )

2015-01-20 13:36:41 +0900 :DONE: reset_force

7. [EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -cleanup

2015-01-20 13:37:54 +0900 :Working: DO: Cleanup ...2015-01-20 13:37:55 +0900 :SUCCESS: DONE: Cleanup

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -patch_check_prereq -ignore_alerts

(-ignore_alerts 옵션없이 진행할 경우 Cell alert 등으로 인해 Fail 됨 )


작업순서

8. [EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# ./patchmgr -cells /root/cell_group -patch -ignore_alerts

( 마찬가지로 -ignore_alerts 옵션없이 진행할 경우 Cell alert 등으로 인해 Fail 됨 )

pre-reboot 절차에 45 분 소요된다고 표시가 됨 . ( 실제론 10 분정도 소요 )

2015-01-20 13:58:26 +0900 Wait for patch pre-reboot procedures

||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Min-utes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minutes left 045///// Minutes left 045----- Minutes left 045\\\\\ Minutes left 045||||| Minutes left 045||||| Minu

2015-01-29 16:15:20 +0900 2 of 5 :SUCCESS: DONE: Waiting to finish pre-reboot patch actions.2015-01-29 16:15:20 +0900 :Working: DO: Execute plugin check for Patching ...2015-01-29 16:15:21 +0900 :SUCCESS: DONE: Execute plugin check for Patching.2015-01-29 16:15:21 +0900 3 of 5 :Working: DO: Finalize patch on cells. Cells will reboot. Up to 5 minutes ...2015-01-29 16:15:43 +0900 3 of 5 :SUCCESS: DONE: Finalize patch on cells.


작업순서

pre-reboot 절차가 끝나면 120 분 소요메세지와 함께 작업진행됨 ( 시간은 예측치이기 때문에 해당경우의 실제 작업은 78 분 남기고 종료되었음 )2015-01-29 16:15:43 +0900 4 of 5 :Working: DO: Wait for cells to reboot and come online. Up to 120 minutes ...2015-01-29 16:16:43 +0900 Wait for patch finalization and reboot

||||| 2015-01-29 16:16:45 +0900 Minutes left 120///// 2015-01-29 16:16:45 +0900 Minutes left 120----- 2015-01-29 16:16:45 +0900 Minutes left 120\\\\\ 2015-01-29 16:16:45 +0900 Minutes left 120||||| 2015-01-29 16:16:45 +0900 Minutes left 120||||| 2015-01-29 16:16:45 +0900 Minutes left 120///// 2015-01-29 16:16:45 +0900 Minutes left 120----- 2015-01-29 16:16:45 +0900 Minutes left 120\\\\\ 2015-01-29 16:16:45 +0900 Minutes left 120||||| 2015-01-29 16:16:45 +0900 Minutes ft…………………………………………………………………………………….. 078///// 2015-01-29 17:12:14 +0900 Minutes left 078----- 2015-01-29 17:12:14 +0900 Minutes left 078\\\\\ 2015-01-29 17:12:14 +0900 Minutes left 078||||| 2015-01-29 17:12:14 +0900 Minutes left 078||||| 2015-01-29 17:12:14 +0900 Minutes left 078///// 2015-01-29 17:12:14 +0900 Minutes left 078----- 2015-01-29 17:12:14 +0900 Minutes left 078\\\\\ 2015-01-29 17:12:14 +0900 Minutes left 078||||| 2015-01-29 17:12:14 +0900 Minutes left 078

2015-01-29 17:13:20 +0900 4 of 5 :SUCCESS: DONE: Wait for cells to reboot and come online.2015-01-29 17:13:20 +0900 5 of 5 :Working: DO: Check the state of patch on cells. Up to 5 minutes ...2015-01-29 17:14:02 +0900 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.2015-01-29 17:14:02 +0900 :Working: DO: Execute plugin check for Post Patch ...2015-01-29 17:14:03 +0900 :SUCCESS: DONE: Execute plugin check for Post Patch.

9. Cell Rebooting 후 상태 확인 (service celld status)

10. [EXA1]root@exa1:/root# crsctl start crs


작업순서

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# dcli -g /root/cell_group -l root imageinfogtceladm01: gtceladm01: Kernel version: 2.6.39-400.128.17.el5uek #1 SMP Tue May 27 13:20:24 PDT 2014 x86_64gtceladm01: Cell version: OSS_11.2.3.3.1_LINUX.X64_140708gtceladm01: Cell rpm version: cell-11.2.3.3.1_LINUX.X64_140708-1gtceladm01: gtceladm01: Active image version: 11.2.3.3.1.140708gtceladm01: Active image activated: 2015-01-20 15:00:37 +0900gtceladm01: Active image status: successgtceladm01: Active system partition on device: /dev/md6gtceladm01: Active software partition on device: /dev/md8gtceladm01: gtceladm01: In partition rollback: Impossiblegtceladm01: gtceladm01: Cell boot usb partition: /dev/sdm1gtceladm01: Cell boot usb version: 11.2.3.3.1.140708gtceladm01: gtceladm01: Inactive image version: 11.2.3.3.0.131014.1gtceladm01: Inactive image activated: 2015-01-06 17:03:56 +0900gtceladm01: Inactive image status: successgtceladm01: Inactive system partition on device: /dev/md5gtceladm01: Inactive software partition on device: /dev/md7gtceladm01: gtceladm01: Boot area has rollback archive for the version: 11.2.3.3.0.131014.1gtceladm01: Rollback to the inactive partitions: Possible


작업순서

[EXA1]root@exa1:/root/patch_11.2.3.3.1.140708# dcli -g /root/cell_group -l root imagehistorygtceladm01: Version : 11.2.3.3.0.131014.1gtceladm01: Image activation date : 2015-01-06 17:03:56 +0900gtceladm01: Imaging mode : freshgtceladm01: Imaging status : successgtceladm01: gtceladm01: Version : 11.2.3.3.1.140708gtceladm01: Image activation date : 2015-01-20 15:00:37 +0900gtceladm01: Imaging mode : out of partition upgradegtceladm01: Imaging status : successgtceladm01: gtceladm02: Version : 11.2.3.3.0.131014.1gtceladm02: Image activation date : 2015-01-06 17:53:13 +0900gtceladm02: Imaging mode : freshgtceladm02: Imaging status : successgtceladm02: gtceladm02: Version : 11.2.3.3.1.140708gtceladm02: Image activation date : 2015-01-20 14:42:04 +0900gtceladm02: Imaging mode : out of partition upgradegtceladm02: Imaging status : successgtceladm02: gtceladm03: Version : 11.2.3.3.0.131014.1gtceladm03: Image activation date : 2015-01-06 16:57:02 +0900gtceladm03: Imaging mode : freshgtceladm03: Imaging status : successgtceladm03: gtceladm03: Version : 11.2.3.3.1.140708gtceladm03: Image activation date : 2015-01-20 14:42:37 +0900gtceladm03: Imaging mode : out of partition upgradegtceladm03: Imaging status : success


작업 유의사항- 작업로그는 아래의 로그를 tail -f 로 열어서 확인 (vi 등 wrte 작업 절대금지 ) tail -f /u01/patch_12.1.2.1.0.141206.1/patchmgr.stdout

각 노드별로 성공적으로 진행되었음을 확인 할수 있음gtceladm03: _EXIT_PASS_Cell gtceladm03 172.16.40.103 2015-01-20 14:44:08 +0900: Found: 11.2.3.3.1.140708 state success Expected: 11.2.3.3.1.140708 state success.2015-01-20 15:02:04 +0900 ++++++++++++++++++ Logs so far end ++++++++++2015-01-20 15:02:04 +0900 5 of 5 :SUCCESS: DONE: Check the state of patch on cells.2015-01-20 15:02:04 +0900 :Working: DO: Execute plugin check for Post Patch ...2015-01-20 15:02:04 +0900 ++++++++++++++++++ Logs so far begin ++++++++++2015-01-20 15:02:05 +0900 ++++++++++++++++++ Logs so far end ++++++++++2015-01-20 15:02:05 +0900 :SUCCESS: DONE: Execute plugin check for Post Patch.================PatchMgr run ended 2015-01-20 15:02:05 +0900 =========== patch 시작시 표시되는 Warning 메시지 반드시 준수

********************************************************************************NOTE Cells will reboot during the patch or rollback process.NOTE For non-rolling patch or rollback, ensure all ASM instances usingNOTE the cells are shut down for the duration of the patch or rollback.NOTE For rolling patch or rollback, ensure all ASM instances usingNOTE the cells are up for the duration of the patch or rollback.

WARNING Do not start more than one instance of patchmgr.WARNING Do not interrupt the patchmgr session.WARNING Do not alter state of ASM instances during patch or rollback.WARNING Do not resize the screen. It may disturb the screen layout.WARNING Do not reboot cells or alter cell services during patch or rollback.WARNING Do not open log files in editor in write mode or try to alter them.

NOTE All time estimates are approximate.NOTE You may interrupt this patchmgr run in next 60 seconds with CONTROL-c.********************************************************************************


작업 유의사항

- patchmgr -h[EXA1]root@exa1:/u01/patch_12.1.2.1.0.141206.1# ./patchmgr -hUsage:./patchmgr -cells cell_host_file [-patch_check_prereq | -rollback_check_prereq [-rolling] [-ignore_alerts]] [-patch | -rollback [-rolling] [-ignore_alerts]] [-cleanup]

./patchmgr -ibswitches [ibswitch_list_file] <-upgrade | -downgrade> [-ibswitch_precheck] [-force]]

./patchmgr -h

OPTIONS -h Displays this screen

Cell patching and rollback The following options are supported for cell patching and(or) rollback:

-cells cell_list_file Specifies the name of the cell list file. The file contains one cell hostname or ip per line. The cell patching will fail if the list file is not specified.

-cleanup Cleanup mode. Cleans up all patch files and temporary content on all cells. Before cleaning up, collects logs and information for problem diagnostics and analysis. Cleaning up patch files can be done manually if patch fails by removing directory /root/_cellupd_dpullec_ on each cell.

-ignore_alerts Ignore any active hardware alerts on the Exadata cell and proceed with the patching.

-patch Applys the patch, including firmware updates, wherever pos-sible (BIOS, Disk Controller and if possible disk drives) to all cells in the cell list file. -patch_check_prereq Runs prerequisite check on all the cells to determine if the patch can be applied to the cells.

-rollback Rolls back the patch. -rollback_check_prereq Runs prerequisite check on all the cells to determin if the cells can be rolled back for this specific patch.

-rolling Applies the patch or executes the rollback in rolling fashion, one cell at a time.

Environment variable EXA_PATCH_ACTIVATE_TIMEOUT_SECONDS controls the timeout value waiting for the grid disks to be activated. The default is set to 36000 (10 hours).


작업 유의사항

- patchmgr -h-smtp_from "addr" The address patchmgr notification sent from. sendmail daemon must be started to send email notification.

-smtp_to "addr1 addr2 addr3 ..." The address(es) patchmgr notification sent to. sendmail daemon must be started to send email notification.

-smtp_set_envelope_sender Use the same from address in Return-Path: mail header.

Infiniband switch upgrade and downgrade The following options are supported for infiniband switch upgrade and(or) downgrade: -ibswitches [ibswitch_list_file] Specifies the name of the InfiniBand switch list file. The file has one switch hostname or ip per line. If no filename is provided, then it runs the command on all InfiniBand switches discovered from this host by running ibswitches command.

-downgrade Downgrade the InfiniBand switches in the list file to 2.1.3-4

-force Specifies to proceed with the upgrade or downgrade even on non-critical failures. -ibswitch_precheck Runs the pre-update validation checks on the InfiniBand switches in the list file.

-upgrade Upgrade the InfiniBand switches in the list file to 2.1.5-1

Infrastructure & Governance

Technology

Exadata cell update