Oracle - ASM : Miroir, mon beau miroir, ….by albanlepunk
Des fois on pense avoir tout prévu pour sécuriser son infrastructure Oracle : cluster RAC étendu, diskgroups ASM en redondance NORMAL, 3° voting disk sur un partage NFS distinct, ….
Ceinture et bretelle donc …
Sauf que …. dans la vraie vie d’un informaticien (et à plus forte raison d’un consultant Oracle) la loi de Murphy est incontournable.
Une coupure d’une baie de stockage suivie d’une fausse manipulation d’un administrateur plus tard et vous vous retrouvez avec la situation suivante qu’on vous demande à vous de dépatouiller :
SQL> select name, STATE, OFFLINE_DISKS from v$asm_diskgroup; NAME STATE OFFLINE_DISKS ------------------------------ ----------- ------------- DGDATA MOUNTED 0 DGFRA MOUNTED 0 OCRMIRROR MOUNTED 0 OCRVOTING DISMOUNTED 0
Oups, le diskgroup OCRVOTING à l’état DISMOUNTED, ça ne sent pas très bon pour le cluster, d’autant plus :
- qu’il est impossible de rajouter des disques à ce diskgroup vu qu’il est DISMOUNTED
- qu’il est impossible de le monter, même en mode FORCE, vu qu’il manque des disques
Heureusement l’architecture a été bien pensée, et l’existence d’un groupe de disques OCRMIRROR (contenant pour l’instant l’OCR mirroir comme son nom l’indique) va nous être utile.
On commence par ‘déplacer’ les voting disks sur ce diskgroup
root@dblc00000025vm01 dev]# crsctl replace votedisk +OCRMIRROR Successful addition of voting disk 86b1b5a6583d4f22bf97f7af26f0ef9d. Successful deletion of voting disk 2bbecebc98fd4fcbbf96de5cc98bed12. Successful deletion of voting disk 18c58cd6ce034f3bbfca178420967aa0. Successfully replaced voting disk group with +OCRMIRROR. CRS-4266: Voting file(s) successfully replaced
Vérification :
[root@dblc00000025vm01 dev]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 86b1b5a6583d4f22bf97f7af26f0ef9d (/dev/oracleasm/disks/OCRVOTING_AB02) [OCRMIRROR] Located 1 voting disk(s).
On continue en supprimant les entêtes des disques concernés puis en les recréant (attention : vous êtes sensés savoir ce que vous faites quand vous faites ce type de manipulation)
[root@dblc00000025vm01 dev]# dd if=/dev/zero of=/dev/xvdh1 bs=1024k count=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.215507 s, 48.7 MB/s [root@dblc00000025vm01 dev]# oracleasm querydisk /dev/xvdj1 Device "/dev/xvdj1" is marked an ASM disk with the label "OCRVOTING_AA02" [root@dblc00000025vm01 dev]# dd if=/dev/zero of=/dev/xvdj1 bs=1024k count=10 10+0 records in 10+0 records out 10485760 bytes (10 MB) copied, 0.199342 s, 52.6 MB/s [root@dblc00000025vm01 dev]# [root@dblc00000025vm01 dev]# oracleasm createdisk OCRVOTING_AA01 /dev/xvdh1 Writing disk header: done Instantiating disk: failed Clearing disk header: done [root@dblc00000025vm01 dev]# oracleasm createdisk OCRVOTING_AA02 /dev/xvdj1 Writing disk header: done Instantiating disk: failed Clearing disk header: done [root@dblc00000025vm01 dev]# [root@dblc00000025vm01 dev]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Cleaning disk "OCRVOTING_AA01" Cleaning disk "OCRVOTING_AA02" Scanning system for ASM disks... [root@dblc00000025vm01 dev]# oracleasm listdisks DATA_AA01 DATA_AB01 FRA_AA01 FRA_AB01 OCRVOTING_AB01 OCRVOTING_AB02 [root@dblc00000025vm01 dev]# [root@dblc00000025vm01 dev]# oracleasm createdisk OCRVOTING_AA02 /dev/xvdj1 Writing disk header: done Instantiating disk: done [root@dblc00000025vm01 dev]# oracleasm createdisk OCRVOTING_AA01 /dev/xvdh1 Writing disk header: done Instantiating disk: done [root@dblc00000025vm01 dev]# oracleasm scandisks Reloading disk partitions: done Cleaning any stale ASM disks... Scanning system for ASM disks... [root@dblc00000025vm01 dev]# oracleasm listdisks DATA_AA01 DATA_AB01 FRA_AA01 FRA_AB01 OCRVOTING_AA01 OCRVOTING_AA02 OCRVOTING_AB01 OCRVOTING_AB02
On va créer un diskgroup temporaire pour faire changer le statut des disques concernés dans ASM de ‘MEMBER’ à ‘FORMER’, puis le supprimer.
SQL> create diskgroup DGTEST external redundancy disk '/dev/oracleasm/disks/OCRVOTING_AA01','/dev/oracleasm/disks/OCRVOTING_AA02'; Diskgroup created. SQL> select name, state from v$asm_diskgroup; NAME STATE ------------------------------ ----------- DGDATA MOUNTED DGFRA MOUNTED OCRMIRROR MOUNTED OCRVOTING DISMOUNTED DGTEST MOUNTED SQL> drop diskgroup DGTEST; Diskgroup dropped. SQL> select name,path,header_status from v$asm_disk NAME PATH HEADER_STATU ------------------------------ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ------------ /dev/oracleasm/disks/OCRVOTING_AA01 FORMER /dev/oracleasm/disks/OCRVOTING_AA02 FORMER /dev/oracleasm/disks/OCRVOTING_AB01 MEMBER /app/voting/vote3 MEMBER OCRMIRROR_0000 /dev/oracleasm/disks/OCRVOTING_AB02 MEMBER DGFRA_0001 /dev/oracleasm/disks/FRA_AB01 MEMBER DGFRA_0002 /dev/oracleasm/disks/FRA_AA01 MEMBER DGDATA_0000 /dev/oracleasm/disks/DATA_AB01 MEMBER DGDATA_0002 /dev/oracleasm/disks/DATA_AA01 MEMBER 9 rows selected.
On droppe l’ancien diskgroup OCRVOTING
SQL> drop diskgroup OCRVOTING force including contents; Diskgroup dropped
On recréé le diskgroup OCRVOTING
SQL> create diskgroup OCRVOTING normal redundancy disk '/dev/oracleasm/disks/OCRVOTING_AA01','/dev/oracleasm/disks/OCRVOTING_AA02'; Diskgroup created. SQL> select name, state from v$asm_diskgroup; NAME STATE ------------------------------ ----------- DGDATA MOUNTED DGFRA MOUNTED OCRMIRROR MOUNTED OCRVOTING MOUNTED
Puis on lui rajoute son disque quorum NFS, il faut modifier le paramètre compatible.asm du diskgroup pour avoir accès à cette fonctionnalité.
SQL> alter diskgroup OCRVOTING SET ATTRIBUTE 'compatible.asm' = '11.2'; Diskgroup altered. SQL> alter diskgroup OCRVOTING add QUORUM FAILGROUP OCRVOTING_0002 DISK '/app/voting/vote3'; Diskgroup altered. SQL> select GROUP_NUMBER, NAME, NAME,FAILGROUP from v$asm_disk; GROUP_NUMBER NAME NAME FAILGROUP ------------ ------------------------------ ------------------------------ ------------------------------ 0 3 OCRMIRROR_0000 OCRMIRROR_0000 OCRMIRROR_0000 4 OCRVOTING_0000 OCRVOTING_0000 OCRVOTING_0000 4 OCRVOTING_0001 OCRVOTING_0001 OCRVOTING_0001 2 DGFRA_0001 DGFRA_0001 FG_FRA_AB01 2 DGFRA_0002 DGFRA_0002 FG_FRA_AA01 1 DGDATA_0000 DGDATA_0000 FG_DATA_AB01 1 DGDATA_0002 DGDATA_0002 FG_DATA_AA01 4 OCRVOTING_0002 OCRVOTING_0002 OCRVOTING_0002 9 rows selected.
il faut ensuite restaurer le spfile ASM dans le diskgroup, puis
on va redéplacer les voting sur OCRVOTING (Attention seulement en 11.2)
[root@dblc00000025vm01 dev]# crsctl replace votedisk +OCRVOTING Successful addition of voting disk 4d47939b7e5c4f36bf3bbfcb04ec9c6d. Successful addition of voting disk 2045390006194f34bf42086af70e5126. Successful addition of voting disk 311f1a648aa04fa1bf241b204fe5652f. Successful deletion of voting disk 86b1b5a6583d4f22bf97f7af26f0ef9d. Successfully replaced voting disk group with +OCRVOTING. CRS-4266: Voting file(s) successfully replaced
Vérifications
[root@dblc00000025vm01 dev]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- ----------------- --------- --------- 1. ONLINE 4d47939b7e5c4f36bf3bbfcb04ec9c6d (/dev/oracleasm/disks/OCRVOTING_AA01) [OCRVOTING] 2. ONLINE 2045390006194f34bf42086af70e5126 (/dev/oracleasm/disks/OCRVOTING_AA02) [OCRVOTING] 3. ONLINE 311f1a648aa04fa1bf241b204fe5652f (/app/voting/vote3) [OCRVOTING] Located 3 voting disk(s).
On va supprimmer la référence à l’OCR d’OCRVOTING
[root@dblc00000025vm01 dev]# ocrconfig -delete +OCRVOTING [root@dblc00000025vm01 dev]# ocrconfig -add +OCRVOTING [root@dblc00000025vm01 dev]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3168 Available space (kbytes) : 258952 ID : 603000217 Device/File Name : +OCRMIRROR Device/File integrity check succeeded Device/File Name : +OCRVOTING Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded [root@dblc00000025vm01 dev]# ocrconfig -delete +OCRMIRROR [root@dblc00000025vm01 dev]# ocrconfig -add +OCRMIRROR [root@dblc00000025vm01 dev]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 3 Total space (kbytes) : 262120 Used space (kbytes) : 3168 Available space (kbytes) : 258952 ID : 603000217 Device/File Name : +OCRVOTING Device/File integrity check succeeded Device/File Name : +OCRMIRROR Device/File integrity check succeeded Device/File not configured Device/File not configured Device/File not configured Cluster registry integrity check succeeded Logical corruption check succeeded
Un test de reboot des serveurs et c’est bon, c’est reparti !