Andreas' Adventures

I finally have made the jump over to Australia. After a somewhat bumpy first week (oh man - what a jet-lag) I started to settle down. On my project I will do a lot of stuff with Coherence, WebLogic Portal, the OSB, Grid Control and a RAC database. I'm looking forward to do this. This is part of re-inventing myself, while being in a nice place. I will also try to blog some of my experiences in http://yetanotheraustraliablog.blogspot.com/

Sometimes you just need a system to be gone immediately. And with immediately I mean NOW. The moment I do something it should be gone. This comes in quite handy when you work in an environment that offers High Availability, such as an Oracle RAC, and you need to test a failover or a node crash. The same applies to a middle tier that runs the Oracle Application Server on multiple machines. Well, if you want to do this you can use these handy commands: [root@mymachine] cd /proc [root@mymachine] echo c > sysrq-trigger Of course you must be root to write into that file. And be aware, you must have (physical) access to the machine as you really need to boot it again. This can be done by pressing the button, or if you use virtualization or any ILO-tool by the available interface.

After having some "nice" experiences with the installation of RAC on Windows I like to share this with you - so you might find a solution easier. The environment is easy, but not without some nice "features". We have a three node RAC cluster with two machines in room 1 and the third machine in room 2 (both on the same site). We did not run cluvfy as this very often gives errors when everything is OK. On the SAN we will start with only one voting disk, and add two more later. Also the OCR will be mirrored. Furthermore we intend to use OCFS on some of the LUN's of the SAN. Well, the installation went ok, the other machine was found, etc etc etc. However the first configuration assistant failed. Research showed some error - the usual stuff about node connectivity. Stopped the OUI and retried from the command line. Found the following errors in the evmd.log Oracle Database 10g CRS Release 10.2.0.1.0 Production Copyright 1996, 2005 Oracle. All rights reserved. 2008-...

More often than once you will encounter the issue that you need to change an init-parameter that cannot be changed during an active state. SQL> alter system set sessions=500 scope=both; alter system set sessions=500 scope=both * ERROR at line 1: ORA-02095: specified initialization parameter cannot be modified Well - the beauty of RAC is of course that you can change init parameters in the spfile and bounce an instance. Your users won't notice - as the database is still available while you accomplish the change of the init parameter. SQL> alter system set sessions=500 scope=spfile sid='ORCL4'; System altered. SQL> shutdown immediate SQL> startup ORACLE instance started. Total System Global Area 1862270976 bytes Fixed Size 2072096 bytes Variable Size 436208096 bytes ...

Hit the following problem: After a manual install of a database the database and its instances were not registered with the CRS. Using srvctl I registered the database and its instances in the CRS. Unfortunately I could not start the database or and instance. Well I could start one instance but none of the other three. Same thing happened when I used another instance (start inst1 but not inst2, inst3 and inst4, then start inst4, but not inst1, inst2, inst3). Then I tried to start one of the other instances manually. See what happened: SQL> startup ORACLE instance started. Total System Global Area 838860800 bytes Fixed Size 2074992 bytes Variable Size 218105488 bytes Database Buffers 612368384 bytes Redo Buffers 6311936 bytes ORA-01102: cannot mount database in EXCLUSIVE mode I found out that the culprit was an nice combination of init parameters. CLUSTER_DATABASE_INSTANCES=4 CLUSTER_DATABASE=NO So my explanation was that the data...

Had a problem with the installation of MRCA in a RAC cluster. This can happen when you are installing the MRCA in a RAC environment and the total length of the description string is longer than 239 chars. Found this in the logfile of the MRCA assistant: [SQLPlusAction] Connect string: (DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=abcdefghijklmno01)(PORT=1582))(ADDRESS=(PROTOCOL=TCP)(HOST=abcdefghijklmno02)(PORT=1582))(ADDRESS=(PROTOCOL=TCP)(HOST=abcdefghijklmno03)(PORT=1582)))(CONNECT_DATA=(SERVICE_NAME=INFRA))) SQL> string beginning "(DESCRIPTI..." is too long. maximum size is 239 characters. Of course the above hostnames are fake (we have different ones, but just as long). The strange thing is that not all parts fail. So apparently each assistant uses a different method. Solution was to use just one or two of the hostnames in the MRCA assistant. It is a RAC cluster - so even one host would be sufficient.

When running the runcluvfy.sh I received the following error: /tmp does not exist and is not writable The reason for this is the fact that during the setup of the user equivalence the ssh to the own host was not performed. This leads to the issue that the commands in the runcluvfy.sh are not issued on the localhost. This leads to the strange error, as the /tmp directory did of course exist.

Just want to store them here for myself, but if others use them - well feel free (and drop me a note). The script for the Cluster Registry #!/bin/ksh # Author: Andreas Chatziantoniou, Accenture Technology Solutions # Date : 25-JUN-2007 # # Purpose: # Create a backup of the Oracle Cluster Registry regularly # # Usage: # ./backup_oracle_crs.sh # # This script should be used from the cron in regular intervals # # Results: # The backupfile of the Oracle Cluster Registry will be created in the /tmp # directory. The filename will contain the hostname and the time that # the backup was created. OCRBCK=/tmp/ocrbackup.`hostname`.`date +"%Y%m%d%H%M"` /opt/oracle/crs/bin/ocrconfig -export $OCRBCK And here is the script for the Voting Disk #!/bin/ksh # Author: Andreas Chatziantoniou, Accenture Technology Solutions # Date : 25-JUN-2007 # # Purpose: # Create a backup of the Oracle Voting Disk regularly # # Usage: # ./backup_oracle_voting_disk.sh # # This script should be used from the cr...

Due to a crash a part of the CRS information was lost. Until now I have not found out how this could happen in the first place. The effects were strange. The instances would not start and there was no indication of an error in the alert_log. Doing some research in the CRS revealed that the ora.ORCL.db was gone while the ora.ORACL.oracl1.inst was still there. Creating a new file in the CRS profiles and using crs_register did do the trick. Such a file should look like this: NAME=ora.ORCL.db TYPE=application ACTION_SCRIPT=/opt/oracle/crs/bin/racgwrap PLACEMENT=balanced CHECK_INTERVAL=600 RESTART_ATTEMPTS=1 FAILOVER_DELAY=0 FAILURE_INTERVAL=60 FAILURE_THRESHOLD=1 AUTO_START=1 SCRIPT_TIMEOUT=600 ACTIVE_PLACEMENT=0 OPTIONAL_RESOURCES= REQUIRED_RESOURCES= RESTART_ATTEMPTS=1 STOP_TIMEOUT=0 UPTIME_THRESHOLD=7d HOSTING_MEMBERS= USR_ORA_ALERT_NAME= USR_ORA_CHECK_TIMEOUT=0 USR_ORA_CONNECT_STR=/ as sysdba USR_ORA_DEBUG=0 USR_ORA_DISCONNECT=false USR_ORA_FLAGS= USR_ORA_IF= USR_ORA_INST_NOT_SHUTDOWN=...

Thanks to my friend Robert I now grasped the concept of the VIP in RAC environments. VIP's were created in order to speed up the process of letting the client know that the server died. Imagine that the server crashed without any possibility to inform the client. The client will then initiate a request and has to wait for the TCP timeout. That can easily be 120 seconds. The client should therefore use the VIP in the tnsnames.ora. Performing a request now will immediately give a response. The client can then initiate its TAF faster.

During the last post I mentioned that we needed rsh in order to fallback as the ssh apparently didn't work. Now when the database should be build with the dbca an error was thrown, stating that the user oracle had no user equivalence on its own node. Searching Metalink revealed that the dbca was looking for ssh in /usr/local/bin/ssh . However, ssh on AIX is located at /usr/bin/ssh . Created a softlink solved this problem. This could also be the explanation for the problems I had earlier. When setting up the user equivalance with ssh you use the commandline. As /usr/bin is in the path before /usr/local/bin from the commandline the ssh works just fine. When this is hardcoded in the Universal Insstaller or the dbca you could get these problems. BTW: it seems that this is solved in 10.2.x cu Andreas

Installing RAC does always have some nice surprises. For example the following: Although CRS and cluvrfy did not pose big problems (from what I've heard - didn't do it myself) the node could not be found. SSH works - of course. After some trying the decision was made to open RSH on both nodes. Voila. It works. This was on an AIX 5.3 . So this might be of some help for you out there. cu Andreas