RAC and ORA-1102

Hit the following problem:

After a manual install of a database the database and its instances were not registered with the CRS.

Using srvctl I registered the database and its instances in the CRS.

Unfortunately I could not start the database or and instance. Well I could start one instance but none of the other three. Same thing happened when I used another instance (start inst1 but not inst2, inst3 and inst4, then start inst4, but not inst1, inst2, inst3).

Then I tried to start one of the other instances manually. See what happened:

SQL> startup
ORACLE instance started.

Total System Global Area 838860800 bytes
Fixed Size 2074992 bytes
Variable Size 218105488 bytes
Database Buffers 612368384 bytes
Redo Buffers 6311936 bytes
ORA-01102: cannot mount database in EXCLUSIVE mode

I found out that the culprit was an nice combination of init parameters.

CLUSTER_DATABASE_INSTANCES=4
CLUSTER_DATABASE=NO

So my explanation was that the database was willing to start one instance, but none of the other three.

So how could this happen?
Well, our partner came with its own setup tool. This is a mix of shell scripts that are governed by a collection of XML files.
As this is their first RAC environment I guess that they usually set CLUSTER_DATABASE to NO while in some other part they count the number of nodes where the instances will run. Hence the logical difference in their init parameters.

As they had pfiles on all nodes the solution was very easy. Correct the CLUSTER_DATABASE=YES, create a spfile and use that.