Ondemand Crashed

[Gobi21]

Hi All,

I was working with CMOD Admin and deleted a Application Group. While deleting the group, it asked for a confirmation and suddenly the Ondemand got disconnected.
I started Ondemand after that and it is not running now.Here is the error message:

25225    1        0000000000000001 11/11/13 09:56:54.173995 INFO   ArcDB_Init Connecting to database (type=2, name=xxxxxxxx)
25225    1        0000000000000001 11/11/13 09:56:58.368366 ERROR  ArcDB_Init Unable to establish connection to the database
25225    1        0000000000000001 11/11/13 09:56:58.369039 ENTER  ArcDBP_ErrorCheck
25225    1        0000000000000001 11/11/13 09:56:58.369997 ENTER  ArcDB_Commit
25225    1        0000000000000001 11/11/13 09:56:58.370416 ENTER  ArcDBP_Commit
25225    1        0000000000000001 11/11/13 09:56:58.373284 RETURN ArcDBP_Commit RC=-1
25225    1        0000000000000001 11/11/13 09:56:58.373853 RETURN ArcDB_Commit RC=6
25225    1        0000000000000001 11/11/13 09:56:58.386605 ERROR  ArcDBP_ErrorCheck A DB error message issued. Check System Log or System Console
25225    1        0000000000000001 11/11/13 09:56:58.387311 RETURN ArcDBP_ErrorCheck RC=6
25225    1        0000000000000001 11/11/13 09:56:58.387864 RETURN ArcDB_Init RC=6
25225    1        0000000000000001 11/11/13 09:56:58.388328 ENTER  ArcDB_Cancel
25225    1        0000000000000001 11/11/13 09:56:58.388768 RETURN ArcDB_Cancel
25225    1        0000000000000001 11/11/13 09:56:58.389243 ENTER  ArcDB_Shutdown
25225    1        0000000000000001 11/11/13 09:56:58.389643 INFO   ArcDB_Shutdown Free the OnDemand application handles
25225    1        0000000000000001 11/11/13 09:56:58.390076 ENTER  ArcDBP_FSTerm
25225    1        0000000000000001 11/11/13 09:56:58.390633 RETURN ArcDBP_FSTerm
25225    1        0000000000000001 11/11/13 09:56:58.391557 ENTER  ArcDBP_TermOptions
25225    1        0000000000000001 11/11/13 09:56:58.392001 RETURN ArcDBP_TermOptions
25225    1        0000000000000001 11/11/13 09:56:58.392332 ENTER  ArcDBP_TermTableNames
25225    1        0000000000000001 11/11/13 09:56:58.392760 RETURN ArcDBP_TermTableNames
25225    1        0000000000000001 11/11/13 09:56:58.393340 ENTER  ArcDBP_UnLoadDBMS
25225    1        0000000000000001 11/11/13 09:56:58.393765 INFO   ArcDBP_UnLoadDBMS Unloading database module
25225    1        0000000000000001 11/11/13 09:56:58.394162 RETURN ArcDBP_UnLoadDBMS
25225    1        0000000000000001 11/11/13 09:56:58.394566 RETURN ArcDB_Shutdown RC=0
25225    1        0000000000000001 11/11/13 09:56:58.395075 INFO   ArcSERVP_ArchiveServer Number of db servers to use: 20
25225    1        0000000000000001 11/11/13 09:56:58.395525 RETURN ArcSERVP_ArchiveServer RC=6
25225    1        0000000000000001 11/11/13 09:56:58.395941 ENTER  ArcCSL_LockTerm
25225    1        0000000000000001 11/11/13 09:56:58.396417 RETURN ArcCSL_LockTerm RC=0
25225    1        0000000000000001 11/11/13 09:56:58.399468 INFO   ArcTRACE_Term Trace function terminated


I have checked with database team and they told that DB is UP and Running. I am not able to get the exact error message from the above trace log. I have changed the database password in ars.cfg file, however while connecting via TOAD, i am able to access only with the old password.Hence, Ondemand didnt come to a point where it has to read the ars.cfg file.

Could you please help me on this?.

[kbsiva]

Hi,
  The message says it is not able to connect to the database. If it is working with the old password from toad, would suggest that you revert back the password in the cfg file and try to bring up the service.

Thanks
Siva

[Gobi21]

Hi Siva,

Thanks for the reply. I have reverted back the old password in CFG file after Checking with the new password.

So the current CFG File has old password only. I am not able to bring the service up still.

Thanks,
Gobi

[Alessandro Perucchi]

Hello Gobi21,

I suppose you still have CMOD 8.5.0.X and that you are working with Oracle, on a Unix server.

Does the SQLPLUS command works from the Unix command line as the CMOD Instance owner user?
Is the tnsnames.ora still correctly configured?

I'm asking such "trivial" questions, because maybe something changed in your environment, and the best way is to test and check every configuration files, and check all the connectivity.
Not only tnsping, but really also if sqlplus works, and enter in the database from the CMOD server.

It is worthless to test with TOAD, because TOAD is not on your server, and you cannot simply say, it works with TOAD so my server must work!!  That's maybe not your case, but I had some customers who told me that looking at me with serious eyes!! :-D

Sincerely yours,
Alessandro

[Gobi21]

Hi Alessandro,

Thanks for the reply. Thats a good view.

I am using CMOD 8.4.1 with Oracle 10.2.0.4.0 - 64bit. we have smiliar setup in other servers and its working fine.

I tried using SQLPLUS Command and i am able to execure a sample query. Its looking good.

Even tnsping <Instance name> from the server is working fine.

There is no change in the tnsnames.ora for the past three years.

I have no clue since the Ondemand crashed just like that when i was perfoming deletion of an Application Group.

Thanks for your help again.

[Gobi21]

Hi Alessandro,

I have managed to get the last day working DB Log and here is the error message i can find:

TNS-12518: TNS:listener could not hand off client connection
 TNS-12547: TNS:lost contact
  TNS-12560: TNS:protocol adapter error
   TNS-00517: Lost contact
    Solaris Error: 32: Broken pipe

[jeffs42885]

Last time i saw a broken pipe issue it was due to a space issue on /var or /usr.

[Gobi21]

May be on that day i was working with CMOD Client.

Today we have more than 50% space still available.

[Alessandro Perucchi]

I would have said open a PMR, but your CMOD is V8.4.1, it is not supported anymore (except if you have an extend support contract)...

difficult :-( to be honest I've never experienced such an error.

And to do an upgrade when CMOD is not running correctly is not a good idea either... :-/

well what you could do... I don't know if it can help or not... but it might worth trying.
In the "System Log" tables in Oracle, you have the tables "SL%", try to find the last one. Let say that the latest SL table that you have is "SL53", then do a select on the table and look at the last 10-20 entries.

With luck you might see something interesting inside, a hint that could point to the correct culprit.

Sincerely yours,
Alessandro

[Gobi21]

Hi All,

I have resolved this issue.. Here is the resolution.

1. when i deleted the application group,the database got refereshed and it was not able to access the database due to wrong password. Five months back i have updated the password for Ondemand database and the old password was referred since we are using only one thread.
2. During the referesh., it was looking for a new thread and the password is wrong since i have not updated the password in config file.
3. Once i have updated the password in cfg file, i was able to start the ondemand without any issues.

Thanks for all your support.

Cheers,
Gobi