Database communication error while login to IMP 10.5

[Mohammed Idris]

CUCM PUB,SUB and IMP PUB are 3 servers in cluster. Dbrelication on PUB and SUB is ok. Can not login to IMP PUB, throws an error "database communication". Restarted PUB and then SUB and then IMP. Services on IMP took long time around 3 hours to become started from starting state except one service, A Cisco DB service which is still in starting state since last 2 days. Checked DNS and NTP reachabilty and functionality and found ok. From RTMT it shows AMC service is down on IMP node. From serviceability on CUCM tried accessing IMP and got error

Connection to the Server Cannot Be Established (Unknown Error)

How to TS further ??

[Aseem Anand]

Hi Mohammed,

Please answer the following:

1. Did you upgrade the IM&P& CUCM from 9.X to 10.5 recently?

2. If not then is it a new setup?

3. Is it for production or lab?

4. Is the IM&P node added as a subscriber in CUCM?

This can happen if the process node table of CUCM does not contain the entries for the SUB. Collect the output of the following commands:

1. Show network cluster from CUCM PUB & IM&P PUb

2. run sql select * from process node from CUCM PUB & IM&P

3. Utils diagnose test from IM&P nodes

4. Utils service list from IM&P nodes

Regards,

Aseem Anand

[Mohammed Idris]

1. Did you upgrade the IM&P& CUCM from 9.X to 10.5 recently?

NO

2. If not then is it a new setup?

no

3. Is it for production or lab?

Production

4. Is the IM&P node added as a subscriber in CUCM?

Don't know. But from logs looks like.

Please find the attached logs and suggest some solution if any. Thanks.

[Aseem Anand]

Hi Mohammed,

I dont see any log file attach. Can you attach the file again and also send me the screenshot of:

System >>> Server 

Aseem Anand

[Mohammed Idris]

Please find the attchments.

[Aseem Anand]

Hi Mohammed,

IM&P is failing to ping its default gateway and to add to that the IM&P is synched with an NTP with a stratum value of 10 whereas it should not be more than 5 as per the SRND. Can you make sure the IM&P can ping its default gateway and the NTP reference which should be ideally CUCM for the IM&P nodes is synched with an NTP reference with value less than 5.

To check the default gateway on the IM&P node run the command " show network eth0" and ensure that the VM is in correct VLAN by checking the port group specified on the VM in ESXi client.

Aseem

[Mohammed Idris]

Thanks for your kind efforts and replies.

IMP server DG gateway is firewall interface and hence not pinging. Its same foe all I mean CUCM, CUC etc.

NTP is PUB server on IMP.

[Aseem Anand]

Hi,

If the ping is blocked then thats fine but the NTP needs to be checked. For IM&P the NTP reference is CUCM and is synched at a higher stratum. Can you check the NTP status on the CUCM, make sure the CUCM is synched with an NTP of stratum less than 5 which is recommended value by Cisco. Once you correct the NTP status on the PUB, restart the NTP on both PUB as well as on IM&P Publisher using the command "utils ntp restart". Also share the install logs of the IM&P with me.

Aseem

[Mohammed Idris]

I have corrected NTP the way you said. Now PUB @stratum 2 and IMP at stratum 3. Still A Cisco DB service is stuck in starting status.

PUB:

admin:utils ntp status
ntpd (pid 6558) is running...

remote refid st t when poll reach delay offset jitter
==============================================================================
*172.16.5.15 .LOCL. 1 u 53 64 1 0.579 0.198 0.076

synchronised to NTP server (172.16.5.15) at stratum 2
time correct to within 189 ms
polling server every 64 s

IMP:

admin:utils ntp status
ntpd (pid 28354) is running...

remote refid st t when poll reach delay offset jitter
==============================================================================
*172.16.5.24 10.66.5.15 2 u 54 64 377 0.483 -0.019 1.380

synchronised to NTP server (172.16.5.24) at stratum 3
time correct to within 18 ms
polling server every 64 s

Current time in UTC is : Thu Oct 6 08:21:15 UTC 2016
Current time in Asia/Muscat is : Thu Oct 6 12:21:15 GST 2016

[Aseem Anand]

Hi Mohammed,

How long the server has been operational? Is it a new installation for which the issue started straight away?

Can you share the install logs along with the event viewer application and system logs?

Aseem

[Mohammed Idris]

I opened TAC case and he found unrecoverable error in sql database. Then by taking root access TAC deleted the server. Now I have to rebuild the server. 

[Aseem Anand]

Also make sure the certificates are not expired on the IM&P using the command " show web-security" if its an existing installation.

There are a couple of bug opened with the same symptoms on version 10.5:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCud67438

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCur10739

Aseem Anand

[Mohammed Idris]

Thanks Aseem. I will check and come back.