No connection to site message on TBA

Albert Wilhelm · ‎04-11-2010

Hello,

I am getting a red highlight around the TBA for customer 1083. The logs indicate the TBA is not able to communicate back to the site - I am assuming the Thunderbolt portal.

Here is the event message:

Event ID 343540

Event Type SERV.CPE.CONN

SERV.CPE.CONN 2010-04-07 01:06:49.151499-05

Event Description CPE Connection, server detected

Event Message Site Comms down: 99.49.39.196

Customer Id 1083.ciscovar.com

Device ID 00:1A:70:8E:25:23

CONN, CLOSE or HEARTBEAT CLOSED

UP or DOWN DOWN

Is the TBA telling me it can not communicate with the "mother ship"? :-)

I have had this error for the last few days, but the TBA was fine prior to that. Is there a way to find out if the TBA is communicating or know to the portal as offline?

Thank you.

Bert

Marcos Hernandez · ‎04-12-2010

Hi Bert,

Was there any change in your network that could be blocking traffic to the Portal? What is the status of the appliance itself? What is the status of the LED's? Can you find it and browse to its administration page?

Thanks,

Marcos

Albert Wilhelm · ‎04-12-2010

Marcus,

Thank you for your response.

1. There was no change to the network that would prevent the TBA from accessing the internet.

2. The portal indicates the TBA with a red highlight indicating down with the message on my first post.

3. The following LEDs are on and green: Power, Ethernet (flashing) and USB

4. Just tried to access the TBA's web GUI and got this error message:

Fatal error: Uncaught exception 'Exception' with message 'Unable to read configuration file' in /usr/local/htdocs/admin/classes/config.inc:39 Stack trace: #0 /usr/local/htdocs/admin/classes/config.inc(29): config->__construct() #1 /usr/local/htdocs/admin/pages/login.inc(31): config::singleton() #2 /usr/local/htdocs/admin/classes/page.inc(96): login->preprocess() #3 /usr/local/htdocs/admin/index.php(86): page->render() #4 {main} thrown in /usr/local/htdocs/admin/classes/config.inc on line 39

5. I unplugged the Ethernet cable to power cycle the TBA. The diags LED comes on and then the LEDs in #3 stay on. When I http to TBA, I still get the error message in #4.

6. Pinging the TBA works and responses.

Are you able to see the TBA? Do I need a new one?

Thank you.

Bert

Marcos Hernandez · ‎04-12-2010

I think we will have to RMA you a new one. Let me confirm and I will get back to you ASAP.

Thanks,

Marcos

Marcos Hernandez · ‎04-12-2010

Bert, can please you try a factory default and reactivation?

Albert Wilhelm · ‎04-22-2010

Hello Marcus,

I performed the steps you requested to reset the TBA to factory default.

When the TBA resets after I released the Reset button, the Diags LED keeps flashing and I am not able to access the unit via the browser.

Do we need to RMA the unit?

Thank you for your support.

Bert

Marcos Hernandez · ‎04-23-2010

Hi Bert,

We will process an RMA. I apologize for the inconvenience.

Thanks,

Marcos

Michael Holloway · ‎04-23-2010

The flashing diag light may indicate that the USB key isn't properly seated. We've seen this happen quite a bit with this case, you might just try giving it a hard push. Should take about 10 minutes after that before the unit finishes pulling down a factory image and is becomes accessible via the web.

-mike

Albert Wilhelm · ‎04-23-2010

Hello Mike,

I did re-seat the flash card and reset the TBA. Unfortunantely, the TBA Diags LED is flashing and it has all night. So, I think this unit is toast. :-(

Albert Wilhelm · ‎05-26-2010

Just wanted to update this thread. The problem was in the TBA. Marcos sent me a new one and it works.

Problem solved. Thanks Marcos.

Bert

Michael Holloway · ‎05-26-2010

Bert,

Thanks for the update.

-mike

netguyz08 · ‎05-26-2010

I actually had my two TBA's go offline, one had done it a few days ago, and I noticed because I wasn't getting reports lately. Another site went down at the beginning of May. Powercycling both of them brought them up and they got firmware 13.100 and are working again.

Michael Holloway · ‎05-26-2010

Hi Edward, this doesn't sounds like an issue we've seen before. Can you provide the customer numbers so that we can take a look through our logs? What were the symptoms you noticed on the portal before reseting the units?

I'm wondering if it's related to the issue at https://supportforums.cisco.com/message/3107704#3107704, with the customer numbers of the sites we should be able to tell.

-mike

netguyz08 · ‎05-26-2010

Mike,

That other forum post is a possibility. I didn't see what firmware they had before, but power-cycling corrected it since they were able to go out and grab 13.100 and are now reporting that they are online again.

The ID's are 1080 and 1086 so you can take a look.

Both sides had similar messages before going down:

Severity	Warning
Type	SERV.CPE.CONN
Message	Site Comms down: xxx.xxx.xxx.xxx
Device ID	xx:xx:xx:xx:xx:xx
CONN, CLOSE or HEARTBEAT	HEARTBEAT
UP or DOWN	DOWN

Michael Holloway · ‎05-27-2010

Edward,

Looks like an unrelated issue to that other forum post. I don't believe there is any forensic evidence for us to look at for these 2 sites, other than the loss of heartbeat event that the portal detected when they each went offline. The fact that they upgraded themselves and restored their connection to the portal after a power cycle seems to indicate that the cause was with the TBA itself, but I can't say what might have locked them up in this manner, we haven't observed this failure mode before. These are effectively linux boxes, and could be susceptible to locking up due to a power spike. Are these devices being fed by clean UPS power?

-mike