MRA. SSH tunnel between Expressway C and E not coming up

[arunas-init]

Hi Experts, 

I'm setting up MRA with Expressway C/E according to Cisco Expressway Basic Configuration Deployment Guide X8.9. I did all configuration according this guide, but SSH tunnel cannot come up. The connection between ExpE and ExpC is being setup using certificates.

I had issues with certificates (only server) and solved them (got client/server). I had issues with Local CA certificates, I uploaded them to ExpE and ExpC. 

Actually the problem is that SSH Tunnel is not comming up after all has been setup.

On ExpC the Traversal Zone is OK and in "Active"state but on ExpE  the Traversal zone is in "Failed"state. How this can happen ?

I went through lot of documentation, lot of discussion forums and youtube video eventually how to setup MRA, but cannot find anything that could help me solve this issue in this particular situation. 

CUCM version - 11.5

CUP vertion - 11.5

Expressway version - X8.9

In the network.log i see that ExpE is responding with code 503 - Service unavailable

Network.log

2017-03-16T12:17:18.156+02:00 tvcs: UTCTime="2017-03-16 10:17:18,155" Module="network.sip" Level="DEBUG": Action="Sent" Local-ip="192.168.XXX.1" Local-port="7001" Dst-ip="192.168.YYY.1" Dst-port="25000" Msg-Hash="3428631679229348547"
SIPMSG:
|SIP/2.0 503 Service Unavailable
Via: SIP/2.0/TLS 192.168.YYY.1:5061;branch=z9hG4bK014642fcc8751175bb9abbcb295d859f201;received=192.168.YYY.1;rport=25000
Call-ID: a40e362b0f858a7b@192.168.YYY.1
CSeq: 1312 OPTIONS
From: <sip:192.168.YYY.1>;tag=7a706ddbfc59c2fa
To: <sip:108.108.108.108:7001>;tag=f82518f8ce63b144
Server: TANDBERG/4134 (X8.9)
Content-Length: 0

2017-03-16T12:17:18.156+02:00 tvcs: UTCTime="2017-03-16 10:17:18,155" Module="network.sip" Level="INFO": Action="Sent" Local-ip="192.168.XXX.1" Local-port="7001" Dst-ip="192.168.YYY.1" Dst-port="25000" Detail="Sending Response Code=503, Method=OPTIONS, CSeq=1312, To=sip:108.108.108.108:7001, Call-ID=a40e362b0f858a7b@192.168.YYY.1, From-Tag=7a706ddbfc59c2fa, To-Tag=f82518f8ce63b144, Msg-Hash=3428631679229348547"

2017-03-16T12:17:18.156+02:00 tvcs: UTCTime="2017-03-16 10:17:18,155" Module="network.sip" Level="DEBUG": Action="Received" Local-ip="192.168.XXX.1" Local-port="7001" Src-ip="192.168.YYY.1" Src-port="25000" Msg-Hash="12109475290715483484"

SIPMSG:
|OPTIONS sip:108.108.108.108:7001;transport=tls SIP/2.0
Via: SIP/2.0/TLS 192.168.YYY.1:5061;branch=z9hG4bK014642fcc8751175bb9abbcb295d859f201;rport
Call-ID: a40e362b0f858a7b@192.168.YYY.1
CSeq: 1312 OPTIONS
From: <sip:192.168.YYY.1>;tag=7a706ddbfc59c2fa
To: <sip:108.108.108.108:7001>
Max-Forwards: 0
User-Agent: TANDBERG/4134 (X8.9)
Supported: com.tandberg.vcs.resourceusage
Content-Type: text/xml
Content-Length: 816

<?xml version="1.0" encoding="utf-8"?>
<info>
<resourceusageinfo>
<traversalcallsavailable>300</traversalcallsavailable>
<nontraversalcallsavailable>1500</nontraversalcallsavailable>
<registrationsavailable>0</registrationsavailable>
<turnrelaysavailable>0</turnrelaysavailable>
</resourceusageinfo>
<timestamp>1489659438</timestamp>
<media>
<encryption>
<mode>on</mode>
</encryption>
</media>
<domains>
<domain sip="true" edgesip="true">domain.external</domain>
<domain sip="true" edgesip="false">domain.local</domain>
</domains>
<edge>
<state>data</state>
<subject>MGYxCzAJBgNVBAYTAkxUMQswCQYDVQQIEwJMVDEQMA4GA1UEBxMHVmlsbml1czENMAsGA1UEChMETEtQQjEMMAoGA1UECxMDSVRWMRswGQYDVQQDExJleHBjMS52b2lwLmxrcGIubHQ=</subject>
<zone>
<id>1</id>
</zone>
<domains>
<domain xmpp="1">domain.local</domain>
<domain sip="1">domain.external</domain>
</domains>
</edge>
</info>

Can you give me any clue how to solve this issue please..

Best regards, 

Aron

[devils_advocate]

Hi

Is there a FW between your E and C nodes? If so, check if port 7001 is open. 

If you go into the zone settings, what error does it give you as to why the Zone is not active?

Thanks

[arunas-init]

Hi, 

There is a firewall between E and C nodes. All ports are opened between them for initial setup. Port 7001 opened as well. I'm trying to minimize impact of the firewall for initial setup.

On ExpC in zone settings state is following:

State                    Failed
SIP port                Active
Cause                  System unreachable

I did some debugging on node C as mentioned earlier and node C is receiving SIP requests on port 7001, but for some reason it's sending back "service unavailable 503"

Regards, 

Aron.

[Alok Jaiswal · Accepted solution]

Hi Aron,

It can be possible if the FW between C and E is not configured properly, or there is a NAT involved to communicate C and E (some organizations do that, we call it as double NAT i.e. inside NAT as well as Outside NAT). 

Please attach the log gz file with tcpdump and i can help you to take a look at this.

Regards,

Alok

[arunas-init]

Hi, 

Thank you all for guiding me towards problem resolution. In my case there was a typo error. This particular installation is using Static NAT solution. Within Traversal Zone configuration on ExpE node NAT address was incorrect, it means there was a typo error. The NAT IP address on node E was different from one which is resolved by internal DNS on node C. This was interesting, because node C says that IP is reachable and all is good even though there is different IP address configured on node E. So this confused me totally. The problem was spotted by Cisco TAC nicely and quickly. 

Cheers,

Aron.