Cisco 7911 Phones registered in CUCM 10.5 failed to answer properly to HTTP requests.

oscar.barrofet · ‎05-09-2016

Within the platfom Cisco CUCM 10.5 there is granted out-of-the-box the possibility to send HTTP POST Messages to Cisco IP Phones via IPPS Phone Services API as reflected in the documentation

https://developer.cisco.com/site/ip-phone-services/documents/index.gsp

It is possible to just check if this functionality has any bug just invoking ‘http://<phone ip address>/CGI/Execute’ from any web browser.

It seems that HTTP requestes posted to Phones 7911 registered in CUCM 10.5 don’t work (but registered in 8.6 they do work), complaining about an error <CiscoIPPhoneError Number="4" />, despites the fact that all the configuration needed is done (web access enable and phone associated to application user with all the permissions required).

Full CallManager version:

It works with phones 7911G and CUCM 8.6.2.22900-9

It doesn’t work with phones 7911G and CUCM 10.5.1.10000-7

Detailed problem description:

We have an application that, using IP Phone Services API, sends HTTP POST requests containing <CiscoIPPhoneText> and <CiscoIPPhoneExecute> xml’s towards phones models 7911G in their port 80 ‘http://10.2.200.180/CGI/Execute’, when they are registered in the CUCM 8.6 works correctly.

We are currently migrating those phones to CUCM 10.5 and the same exact request (that worked in CUCUM 8.6) towards 7911G responds (now with the CUCM 10.5) with a <CiscoIPPhoneError Number="4" /> having the user associated with a user with exact same name, password, and roles.

But, with phones model 7861 and exactly the same HTTP requests while they are associated with the very same user respond with a correct code.

The firmware of such 7911 phones is SCCP11.9-2-3S, now in phones 7911 registered in both CUCMs.

Additional information. (What we've tried so far)

The phones have web access: ‘enabled’, and they are associated with an Application user with the phones as Associated devices, and, to make sure, belonging the user to all groups and having all the roles possible.
We checked the Enterprise parameters – URL Authentication and we set IP’s addresses instead of names to avoid DNS issues.
We check the documentation for using Cisco Unified IP Phone Services Application Development Notes, in

https://developer.cisco.com/site/ip-phone-services/documents/index.gsp

we saw there is information up to Release 9.1(1) and Later,

so we presumed is also valid for CUCM 10.5 (not found similar document for 10.5), also as mentioned here:

https://communities.cisco.com/thread/46538?start=0&tstart=0

In the mentioned document states that both containing <CiscoIPPhoneText> and <CiscoIPPhoneExecute> are supported for 7911G, shown below:

--> We also tried to see inside phone logs (comparing both 7911 and 7961) when invoked in the 7911 from the browser says

489: ERR 07:16:21.090946 JVM: StcpActiveSSLConnectionStatus: SUCCESS

490: WRN 07:16:21.270092 JVM: HTTP JNI| incomplete/incorrect CGI request

491: WRN 07:16:21.270828 JVM: HTTP JNI| *** HTTP_CGI_ERROR ***

but when invoked from the application it seems as the 7911 tries to connect using HTTPS

0637 NOT May 09 07:48:55.427969 JAVA-HTTP JNI| httpServerHandler: uri=/CGI/Execute

0638 WRN May 09 07:48:55.431723 JAVA-Thread-11|cip.xml.XmlParser:parse - Encoding Updated to UTF-8

0639 NOT May 09 07:48:55.439018 JAVA-Sec SSL Close Connection successful.

0640 ERR May 09 07:48:55.439598 JAVA-HTTP JNI| processHttpRequest: error process connection 3

531: NOT 07:33:44.849893 SECD: clpConnInfo: SSL/TLS conn info:

532: NOT 07:33:44.851304 SECD: server : ** HTTPS ** 10.2.200.10

533: NOT 07:33:44.851903 SECD: IP tos value : 0

534: NOT 07:33:44.852486 SECD: protocol : TLSv1

535: NOT 07:33:44.853097 SECD: cipher : AES256-SHA (256 of 256 bits)

536: NOT 07:33:44.853710 SECD: client auth : not done (could use MIC)

537: NOT 07:33:44.854360 SECD: SSL session : reused

538: NOT 07:33:44.855006 SECD: clpSetupSsl: SSL/TLS active, <10.2.200.10> c:9 s:10

539: ERR 07:33:44.861237 JVM: StcpActiveSSLConnectionStatus: SUCCESS

-->to solve that, Also we tried to override the enterprise parameter with the parameter in the phone 7911: 'Authentication Server' but with no succes.

--> we also tried to factory hard reset the 7911 phone with no result

Those are they HTTP request/response being sent (the Authorization: Basic token is the same in both cases)

POST request TOMCAT to phone 7911G

Response from phone 7911 a Tomcat PRO2

Man: POST http:// 10.192.136.240/CGI/Execute HTTP/1.1

MessageType: CALL

Content-Type: text/xml

Host: 10.2.200.180:80

Authorization: Basic

XXXXXXXXXXXXXXXXXXXXXXXXX

Content-length: 119

<?xml version="1.0" encoding="UTF-8"?>

</CiscoIPPhoneExecute>

<?xml version="1.0" encoding="UTF-8"?>

We tried to configure the phone with span to pc port: enable and connect a cable from the phone 7911 to the PC and used wireshark trying to see communication between the phone and call manager with no practical result

I attach to this email the capture 7911 span to pc port, being 10.2.200.10 the CUCM 10.5 and 10.2.200.180 the 7911 phone.

We tried to invoke as well

We used to have firmware SCCP11.9-1-1SR1S and then we got <CiscoIPPhoneError Number="0" /> instead, but checking the table we saw the need to upgrade

http://www.cisco.com/c/en/us/td/docs/voice_ip_comm/cucme/requirements/guide/cme105spc.html

and after the upgrade the error went to <CiscoIPPhoneError Number="4" />

Cisco Unified IP Phone7906 and 7911

SCCP11.9-2-1S.loads
apps11.9-2-1TH1-13.sbn
cnu11.9-2-1TH1-13.sbn
cvm11sccp.9-2-1TH1-13.sbn
dsp11.9-2-1TH1-13.sbn
jar11sccp.9-2-1TH1-13.sbn
term06.default.loads
term11.default.loads

SIP11.9-2-1S.loads
apps11.9-2-1TH1-13.sbn
cnu11.9-2-1TH1-13.sbn
cvm11sip.9-2-1TH1-13.sbn
dsp11.9-2-1TH1-13.sbn
jar11sip.9-2-1TH1-13.sbn
term06.default.loads
term11.default.loads

Also we tried with a phone 7911 with even a newer firmware version SCCP41.9-2-3S, also resulting in failure when registered in CUCM 10.5

To summarize:

CUCM version	Firmware 7911		result	CUCM version
CUCM 8.6	SCCP11.9-2-3S	7911	OK	CUCM 8.6
CUCM 10.5	SCCP11.9-2-3S	7911	*NO OK*	CUCM 10.5
CUCM 10.5	SCCP11.9-1-1SR1S	7911	*NO OK*	CUCM 10.5
CUCM 10.5	SCCP11.9-4-2-1S	7911	*NO OK*	CUCM 10.5
CUCM 8.6	SCCP41.9-2-3S	7961	OK	CUCM 8.6
CUCM 10.5	SCCP41.9-2-3S	7961	OK	CUCM 10.5
CUCM 8.6	SCCP42.9-2-3S	7962	OK	CUCM 8.6
CUCM 8.6	SCCP41.9-2-3S	7961	OK	CUCM 8.6
CUCM 8.6	SCCP70.9-2-3S	7971	OK	CUCM 8.6
CUCM 8.6	SCCP75.9-2-3S	7975	OK	CUCM 8.6

Any help will be greatly appreciated.

Kind Regards

dstaudt · ‎05-09-2016

As this looks to be a problem with authenticating the Basic Auth credentials, it will probably be helpful to see what's happening when the phone attempts to verify the credentials: after the POST to the phone (and before the phone replies to the POST), you should see the phone attempt a GET request to the configured Authentication URL with the provided username/password/devicename - CUCM will respond with AUTHORIZED or UNAUTHORIZED.

Assuming you are able to look at the HTTP details of this operation for both working and non-working scenarios, hopefully some difference will become apparent. I don't see the referenced packet capture attached here, you may want to go ahead an open a ticket with DevNet Developer Support for one-on-one support (and log confidentiality:)

https://developer.cisco.com/site/devnet/support/

If you want to attach the pcap here, that's fine - it would be good to get the capture from the phone start-up so as to include the *.cnf.xml configuration file for the phone (this can also be retrieved from your pc via TFTP as [devicename].cnf.xml from the CUCM IP.)

As you mention HTTPS in the logs, you may want to check and see if any of the enterprise or device-specific HTTPS URLs are configured (i.e. the Authentication URL) - I believe there are some non-intuitive rules for how these are used, i.e. phones which support HTTPS will use any configured HTTPS URLs, ignoring HTTP URLs...

oscar.barrofet · ‎05-20-2016

Many thanks for your response. We've tried the AUTHORIZED or UNAUTHORIZED, checked the https url and so on.

But as an update we have found a workaround that works partially!

Instead of associating the phone to an application user we are associating the phone to an end user belonging to the groups Standard CCM End Users, Standard CTI Allow Control of All Devices and Standard CTI Enabled.

The problem originally reported is thereby solved.

Nevertheless we have found a new situation, when we tried to use getIPV4Address (previous to the operation of sending request) from the object CiscoTerminal it says InvalidStateException

1 obtenerIp! InvalidStateException (if CiscoTerminal is not registered) SEPA8B1D4FA6162

com.cisco.jtapi.InvalidStateExceptionImpl: Terminal is not registered

at com.cisco.jtapi.TerminalImpl.getIPV4Address(TerminalImpl.java:2610)

at com.tecnocom.services.TerminalUtils.obtenerIp(TerminalUtils.java:33)

at com.tecnocom.click2call.ActorLlamador$ActorConvocadorThread.run(ActorLlamador.java:79)

Just before that, traces show CiscoTermOutOfServiceEv and CiscoAddrOutOfServiceEv and phone is never sending CiscoTermInServiceEv and CiscoAddrInServiceEv as it effectively does for the models 7961

We are sleeping the main thread to allow the observer to catch the CiscoTermInServiceEv but it never happens. We tried first to wait for the semaphore object to signal the arrival and it is never coming, so to avoid waiting indefinitely we left the sleeping thread solution.

The firmware of the phones are the same. Strangely enough this is happening in the production environment but is not

happening in our lab, so something we are missing.

Regads.

dstaudt · ‎05-24-2016

From what you describe, it sounds very much like the device in question became unregistered from CUCM - i.e. by being powered off, disconnected from the network, etc. - and never came back online during the time you were observing. That would certainly result in the invalid state exception, as CUCM cannot retrieve info (like an IP address) if the device is not registered.

If you know for sure the device was in fact registered and working at the time, then it's possible that some layer involved in the scenario - CUCM, CTI Manager, JTAPI, application - is not getting updated on the re-registration of the device (or even more unlikely, is getting into a state where it thinks the device unregistered when it hasn't). To troubleshoot that will involve looking at detailed logs, for which you will want to open a ticket with DevNet Developer Support: https://developer.cisco.com/site/devnet/support/

oscar.barrofet · ‎06-06-2016

Finally it seems we reach to a conclusion for the problem:

We needed to activate firewall rules between the port 80 of the phones and the port 8433 of the call manager for the ip range of the involved phones.

This firewall rules were requested at the beginning of the project but they were never activated, and nobody checked that.

Also we have 2 different VLAN, in one of those the rules were set. Unfortunately 7911 phones were the only ones tested in the VLAN were the firewall rules were not set. So causing the confusion. We were misleaded to believe the problem was related to the model (7911) instead of the different VLAN.

we found out checking with Q-Radar that the request from the phone to the CUCM asking if the user and password had permissions to post to the phone and this request was killed by the firewall.

goutham.mp · ‎06-17-2016

I found some of the information relevant to what I am trying to achieve. I am also trying to control the IP phone from webpage. I triggered a POST request as shown below. Getting CiscoIPPhoneError 0 response. Could it be a firewall issue? The GET request to fetch the Screenshot is working fine.

Could you please share if you have relevant information? Thanks,

POST /CGI/Execute/ HTTP/1.1

Host: 10.204.45.103

Connection: keep-alive

Content-Length: 104

Authorization: Basic Z292aW5kcnI6MTIzNA==

Accept: */*

Cache-Control: no-cache

Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36

Content-Type: application/x-www-form-urlencoded

Accept-Encoding: gzip, deflate

Accept-Language: en-US,en;q=0.8

XML%3D=%3CCiscoIPPhoneExecute%3E%3CExecuteItem+URL%3D%22Key%3ALine2%22%2F%3E%3C%2FCiscoIPPhoneExecute%3E

goutham.mp · ‎06-17-2016

Found that there is no firewall blocking. So it is unknown issue. kindly help