REST API sometimes returns 401 Unauthorized

[ed_rs]

We have Cisco Prime 3.0.  When my Java client code is making the following REST API calls it sometimes returns 401 from the http request.

Requesting Device data:

/webacs/api/v1/data/Devices?productFamily=notStartsWith%28%22Third+Party%22%29&.full=true&softwareType=%22ASA%22&.maxResults=1000&.firstResult=0

Requesting the unsanitized device config:

/webacs/api/v1/op/configArchiveService/extractUnsanitizedFile?fileId=223224

I have added the "Connection:close" to the http request header as mentioned in the FAQ, but still getting the 401 error code.

https://developer.cisco.com/media/prime-infrastructure-api-reference-v3-0/192.168.115.187/webacs/api/v1/index4fcb.html?id=faq#persistent-connections

Is there another setting I can change?  I also have the following in my nbi.properties file on the Cisco Prime server:

nbi.rateLimit.totalRequestThreshold=20

nbi.rateLimit.perUserThreshold=-1

nbi.rateLimit.maxConcurrentQueries=20

nbi.rateLimit.maxAllUserConcurrentQueries=45

nbi.rateLimit.limitPageSize=1000

Thanks,

Eddy

[Spencer Zier]

So, there are three general reasons for 401 errors.

1. The credentials are invalid.  This is probably not what you're running into, but I wanted to include it for the sake of completeness.
2. The concurrent session limit is reached.  If you make five requests with persistent connections in a short period of time, the sixth and subsequent requests will yield 401 errors.  In Prime Infrastructure 3.1 and 3.2, we changed the way the session manager handles persistent connections for API requests, however, this limit can still be hit when more than five sessions are active at once for a given user.
3. The external/third-party AAA server (ie TACACS) is not responding quickly enough.  As is noted in the FAQ page in 3.1+, increasing the timeout and retry count configured in Prime Infrastructure for the AAA provider will help.  I also spent some time optimizing the interfacing between Prime Infrastructure and TACACS to reduce these errors in 3.1, but a slow or unresponsive TACACS server will still cause us to throw back 401 responses.

[ed_rs]

Hi Spencer,

Regarding concurrent session limit reach.  Is it possible to increase from the 5 active session limit at once for a given user in Cisco 3.0 and 3.1?

I am doing a netstat -an | grep <java client ip address> on the Cisco Prime server to see all the TCP connections states for my Java client making HTTP REST API calls to the Cisco Prime server.  Is this a good way to determine how many concurrent sessions for a user are being used between my Java Client and Cisco Prime?  In my tests with my Java client I am using the same user.

When making a HTTP REST API call in my Java client, is a new session created each time since I have to pass the username and password in the header of the HTTP GET request each time?  Or is the new session only created when adding the connection:close to the HTTP header?  If the connection:close is not issued then I would have to wait for the server to time out the connection which could result in a large number of sessions opened?  I am also using the HttpClient class to make my HTTP GET requests in my Java client.   Is my understanding correct?

I am not clear of the definition of a session.  Are sessions and the TCP connection viewed in the netstat -an command considered the same?

Thanks for all your help!! 

[Spencer Zier]

Starting in 3.2, we added support to configure the concurrent session limit via the UI.  Before that point, it was fixed at five.  To configure, go to Administration > Settings > System Settings, then General > Server.  You'll need to restart Prime Infrastructure for the change to take effect.

Yes, using netstat is generally a good approach.  But there are some caveats:

1. Unless you're using 3.1+, if you send the "Connection: Keep-Alive" header in your request, even if you close the TCP connection, Prime Infrastructure will continue to keep the session alive.
2. The TCP connection may be active but no session exists.  This generally occurs (1) for the small window of time that exists after the TCP handshaking but before HTTP traffic is sent and (2) if you have your HTTP client setup to use persistent connections.
3. Like you already mentioned, that one foreign address may be sending traffic for multiple users (which each have their own session limit) or multiple foreign IPs may be sending traffic for the same user.  This is something you can control though.

Yes, a new session is created upon the receipt of each HTTP request, and (in 3.1+) torn down after the request has been handled.  The TCP connection is not really the same as a Prime Infrastructure session, but because the API uses basic authentication, if you send "Connection: close" or just ensure that you gracefully close the TCP connection after every request, then they will be close to the same.

I hope that clarifies things

[ed_rs]

Regarding reason 2 for the 401 error:

The concurrent session limit is reached.  If you make five requests with persistent connections in a short period of time, the sixth and subsequent requests will yield 401 errors.  In Prime Infrastructure 3.1 and 3.2, we changed the way the session manager handles persistent connections for API requests, however, this limit can still be hit when more than five sessions are active at once for a given user.

When you mention five requests with persistent connections in a short period of time:

1. What is the short period of time interval
2. Can it be set or is it hard coded

What was the change made to the session manager in handling persistent connections for API requests in Cisco Prime 3.1 and 3.2?

When you mentioned more than fire sessions are active at once for a given user, does that mean when I run a netstat -an | grep <java client ip address> on the Cisco Prime server, the ESTABLISHED state is counted against the active count  and the CLOSE_WAIT state does not count against the active session count? I am including a Connection:Close in the header with every HTTP API request I make from my Java client to Cisco Prime.  Trying to understand why I am still getting the 401 error.

Thanks,

Eddy

[Spencer Zier]

The period of time is the session duration (which, if I recall correctly, defaults at 60 minutes).  The session duration is configurable, however lowering it too low may have a negative impact on the UI user experience.

The change in 3.1 was to kill the session after we finished preparing the response to the API query.

Regarding the relation between TCP connections and sessions, this method of estimating the sessions is most accurate when persistent connections are not in use.  A connection in CLOSE_WAIT, in 3.1+, is unlikely have an active session associated with it.