Cisco ISE, FEEDAUTODOWNLOAD updated Cisco-Device Profiling Policy

[jyla]

Wonder if anyone else experienced this strange thing 2 days ago.

Suddenly ISE decided to perform CoA on a lot of endpoints, some of them several times, some only once.
All the endpoints were Cisco accesspoints, which in our setup results in a port-bounce, so the AP reloads and users experienced downtime.

The reason given in the authentication log is this:

"CoAReason Change in endpoint identity group/policy/logical profile which are used in authorization policies"

I really struggled to find the reason for this, and provided all the data I can for Cisco TAC to investigated, but I just stumbled upon the following update of the Profiling Policy in the application console log ( show logging application console.log.2024-01-31.1)

2024-01-31 08:07:57,595 INFO [FEEDAUTODOWNLOAD][] SystemConsole -::::- PolicyExporter: Policy ID - 0eba7c50-8c00-11e6-996c-525400b48521 & EP object - EndpointPolicy[id=0eba7c50-8c00-11e6-996c-525400b48521,name=Cisco-Device]
FullName:Cisco-Device
Description:Generic policy for all Cisco devices
Version:2
MinimumCertaintyMetric:10
ActionId:
ScanActionId:fe71c8d0-8bff-11e6-996c-525400b48521
ParentId:null
Enabled:true
HasIdentityGrp:false
IdentityGrpID:
PolicyRules:{Cisco-DeviceRule3=10, Cisco-DeviceRule2=10, Cisco-DHCP-Class-Id-Rule=10, Cisco-DeviceRule-SNMP=10, Cisco-DeviceRule4=5, Cisco-CDP-Platform-Rule=10, Cisco-DeviceRule1=10, Cisco-DeviceRule7=10, Cisco-Meraki-DeviceRule1=10, Cisco-DeviceRule9=5, Cisco-DeviceRule1_SCAN=-3, Cisco-DeviceRule8=10, Cisco-DeviceRule-LLDP=10, Cisco-TelePresence-CDP-Platform-Rule=10, NMAP-Cisco-Rule=10}
PolicyType:CISCOPROVIDED

This looks like the profiler feed for the built-in Profiling Policy "Cisco-Device", which is the parent policy for all cisco devices being profiled in ISE, was updated.

Strange that the update of the Cisco-Device profiling policy is not mentioned in the "Change Configuration Audit" log, it only mentions the creation of a new profiling policy "Cisco-Collaboration-Device" located under "Cisco-Device" and with 4 policies below it:

Could this has been the cause for the CoA's ?

And did any of you see the same update ?

[Aref Alsouqi]

I would try to generate the profile changes report, it should be called "Endpoint Profile Changes" or similar. That would give a couple of pie charts one for the profiles state before any change, and the other one is for any changes that happened after the last profiler update. Also, if I remember correctly it will also give you a table with all the changes that happened. Maybe you can check in there and if you still see inconsistent results I would think maybe it is a bug.

[jyla]

Thanks for the feedback.  I did check the profiling change report, which shows that many accesspoint were indeed re-profiled during that day, but all ended up with the same profile in all instances (so no changes).

I look forward to my session with TAC monday, and will report back if anything is found.

[Aref Alsouqi]

One strange thing though is that you mentioned the ports have been bounced, this shouldn't be the default setting on the profile policies.

[jyla]

No you are correct. At some point, when implementing profiling, someone decided to make port-bounce the default setting, since they were struggling with some ip phones which wouldn't fetch a new ip address from DHCP after being profiled and moved to a separate vlan. So the portbounce forces them to reload and fetch an ip address correctly.

Unfortunately no one thought this would also affect anything else. And normally it wouldn't, since once profiled everything is fine.

I have now changed the profiling policy for our accesspoints to not use the global setting (port-bounce) and instead just do a reauth, which works fine.

[Ruben Cocheno]

@jyla 

i personally disable the auto-download of those due previous issues/

[chalkspray]

We just experienced something similar today, but it looks like the last feed update occurred at 1am UTC on 2/1 (our update is scheduled for 7pm local time nightly). All our older video codecs were re-profiled as "Cisco-Collaboration-Device", and since our policy was using another profile type as part of the authorization process, these devices were denied from accessing the network. We don't bounce ports automatically after a change of authorization, however.

Like Ruben mentioned, I'm contemplating disabling the auto update for the feed service as well, and looking for alternatives. I wish we could schedule the day of the week at a minimum, rather than being forced into daily updates if we want it automated. I didn't have the feed download notification enabled before, so I've done that now and we'll see what that includes, but I'm guessing it's not going to be very helpful (such as a change report). I'm going to look into the report that was mentioned by Aref.

[Aref Alsouqi]

You can still undo the update from within the profiler page. The option should be at the bottom left of the page if I remember correctly.

[chalkspray]

Thanks, good to know for the future. In my case, once I realized the issue I just quickly modified the policy to include the new device profile type in addition to the old.

[jyla · Accepted solution]

After a long session with TAC we ended up with conclusions/actions:

- We have changed the behaviour of profiling from port-bounce to re-auth, to avoid AP's and ip phones being reloaded in case of re-profiling.

- We have disabled automated profiler feed updates. This is because that in case new profiles are received, all devices COULD be re-profiled - and we want to be in control with that process.

- Using automated feed updates during daytime is a bad idea anyway, since you could have devices being reprofiled and not allowed (as @chalkspray also experienced)

The only thing Cisco TAC couldn't explain was why some endpoints (AP's) were re-profiled more than once - but since the issue could not be re-produced, we decided to end the investigation here.