ACI Opflex Objects - K8

[simon.birtles]

I am having some inconsistent results with the APIC UI Kubernetes Nodes output (Virtual Networking/Container Domains/Kubernetes/<domain>/Nodes). For example I have a 3 node K8 cluster (1xmaster,2xworker) integrated with ACI. I do not always get all the cluster nodes shown in the UI, its almost random which ones show but never all. So for example, one worker node only, and after an APIC & K8 node restart I have 1xmaster and 1xworker. 

What I have found is that the UI API queries the class 'opflexODev' where the 'opflexODev.IsSecondary' string attribute is NOT "false". The ones set to "true" are shown in the UI. All of the 3 objects for the K8 cluster are created and I can see these with a REST query of https://{{APIC}}/api/node/class/opflexODev.json?query-target-filter=eq(opflexODev.domName,"xxxxx") and have all three objects related to K8 nodes returned each with a IsSecondary attribute, some with value "true" (not shown in ui) and some with "false" (shown in ui).

Having looked at the MIM class reference there is no comment about the meaning/function/source of data for the IsSecondary attribute so difficult to figure out the issue (if indeed there is one ?) - I would expect all K8 nodes to appear in the K8 domain nodes UI section and therefore IsSecondary attribute is "false" for all K8 nodes.

Any understanding of what this attribute "opflexODev.IsSecondary" represents and what the source of this is from ?

ACI 4.2(6h)

K8 1.20

Docker 18.06

CentOS 7

Thanks.

[simon.birtles]

Resolved.

[Robert Burns]

What was the issue/fix?  Helps the community to close the loop.

Robert

[simon.birtles · Accepted solution]

Sure - Was assisting a friend with a customer building some K8 clusters hence not having the time - priorities :), but here goes.. 

So the short story is the problem was mac-pinning in use vs LACP/Active on 'some' of the hosts

If anyone is interested in the long version.....

I found this (as it was understood LACP had been configured) looking at the results from a REST query (/api/node/class/opflexODev.json?query-target-filter=eq(opflexODev.domName,"xxxx")) as stated above, where some nodes were showing a secondary (IsSecondary=true and some IsSecondary=false). Each node had a single entry in the REST results although a dual connected host should show two entries per host. The opflexODev.fabricPathDn attribute was (eventually) a give away as it showed for each host the fabric path based on a single port (node/port) not a VPC IPG name with both vpc switch nodes in the path. i.e. (expected would be) "fabricPathDn": "topology/pod-1/protpaths-211-212/pathep-[VPC_10G_PL_......]. 

The hosts that were showing in the UI (with IsSecondary=false) were being seen by the fabric via the primary switch in the VPC pair only and (you guessed it), the nodes with IsSecondary=true were being seen by the fabric via the second switch in the VPC pair only!

So took a look the IPG setups and sure enough on the IPG (mac-pinning) and the VMM VMWare Domain (mac-pinning). Changed these to be LACP/Active (ip hash VMWare DVS side) and all hosts were populated in the APIC K8 Domain UI with 2 entries per host (1xIsSecondary=false,1xIsSecondary=true) back from the REST query above.