Nexus 5548UP Replacement Help

[christopher.wood]

Hello,

I am trying to get ready to replace a 5K switch in our network with multiple 2Ks attached and a VPC connection to another 5K for redundancy. I expect the redundant switch to keep the traffic up and running while I replace the defective 5K, but I'm not sure what the best way to replace it is.

I figure I can't configure the 2K ports at they won't be attached until the night of the cutover. So this is the steps I have in my head.

1) Insert 16 Port Module from defective switch(Didn't receive this with the replacement switch)
2) Connect 2Ks

3) Load Config
4) Connect 5K (redundant switch)
5) Connect 7K (Core switch)

Does that make sense? Is there an easier way to have everything configured ahead of time? Any suggestions?

Thanks.

[Reza Sharifi]

Hi,

As long as your end devises (servers, storage, etc..) are attached to 2 FEXs, you should be fine.  Just make sure to load the same exact image on the new 5500 as the existing one, if not the new 5500 will push a different image to the 2ks and will cause outage, reboot, etc..

HTH

[Paul Chapman · Accepted solution]

Hi -

I would suggest the following procedure:

1. Pre-cutover - During Business hours
1. Check licensing on replacement switch.  If mismatch to existing (or peer if existing is dead), call Cisco to provide correct license files for migration.
2. Upgrade / Downgrade NX-OS to match existing or peer 5K
3. If using NPV, then enable feature and reboot
4. If using native Fiber Channel, assign FC Ports and reboot
5. On the assumption that your N2Ks are dual-homed to the 5Ks, pre-provision the FEX modules using the procedure found HERE.
6. Restore old configuration, including FEX ports. Note any errors reported during the process. T/S errors as needed; any errors related to expansion module will need to be resolved during cutover
7. Administratively shutdown all non-FEX ports
8. Arrange for application testers for the night of your cutover to minimize impact on following business day
2. Cutover
1. Power off defective 5K
2. Install replacement, re-cable, & move expansion module
3. Power up new 5K
4. Restore config pieces that could not be installed due to missing module
5. Enable vPC peer-keepalive link
6. Enable vPC Peer Link
7. Check vPC status and t/s issues as needed
8. Enable N7K ports and verify
9. Enable FEX downlink ports and verify
1. Check "show interface status" for inconsistent ports and resolve
10. Enable ports for other devices and verify
11. If using FC/FCOE, check:
1. FLOGIs from FC devices
2. VSAN mismatches on ports
3. Zoning errors
12. Contact testing teams to verify applications are working as expected.

NOTE: The cutover is likely to cause disruption, particularly for the Nexus 2000 ports (for dual-homed FEXs).  This procedure attempts to minimize, but doesn't guarantee no downtime.  Other unknown factors can have an influence.  It is up to you to be technical enough to perform the work and to have 24x7 SmartNet for all device in case problems do occur.

HTH

PSC

[christopher.wood]

What is Dual Home FEXs? Are they when the 2K goes to both 5Ks? We only connect one 2K to one 5K, not both 5Ks, but the servers are connected to each 5K via the 2Ks. About three 2Ks per 5K.

PS. This is really great info. Thanks!

[Paul Chapman]

Hi -

Yes. Dual home means that the FEX is connected to 2 upstream 5Ks.  Since you're not doing that the config will be easier for you.

PSC

[christopher.wood]

Just wanted to ask one other question. When I start bringing up the FEXs, they'll have no configuration since they get it from the 5K. I can't configure those ports ahead of time since they're not plugged in.

So probably best just to bring up one FEX, configure the interfaces, make sure it's good, and then move onto the next FEX?

The servers with LACP should still only use the redundant side when the interfaces come up since there channel-group won't be configured on the FEXs when they first come up?

Thanks.

[Paul Chapman]

Hi -

In 1.5 I provided a link on how to pre-provision the FEX and pre-install the configuration.  Step 1.7 tells you to shut down all the non-FEX ports (meaning exclude the ports with the "switch mode fex-fabric" command).

Between these 2 you can bring the switch parent switch and FEXs online prior to your server LACP connections, which will give you granular control over the reconnection of the servers.

PSC

[christopher.wood]

Oh, darn. I thought that guide was only if the FEXs were Dual-Homed. I was able to implement the pre-provisions during the cutover before shutting down the defective 5K. On the redundant 5K I made it's VPC priority lower before bringing up VPC Peer Links to make it the Primary while I was turning up the links to the 2Ks.

The Pre-Provisions seemed to work during the upgrade as the 2Ks came up with the configurations with no issues and the connections were up on the two FEXs I managed to bring up.

The VPC Peer Link looked fine when it came up. Then the uplinks to the 7K looked fine. I started to bring up FEXs and then I started getting complaints of servers down after bringing up the second one. Everything looked fine on the VPC Peer Link between the 5Ks.

Not sure if I messed it up by changing the Priority... that seems odd to me that changing the Priority to make the redundant Primary would cause traffic to fail when bringing up the 2Ks on the now Secondary. So I don't feel like that was the issue.

Thinking I'll probably get TAC involved to take a look at our setup, explain my procedure and see if they can tell me where the issue occurred.

Thanks for your help, Paul. It is much appreciated.

[Paul Chapman]

Hi -

Be mindful that vPC is considered a L2 protocol.  A lower priority number means that the switch is more preferred as the vPC primary.

Note that if you don't have "spanning-tree port type edge" or "spanning-tree port type edge trunk" on your server ports, then bringing them online will subject the server to a standard 30-60 second outage due to STP.

Check for STP inconsistent ports with "show spanning-tree".

PSC