I assume everything was working correctly prior to the upgrade. When the VSM reports the SVS connection sync is "in progress" this usually points to some issue on the vCenter server. You mention you "re-built" the VC - I assume you re-imported the extenion key etc again correct? Is everything else on the vCenter working as expected?
A couple things you can try is:
- Check the vCenter task manager for resource utilization Ensure you're not peaking on CPU or memory. Might also want to check the VC logs - there should be some clues in there.
- Restart the "Virtual Center Server" service. Try to connect again.
- Restart the vCenter server completely.
If that doesn't work, let me know and we'll dig deeper.
Yep, everything was working properly prior to the upgrade. It seemed to break when we went to 4.1 update 1. When I upgraded our VSM software to SV1(3a) from SV1(2), sync completed to the vcenter server.
The .xml for the extension key does exist within vcenter, although we're not exactly sure how to "un-register" and "re-register" it.
We've attempted to restart the vcenter server service, and we also restarted the vcenter server itself with no luck.
CPU/memory usage on the server is actually in very good shape right now.
So the 1000v DVS does not exist in vCenter correct?
Normally when trying to "Connect" to the svs connection, if the extension key is not registered it will throw up a warning regarding "... extension key not registered before being used...". I'm a little suspect if it's not showing up in vCenter and we're expecting it to.
If you need to re-import the extension key (assuming you've rebuilt the vCenter from scratch) do the following:
1. Go to http://<vCenter IP>/mob & enter your vCenter administrator credentials.
2. Navigate to CONTENT - EXTENSION MANAGER. Under the properties section you may/may not see an extensionList entry for "Cisco_Nexus _1000v_xxxxxx". If you do - then your VSM extension key is already registered. You can compare this to the value output of "show vmware vc extension-key" from your VSM, which they should match. If they don't match, then you need to remove it and re-register it. If it's not there, you'll need to register it.
3. To remove an extension key, from "https://<vCenter IP>/mob/?moid=ExtensionManager" copy the exisitng extension key shown in the form of "Cisco_Nexus 1000v_xxxxxxxxx" and select the "UnregisterExtension" operation. Paste the extension key from the last step in the box and click "Invoke Method". Once invoked, you can restart the vCenter Server service from windows services and the extension key should no longer show up.
4. To register an extension key to go your VSM's IP address, "Save As" the extension key, and then using the "Plug-in Manager" from the VI Client register the XML extension key. (Full procedure can be found in the VSM install guide).
If this doesn't solve your issue let me know and we'll dig deeper.
We re-registered the extension (it was there, we actually imported the vcenter db) on the server, and we weren't able to get a different result.
I also tried to "no connect" again, and got the following message:
"SVS connection service is busy. Please try again later."
When I try to "no remote ip..." to remove the vcenter server altogether, and re-add it, I see the following message:
N1KV-VSM1(config-svs-conn)# no remote ip address x.x.x.x port 80
ERROR: Cannot change connection configuration in 'Enabled' state.
It seems like the SVS connection on the VSM is completely hung. Not sure. I don't even see it "drop" the connection to the vcenter when the server is rebooted/services shut down.
Just wanted to follow up on this...
We tried to perform a "system switchover" to the secondary, and back to the primary.
When we did, one of our host VEMs started bouncing in and out of the DVS with the error "Removing VEM 5 due to state conflict VSM(ModIns Start Rcvd), VEM(ModIns End Rcvd)".
We ended up rebooting that host completely, and two things happened -
1) vCenter <-> VSM sync completed.
2) The server connected back to the VSM gracefully with no further issues.
Does this make any sense?
We have another issue but I think it's unrelated to this thread. Right now we're just trying to determine why if we move a VM from a 4.0 box to a 4.1 box, it will lose network connectivity after a short period of time.
It's possible you were encountering CSCth01542 where the VSM gets stuck with SVS connection busy
This bug was fixed in the SV1(3b) release.
Further information can be retrieved through the Bug Toolkit:
Thanks for the link. We had a few bugs in 3a we needed to upgrade to address, so we ended up going to 3c.
Currently, we're still experiencing this issue where the connection between VSM/vCenter is sync "in progress". I also see messages of the following nature:
2011 Mar 14 07:58:57 N1KV-VSM1 %VMS-5-VMS_PPM_SYNC_TIME_EXP: Attempt #0 to resync Port-Profile Manager to local vCenter Server cache due to sync timer expiration
2011 Mar 14 07:58:57 N1KV-VSM1 %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server.
2011 Mar 14 07:58:57 N1KV-VSM1 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.32166
2011 Mar 14 08:01:57 N1KV-VSM1 %DAEMON-3-SYSTEM_MSG: <<%PORTPROFILE-3-SYNC_COMPLETE_ERR>> Sync completed with some errors. - portprofile
I do have IP connectivity between my VSM and vCenter, and there are no ports being filtered between the two.
It looks like there is some issue between the component that communicates with the vCenter Server and PPM. Can you send me the output of "show running-config port-profile"? This looks like an issue I've seen before (and is fixed in later releases)
Also it looks like there are a lot of timeouts between the VSM and vCenter Server. You may want to investigate that a bit
Yeah, the timeouts are strange... if you run a ping between the two, even when the issue crops up, there isn't any break in connectivity... so I am not sure what is going on there.
Attached the output you requested.
port-profile.txt 2.8 K
I'm not sure what the outcome of Ryan's issue was. It appeared to be some sort of issue with the communication between the VSM and vCenter Server. As far as I could tell from the logs, the issue appeared to be a non-N1K issue.
I did find this VMware KB article that may be related. The main point of the article is that the port configured on the N1K under the SVS connection (defaults to 80) should be the same as the HTTP port on the vCenter Server (also defaults to 80). Not sure if this was the case with Ryan's problem. Maybe he can comment