1 2 Previous Next 15 Replies Latest reply: Mar 20, 2012 10:44 AM by ryan.lambert RSS

SVS connection sync status stuck "in progress"

ryan.lambert

Hi all,

 

Not sure if anyone has seen this. We have just recently upgraded from SV1(2) to SV1(3a) in order to support 4.1. We also rebuilt our vcenter server, and seem to be having a strange issue where the sync status is stuck "in progress".

 

N1KV-VSM1(config-svs-conn)# sh svs connections

connection vcenter:
    ip address: x.x.x.x
    remote port: 80
    protocol: vmware-vim https
    certificate: default
    datacenter name: Datacenter
    DVS uuid: 22 ea 3a 50 ed 7a 7e 6e-43 f8 c2 f8 3c 06 97 62

    config status: Enabled
    operational status: Connected
    sync status: in progress
    version: VMware vCenter Server 4.1.0 build-258902

 

If I attempt to perform a "no connect" and "connect", I see the following error -

 

N1KV-VSM1(config-svs-conn)# no connect

SVS connection service is busy. Please try again later.

Has anyone experienced this? Resolutions?

  • 1. Re: SVS connection sync status stuck "in progress"
    roberbur

    Hey Ryan,

     

    I assume everything was working correctly prior to the upgrade.  When the VSM reports the SVS connection sync is "in progress" this usually points to some issue on the vCenter server.  You mention you "re-built" the VC - I assume you re-imported the extenion key etc again correct? Is everything else on the vCenter working as expected?

     

    A couple things you can try is:

    - Check the vCenter task manager for resource utilization  Ensure you're not peaking on CPU or memory. Might also want to check the VC logs - there should be some clues in there.

    - Restart the "Virtual Center Server" service.  Try to connect again.

    - Restart the vCenter server completely.

     

    If that doesn't work, let me know and we'll dig deeper.

     

    Regards,

     

    Robert

  • 2. Re: SVS connection sync status stuck "in progress"
    ryan.lambert

    Hey Robert,

     

    Yep, everything was working properly prior to the upgrade. It seemed to break when we went to 4.1 update 1. When I upgraded our VSM software to SV1(3a) from SV1(2), sync completed to the vcenter server.

     

    The .xml for the extension key does exist within vcenter, although we're not exactly sure how to "un-register" and "re-register" it.

     

    We've attempted to restart the vcenter server service, and we also restarted the vcenter server itself with no luck.

     

    CPU/memory usage on the server is actually in very good shape right now.

     

    Thanks!

  • 3. Re: SVS connection sync status stuck "in progress"
    roberbur

    Ryan,

     

    So the 1000v DVS does not exist in vCenter correct?

     

    Normally when trying to "Connect" to the svs connection, if the extension key is not registered it will throw up a warning regarding "... extension key not registered before being used...".  I'm a little suspect if it's not showing up in vCenter and we're expecting it to.

     

    If you need to re-import the extension key (assuming you've rebuilt the vCenter from scratch) do the following:

     

    1. Go to http://<vCenter IP>/mob & enter your vCenter administrator credentials.

     

    2.  Navigate to CONTENT - EXTENSION MANAGER.  Under the properties section you may/may not see an extensionList entry for "Cisco_Nexus _1000v_xxxxxx".  If you do - then your VSM extension key is already registered.  You can compare this to the value output of "show vmware vc extension-key" from your VSM, which they should match.  If they don't match, then you need to remove it and re-register it.  If it's not there, you'll need to register it.

     

    3. To remove an extension key, from "https://<vCenter IP>/mob/?moid=ExtensionManager" copy the exisitng extension key shown in the form of "Cisco_Nexus 1000v_xxxxxxxxx" and select the "UnregisterExtension" operation.  Paste the extension key from the last step in the box and click "Invoke Method".  Once invoked, you can restart the vCenter Server service from windows services and the extension key should no longer show up.

     

    4. To register an extension key to go your VSM's IP address, "Save As" the extension key, and then using the "Plug-in Manager" from the VI Client register the XML extension key.  (Full procedure can be found in the VSM install guide).

     

    If this doesn't solve your issue let me know and we'll dig deeper.

     

    Regards,

     

    Robert

  • 4. Re: SVS connection sync status stuck &quot;in progress&quot;
    ryan.lambert

    Hi Robert,

     

    We re-registered the extension (it was there, we actually imported the vcenter db) on the server, and we weren't able to get a different result.

     

    I also tried to "no connect" again, and got the following message:

     

    "SVS connection service is busy. Please try again later."

     

    When I try to "no remote ip..." to remove the vcenter server altogether, and re-add it, I see the following message:

     

    N1KV-VSM1(config-svs-conn)# no remote ip address x.x.x.x port 80

    ERROR: Cannot change connection configuration in 'Enabled' state.

     

    It seems like the SVS connection on the VSM is completely hung. Not sure. I don't even see it "drop" the connection to the vcenter when the server is rebooted/services shut down.

  • 5. Re: SVS connection sync status stuck &quot;in progress&quot;
    ryan.lambert

    Just wanted to follow up on this...

     

    We tried to perform a "system switchover" to the secondary, and back to the primary.

     

    When we did, one of our host VEMs started bouncing in and out of the DVS with the error "Removing VEM 5 due to state conflict VSM(ModIns Start Rcvd), VEM(ModIns End Rcvd)".

     

    We ended up rebooting that host completely, and two things happened -

     

    1) vCenter <-> VSM sync completed.

    2) The server connected back to the VSM gracefully with no further issues.

     

    Does this make any sense?

     

    We have another issue but I think it's unrelated to this thread. Right now we're just trying to determine why if we move a VM from a 4.0 box to a 4.1 box, it will lose network connectivity after a short period of time.

  • 6. Re: SVS connection sync status stuck &quot;in progress&quot;
    sachthak

    Hi Ryan,

     

    It's possible you were encountering CSCth01542 where the VSM gets stuck with SVS connection busy

     

    This bug was fixed in the SV1(3b) release.

     

    Further information can be retrieved through the Bug Toolkit:

    http://tools.cisco.com/Support/BugToolKit/action.do?hdnAction=searchBugs

     

    Thanks,

    Sachin

  • 7. Re: SVS connection sync status stuck &quot;in progress&quot;
    ryan.lambert

    Hi Sachin,

     

    Thanks for the link. We had a few bugs in 3a we needed to upgrade to address, so we ended up going to 3c.

     

    Currently, we're still experiencing this issue where the connection between VSM/vCenter is sync "in progress". I also see messages of the following nature:

     

    2011 Mar 14 07:58:57 N1KV-VSM1 %VMS-5-VMS_PPM_SYNC_TIME_EXP: Attempt #0 to resync Port-Profile Manager to local vCenter Server cache due to sync timer expiration
    2011 Mar 14 07:58:57 N1KV-VSM1 %MSP-5-DOMAIN_CFG_SYNC_DONE: Domain config successfully pushed to the management server.
    2011 Mar 14 07:58:57 N1KV-VSM1 %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by root on vsh.32166

    2011 Mar 14 08:01:57 N1KV-VSM1  %DAEMON-3-SYSTEM_MSG: <<%PORTPROFILE-3-SYNC_COMPLETE_ERR>> Sync completed with some errors. - portprofile[4238]

     

     

    I do have IP connectivity between my VSM and vCenter, and there are no ports being filtered between the two.

  • 8. Re: SVS connection sync status stuck &quot;in progress&quot;
    sbacheld

    Hey Ryan,

     

    Can you provide the output of "show vms internal errors" and "show port-profile internal errors"? Also, can you verify if there are portgroup operations taking place on the VC when the sync operation is "in progress"?

     

    Thanks,

    Sean

  • 9. Re: SVS connection sync status stuck &quot;in progress&quot;
    ryan.lambert

    Hey Sean,

     

    I attached the output of that command for you. At least from what I can tell it looks like we have a port profile sync issue at a minimum.

  • 10. Re: SVS connection sync status stuck &quot;in progress&quot;
    sbacheld

    It looks like there is some issue between the component that communicates with the vCenter Server and PPM. Can you send me the output of "show running-config port-profile"? This looks like an issue I've seen before (and is fixed in later releases)

     

    Also it looks like there are a lot of timeouts between the VSM and vCenter Server. You may want to investigate that a bit

     

    Thanks,

    Sean

  • 11. Re: SVS connection sync status stuck &quot;in progress&quot;
    ryan.lambert

    Yeah, the timeouts are strange... if you run a ping between the two, even when the issue crops up, there isn't any break in connectivity... so I am not sure what is going on there.

     

    Attached the output you requested.

  • 12. Re: SVS connection sync status stuck &quot;in progress&quot;
    sbacheld

    Hey Ryan,

     

    This actually appears to be a different issue. It looks like the the sync between the VSM and the VC is taking too long. I'm suspecting the timeouts may be playing some role here. You'll want to investigate along those lines

     

    Thanks,

    Sean

  • 13. Re: SVS connection sync status stuck "in progress"
    aramdin1980

    We're running SV1(4a) and are running into the same issue. Is this problem solved?

  • 14. Re: SVS connection sync status stuck "in progress"
    sbacheld

    Hey Ashwin,

     

    I'm not sure what the outcome of Ryan's issue was. It appeared to be some sort of issue with the communication between the VSM and vCenter Server. As far as I could tell from the logs, the issue appeared to be a non-N1K issue.

     

    I did find this VMware KB article that may be related. The main point of the article is that the port configured on the N1K under the SVS connection (defaults to 80) should be the same as the HTTP port on the vCenter Server (also defaults to 80). Not sure if this was the case with Ryan's problem. Maybe he can comment

     

    Thanks,

    Sean

1 2 Previous Next