Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Create a new article
1588
Views
6
Helpful
0
Comments

UCSD - Glass House new LAB Build out (ks.cfg example) (Parallel ESXi Deployment)(Pure Disk Create)(SSH MDS example)

Orf Gelbrich
Cisco Employee
Cisco Employee

‎09-05-2017 04:46 AM - edited ‎03-01-2019 06:47 AM

Task at hand is to stand up a new Lab with UCSD

     - Install 32 UCSM blades with ESXi

     - Create boot LUNs on small Netapp

     - Create Datastores on small Netapp

     - Create Datastores on big Netapp

     - Create UCSD layout for the Winter team (Sites/POD's/VDC's)

The resulting service / catalog offering

Screen Shot 2017-09-05 at 6.36.05 AM.png

The results in vCenter

Screen Shot 2017-09-05 at 6.36.54 AM.png

Screen Shot 2017-09-05 at 6.37.29 AM.png

The Sites and PODs in UCSD (Common)

Screen Shot 2017-09-05 at 6.37.57 AM.png

Screen Shot 2017-09-05 at 6.38.08 AM.png

The Sites and PODs in UCSD (Winter Team)

Screen Shot 2017-09-05 at 6.38.34 AM.png

The Sites and PODs in UCSD (DNA Center)

Screen Shot 2017-09-05 at 6.39.08 AM.png

Workflow to create initial initiators and iGroup for big Netapp datastore (8 ESXi hosts):

Screen Shot 2017-09-02 at 6.24.06 AM.png

Screen Shot 2017-09-02 at 6.34.21 AM.png

Work in progress...

The workflows in the workflow file:

Screen Shot 2017-09-05 at 6.50.50 AM.png

The export (15 Files)

Screen Shot 2017-09-05 at 6.52.19 AM.png

The custom tasks

Screen Shot 2017-09-05 at 6.52.50 AM.png

My Kickstart ks.cfg file:

Rusty ks.cfg file updates

=========================

cd /opt/cnsaroot/templates/ESXi65UCSBLADES

vi ks.cfg

File has escape characters so that UCSD does not touch some of the script variables.

[root@UCSD65BMA ESXi65UCSBLADES]# cat ks.cfg

vmaccepteula

rootpw $PXE_PASSWD

#for Local boot

install --firstdisk --overwritevmfs

#for SAN boot

#install --firstdisk=remote --overwritevmfs

$PXE_DHCP_IP_TYPE network --bootproto=dhcp --vlanid=$PXE_MGMTVLAN

$PXE_STATIC_IP_TYPE network --bootproto=static --ip=$PXE_IP --netmask=$PXE_NETMASK --gateway=$PXE_GATEWAY --vlanid=$PXE_MGMTVLAN  --nameserver=$PXE_NS --hostname=$PXE_NAME

#we poweroff ourselves in the 'post' section, and ucs will later power us on after changing the service profile

#reboot

%pre --interpreter=busybox

hwclock -d %LIVE_VAR_DATE_1% -t %LIVE_VAR_TIME_UTC_1%

date -s %LIVE_VAR_DATE_TIME_UTC_1%

cd /tmp

%firstboot --interpreter=busybox

cd /tmp

esxcfg-vswitch -A 'VM Network' vSwitch0

###############################

# enable & start remote ESXi Shell  (SSH)

###############################

vim-cmd hostsvc/enable_ssh

vim-cmd hostsvc/start_ssh

###############################

# enable & start ESXi Shell (TSM)

###############################

vim-cmd hostsvc/enable_esx_shell

vim-cmd hostsvc/start_esx_shell

$PXE_DHCP_IP_TYPE esxcli system hostname set --host=$PXE_NAME

%firstboot --interpreter=busybox

cd /tmp

#Orf added for Rusty

esxcli system settings advanced set -o /UserVars/SuppressShellWarning -i 1

esxcli network vswitch standard portgroup add --portgroup-name VMNetwork32 --vswitch-name vSwitch0

esxcli network vswitch standard portgroup set --portgroup-name VMNetwork32 --vlan-id 32

esxcli network vswitch standard policy security set --allow-forged-transmits yes --allow-mac-change no --allow-promiscuous yes --vswitch-name vSwitch0

esxcli network vswitch standard policy shaping set --enabled yes --avg-bandwidth 100000 --peak-bandwidth 100000 --burst-size 819200 --vswitch-name vSwitch0

esxcli network vswitch standard uplink add --uplink-name vmnic1 --vswitch-name vSwitch0

#esxcli network vswitch standard policy failover set --active-uplinks vmnic0 x--standby-uplinks vmnic1 --vswitch-name vSwitch0

esxcli network vswitch standard policy failover set --active-uplinks vmnic0,vmnic1  vmnic1 --vswitch-name vSwitch0

esxcli network vswitch standard policy failover set --failback yes --failure-detection beacon --load-balancing portid --notify-switches yes --vswitch-name vSwitch0

#

esxcli network vswitch standard add --ports 256 --vswitch-name vSwitch1

esxcli network vswitch standard add --ports 256 --vswitch-name vSwitch2

esxcli network vswitch standard uplink add --uplink-name vmnic2 --vswitch-name vSwitch1

esxcli network vswitch standard uplink add --uplink-name vmnic3 --vswitch-name vSwitch2

esxcli network vswitch standard set --mtu 9000 --cdp-status listen --vswitch-name vSwitch1

esxcli network vswitch standard set --mtu 9000 --cdp-status listen --vswitch-name vSwitch2

esxcli network vswitch standard portgroup add --portgroup-name Kernel --vswitch-name vSwitch1

esxcli network vswitch standard portgroup set --portgroup-name Kernel --vlan-id 35

esxcli network vswitch standard portgroup add --portgroup-name vMotion --vswitch-name vSwitch2

esxcli network vswitch standard portgroup set --portgroup-name vMotion --vlan-id 35

%firstboot --interpreter=busybox

cd /tmp

VMK0_IPADDR=\$(esxcli network ip interface ipv4 get | grep vmk0 | awk '{print \$2}')

VMK1_IPADDR=\$(echo \${VMK0_IPADDR} | awk '{print "10.10.10."\$4}' FS=.)

VMK2_IPADDR=\$(echo \${VMK0_IPADDR} | awk '{print "10.10.20."\$4}' FS=.)

esxcli network ip interface add --interface-name vmk2 --mtu 9000 --portgroup-name Kernel

esxcli network ip interface ipv4 set --interface-name vmk2 --ipv4 \${VMK1_IPADDR} --netmask 255.255.255.0 --type static

esxcli network ip interface add --interface-name vmk3 --mtu 9000 --portgroup-name vMotion

esxcli network ip interface ipv4 set --interface-name vmk3 --ipv4 \${VMK2_IPADDR} --netmask 255.255.255.0 --type static

### NTP CONFIGURATIONS ###

cat > /etc/ntp.conf << __NTP_CONFIG__

restrict default kod nomodify notrap noquery nopeer

restrict 127.0.0.1

server 171.68.10.80

server 10.81.254.131

__NTP_CONFIG__

# backup ESXi configuration to persist changes

#/sbin/auto-backup.sh

%post --interpreter=busybox --ignorefailure=true

cd /tmp

localcli network firewall set --default-action true

localcli network firewall set --enabled false

## Notifying back to Cisco UCS Director once the blade is up ##

wget http://$PXE_WEBSERVER_MGMT_VLAN_IP/$PXEID/notify.html -O /tmp/notify.html

localcli network firewall set --default-action false

localcli network firewall set --enabled true

# Let us poweroff/shutdown our selves.

poweroff

Produces this out put

=====================

[root@UCSDLastTimeTonight468:~] esxcli network nic list

Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description                            

------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  ----------------------------------------

vmnic0  0000:06:00.0  nenic   Up            Up           10000  Full    00:25:b5:06:00:00  1500  Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic1  0000:07:00.0  nenic   Up            Up           10000  Full    00:25:b5:06:00:01  1500  Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic2  0000:08:00.0  nenic   Up            Up           10000  Full    00:25:b5:06:00:02  9000  Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic3  0000:09:00.0  nenic   Up            Up           10000  Full    00:25:b5:06:00:03  9000  Cisco Systems Inc Cisco VIC Ethernet NIC

[root@UCSDLastTimeTonight468:~] esxcli network ip interface ipv4 get

Name  IPv4 Address   IPv4 Netmask   IPv4 Broadcast  Address Type  Gateway  DHCP DNS

----  -------------  -------------  --------------  ------------  -------  --------

vmk0  172.16.32.202  255.255.254.0  172.16.33.255   STATIC        0.0.0.0     false

vmk2  10.10.10.202   255.255.255.0  10.10.10.255    STATIC        0.0.0.0     false

vmk3  10.10.20.202   255.255.255.0  10.10.20.255    STATIC        0.0.0.0     false

[root@UCSDLastTimeTonight468:~] esxcli network vswitch standard list

vSwitch0

   Name: vSwitch0

   Class: etherswitch

   Num Ports: 5632

   Used Ports: 6

   Configured Ports: 128

   MTU: 1500

   CDP Status: listen

   Beacon Enabled: false

   Beacon Interval: 5

   Beacon Threshold: 3

   Beacon Required By: VS:vSwitch0

   Uplinks: vmnic1, vmnic0

   Portgroups: VMNetwork32, VM Network, Management Network

vSwitch1

   Name: vSwitch1

   Class: etherswitch

   Num Ports: 5632

   Used Ports: 4

   Configured Ports: 256

   MTU: 9000

   CDP Status: listen

   Beacon Enabled: false

   Beacon Interval: 1

   Beacon Threshold: 3

   Beacon Required By:

   Uplinks: vmnic2

   Portgroups: Kernel

vSwitch2

   Name: vSwitch2

   Class: etherswitch

   Num Ports: 5632

   Used Ports: 4

   Configured Ports: 256

   MTU: 9000

   CDP Status: listen

   Beacon Enabled: false

   Beacon Interval: 1

   Beacon Threshold: 3

   Beacon Required By:

   Uplinks: vmnic3

   Portgroups: vMotion

[root@UCSDLastTimeTonight468:~] esxcli network vswitch standard policy failover

get -v vSwitch0

   Load Balancing: srcport

   Network Failure Detection: beacon

   Notify Switches: true

   Failback: true

   Active Adapters: vmnic0

   Standby Adapters: vmnic1

   Unused Adapters:

[root@UCSDLastTimeTonight468:~]

ESXi/vCenter cli commands

=========================

esxcli network nic list

esxcli network ip interface ipv4 get

esxcli network vswitch standard list

esxcli network vswitch standard policy failover  get -v vSwitch0

Parallel Server Deployment

=====================

8 ESXi Servers at the same time in this example:

Screen Shot 2017-09-13 at 10.02.07 PM.png

Master loop calling the provisioning workflows:

Screen Shot 2017-09-13 at 10.02.20 PM.png

Pure Disk

Creation of Disk on any UCSm Service Profile

workflow_415.png

PURE disk on many ESXi Servers:

workflow_442 (2).png

Here are the entries for the SSH task to create the zoning in the MDS-A and MDS-B for the Pure Datastore:

SSH command for MDS A

=====================

Forward:

configure terminal

zone name ${custom_ADDSRID2String_4739.B} vsan ${custom_CleanUpVSANNumber_4754.B1}

member pwwn ${SelectUCSServiceProfile_4735.SP_VHBA1_WWPN}

member pwwn 52:4A:93:78:77:2B:29:01

zoneset name flexpod vsan ${custom_CleanUpVSANNumber_4754.B1}

member ${custom_ADDSRID2String_4739.B}

exit

zoneset activate name flexpod vsan ${custom_CleanUpVSANNumber_4754.B1}

exit

copy running-config startup-config

Reverse:

configure terminal

zoneset name flexpod vsan ${custom_CleanUpVSANNumber_4754.B1}

no member ${custom_ADDSRID2String_4739.B}

exit

zone name ${custom_ADDSRID2String_4739.B} vsan ${custom_CleanUpVSANNumber_4754.B1}

no member pwwn ${SelectUCSServiceProfile_4735.SP_VHBA1_WWPN}

no member pwwn 52:4A:93:78:77:2B:29:01

exit

no zone name ${custom_ADDSRID2String_4739.B} vsan ${custom_CleanUpVSANNumber_4754.B1}

zoneset activate name flexpod vsan ${custom_CleanUpVSANNumber_4754.B1}

exit

copy running-config startup-config

SSH command for MDS B

=====================

Forward:

configure terminal

zone name ${custom_ADDSRID2String_4739.B} vsan ${custom_CleanUpVSANNumber_4754.B2}

member pwwn ${SelectUCSServiceProfile_4735.SP_VHBA2_WWPN}

member pwwn 52:4A:93:78:77:2B:29:01

zoneset name flexpod vsan ${custom_CleanUpVSANNumber_4754.B2}

member ${custom_ADDSRID2String_4739.B}

exit

zoneset activate name flexpod vsan ${custom_CleanUpVSANNumber_4754.B2}

exit

copy running-config startup-config

Reverse:

configure terminal

zoneset name flexpod vsan ${custom_CleanUpVSANNumber_4754.B2}

no member ${custom_ADDSRID2String_4739.B}

exit

zone name ${custom_ADDSRID2String_4739.B} vsan ${custom_CleanUpVSANNumber_4754.B2}

no member pwwn ${SelectUCSServiceProfile_4735.SP_VHBA2_WWPN}

no member pwwn 52:4A:93:78:77:2B:29:01

exit

no zone name ${custom_ADDSRID2String_4739.B} vsan ${custom_CleanUpVSANNumber_4754.B2}

zoneset activate name flexpod vsan ${custom_CleanUpVSANNumber_4754.B2}

exit

copy running-config startup-config

RustyLabEsxiParallelServerDeployment.wfdx.zip
PureDiskonEsxiServer.wfdx.zip
PureDsforManyEsxServers.wfdx.zip
GlassHouseLabBuildOut15WorkFlows.wfdx.zip
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Quick Links
Discussions
Customers Also Viewed These Support Documents