PowerShell – It is frustratingly awesome!

As part of the service I’m providing to the client I’m working for, I’ve been developing scripts to both automate the initial deployment of ESXi (Unattended install), which has gone well and finished to the point it needs to be where a PXE boot environment sits on the LAN to automate the entire process from host server network boot to a standalone ESXi host existing on the network.  To follow on from this I opted to use PowerShell to provide follow-up configuration to that unattended phase.  This then expanded to configuring the vCenter server and auto-deploying VM’s.  A lot has changed about my scripts as I’ve learnt more about PowerShell and seen the trends used in writing PowerShell scripts from others on the Internet.

I’ve reached a point where my ESXi follow-on does as much as I need for now, but needs a vCenter in place first, so that has been a recent focus along with a deployment script (we are at a point where the client production environment needs VM’s so I had to escalate that one!).  I’ve really learnt a lot about PowerShell these last few weeks and am quite pleased with what my scripts can do as far as completing a lot of timely tasks in a matter of minutes.  In the case of the vCenter and VM deployment scripts, as I’ve developed them I’ve thought of more things I’d like them to do so it automated as much as possible and I’ve almost achieved a 100% VM deployment…I think!

The vCenter configuration script takes and empty vCenter server (newly deployed VCSA or Windows based) and populates the datacenters, clusters, resource pools, VM folders and vApps, whilst also setting the cluster HA/DRS and Resource Pool/vApps (to a basic level but has capacity to cover all values possible through PowerShell), all from a single CSV based database.  It makes a lot of checks to ensure objects can be created and values can be set to avoid errors, and can account for duplicate resource pool name across clusters when deploying vApps.  Folders are created in the right place and not in the ‘Hosts and Clusters’ window.  This has been enough to satisfy my need to deploy a production vCenter quickly and allow additional vCenter servers to be configured just as quick as and when necessary.

The ESXi follow-on script completes the customization of the host itself and joins it to the vCenter server, putting into the correct cluster.  There is a little more work needed here to eliminate hard-coded information but again, it is at the point where it does what I need easily and quickly.

The VM deployment script has been a challenging one as the VM’s will have 2x network adapters and some will have multiple disks.  The template is built with a single C: drive and configured as much as necessary with just one NIC.  Therefore I’ve had to figure out how to add disks during the VM build-out and then add additional adapters.  I also wanted to configure the 2nd NIC IP before the script finished and this was probably the most challenging part.  I have actually only finished this final piece of the jigsaw tonight and suffice to say, it works 🙂  The script itself uses another CSV file holding all critical details for each VM to be deployed.  The first phase sets up a unique copy of the Guest Customization Specification, configured in vCenter (I guess I could have created this with PowerShell too!) and applies the primary NIC settings which can then be passed to the VM during phase 2 which is to create the VM itself from the prepared template.  The VM placement is set and ensures the right cluster resource pool is used in the event duplicate names exist.  I’ve then added the second NIC ready for configuration later in the script because the VM hasn’t been powered on yet!  Phase 3 configures the CPU count, RAM, adds additional VMDK’s as necessary (applying the correct storage format), finally setting DRS Automation and HA Isolation Response/Restart Priority (if they are specified).  At this point the VM is powered on.  The final phase is to set the second NIC IP but the customization hasn’t completed yet so we need to wait.  The powered on VM will reboot itself approx. 2 minutes after first power-on to begin the customization and then reboot itself 1-2 more times before completing.  We really need to wait until that final boot before trying to set NIC 2’s IP.  Therefore I’ve put a loop in where I ping the VM’s custom primary IP (which won’t be active until customization has completed at which point we know the VM is ready).  Once a response is received there a further pause to allow the boot process to complete since the NIC becomes active prior to log-on ability (and we need to login to set the NIC), before PowerShell cmdlets are passed to the GuestOS with the local admin account to apply the secondary custom IP details.

That’s it!  To be honest, it doesn’t seem a lot to me but the challenge has been to learn and achieve it.  I don’t deny I’ve used the expertise of others from their Internet Blogs and forum posts to reach my goal but I can also say I’ve not found any one script out there that does all the things mine do, so I’ve had to use my brain to link everything together 🙂  My scripts are available @ https://drive.google.com/folderview?id=0B6F_8rRvfphJU0RSRlE2QUk2Z2c&usp=sharing

New Lab Setup

Well, I’ve had the Qnap TS-670 for just about a week now and it has certainly improved things.  I had to rebuild my nest ESXi hosts for some reason because they weren’t able to provision 2012R2 servers, even after starting with a fresh vCenter but that didn’t take long.  I’ve now got the extra NIC card in the TS-670 (4 NICs total) with 1 for mgmt, 2 for iSCSI and one for NAS storage access.  I’ve beefed up the DL380G6 server by doubling the RAM to 64Gb and adding a quad 1Gb NIC card to give 8 NICs overall.  This is now permitting an iSCSI multi-pathing set-up and additional networks to really start playing with things, including hands-on with Cisco IOS!  The aim, to have the following networks routed to the primary LAN where the ADSL router sits:

  • Router LAN
  • Home device LAN
  • Lab LAN
  • iSCSI LAN (not routed)
  • IPStorage LAN (not routed)
  • vMotion LAN (not routed)

I’ve got some network cabling to install at some point because I’m not able to properly set-up IP routing on the Cisco and am moving my study/office/junk room! elsewhere in the house.  I’ll be running a small number of Cat-6 cables from wall sockets to an RJ-45 patch panel and then into the switch.

Since getting the new-lab in place, I’ve been really working on the PowerShell scripts I’ve blogged about previously as the new lab really speeds t hings up, and will be posting another blog on that after this one.

Qnap TS-670 Pro

Having some issues with my Lab set-up and the convoluted iSCSI presentation I setup.  It comprises a FreeNAS VM running on NAS storage that then has a VMDK attached running on the same NAS storage.  This VMDK is presented as iSCSI to the ESXi hosts (Nested and Physical hosts), so it give the impression it is SAN to those hosts.  I’m getting a lot of errors which cause the entire setup to freeze since the vCenter server is running on that storage.  I’m moving it off to a Qnap NAS to see if it stabilises the iSCSI but since that Qnap is shared with general CIFS file services for my Lab and home networks, am not confident about having my vCenter using it long-term!  Therefore, I have just bought myself, or rather bought Virtual Earthrunner, a Qnap TS-670 with 5x 3Tb Western Digital Red drives and a dual 1Gb NIC expansion to give me 4x NICs in total (1 for management and up to 3 for iSCSI).  The unit support up-to 6 disks so I’m going to make use of the SSD caching option available on that particular unit to house an old 40Gb SSD I’ve got hanging around doing nothing to experiment with.  With RAID5 this should give me a modest 12Tb usable capacity for my Lab network so my plan was to break this up into a small LUN for physical ESXi host, a small LUN for ISO storage and then 3 or 4 LUNs to present to the nested ESXi hosts and configure in a cluster to experiment with StorageDRS etc.




The main unit should arrive this week so I’ll hopefully have positive comments come next weekend.  It’ll be nice to actually have a true iSCSI storage solution that should reduce load on the DL380 as well since I’ll not be doing everything through a VM on that host.  I’m also planning to add an additional 32Gb RAM (64Gb total) and a quad-port NIC to expand to 8 ports so I can double up on storage and VM connections.

Since first publishing this article, the FreeNAS has experienced the same issue, but since moving my vCenter I don’t lose the complete environment.



PowerShell Help Needed

I’ve reached a stumbling point in my PowerShell development for automating ESXi/vCenter customisation.  In a bid to leave out any hard-coded localised details from the scripts, by using CSV files, I’m stuck producing a list of clusters for the admin running the script to pick from.  The cluster details are in a CSV and I’ve used a foreach loop to pull out the names of the clusters, presenting them to the admin one by one in a numbered list.  Prior to this, I hard-coded the list and used a switch process to set a variable for the selected cluster.  I’m not sure how I can do this having used a foreach loop to pull CSV content, or if there is a better solution completely!

Could really use some help please!

Unattended & Automated ESXi

Today I managed to succeed in my objective of fully automating the base ESXi install and initial configuration using a PXE setup with TFTP and DHCP.  I have to configure the datastore array on the server (ready for automated partitioning) and ensure UEFI network book is off (opting for Legacy), but with the Lights-Out capabilities of the server this is easily done remotely.  Each host uses it’s own ks.cfg file so this has to be edited prior to deployment and a corresponding MAC address pxeclient file created.  I plan to use just one ks.cfg file with the kickstart process pulling network details during an initial DHCP boot via DNS (if possible) and this should eliminate the need to setup the MAC pxeclient file too if I decide to.

I used information gleaned from http://everythingshouldbevirtual.com/build-tftp-server-esxi-installs and http://www.geeksecrets.net/ to inspire me.

With hardware shipping to the DC soon, I wanted to be in a position where I could build and rebuild hosts from a remote location without having to faff around with USB or CDROM virtually mounted media.

More PowerCLI Work

Further to my previous post, I’ve been expanding on my PowerCLI scripts to form a more comprehensive automated deployment/configuration of ESXi, vCenter and virtual machines (although I’ve not reached the VM bit yet).  The vCenter and VM deployment/configuration phases will utilise CSV files to hold the relevant information and keep such sensitive details away from the script, which allows the script to be more visible whilst the CSV’s are kept secure.  By having a small set of CSV’s that contain detailed settings I hope to keep management and future changes simple as well as well as provide a controlled means of updating the live environment with changes into those CSV’s without the need to manually connect and apply.  I’m now thinking of taking out any hardening or general configuration actions and putting them into a CSV where they are called and applied, so again, only the CSV needs updating whilst the script remains simple and will perform the same function with whatever data is called.  I don’t have 3rd party deployment/automation tool at my disposal so presently, Excel/esxcli/PowerCLI/Kickstart is all I have available.

I’ve split the phases into separate scripts and used a simple menu script to call the relevant one as chosen by the admin.  The vCenter customisation script manages datacenter creation (if needed), cluster creation or reset to custom defaults, folder creation and vApp/Resource Pool creation/reset.  It will apply in-depth settings for objects such as clusters and vApps once they’ve been agreed.  There are some hardening requirements here also but these will be added later once scripted and probably already deployed into what will be a production vCenter.  The ESXi script I’ve already mentioned in the previous post, whilst the VM deployment script will deploy VM’s using a selection of templates and customisation profiles, into the intended cluster (of 3), custom folders/RP’s/vApps, add disks according to the requirements of the intended guest role (and provision as thin or TEZ) into the appropriate datastore, set CPU/RAM, set IP/Mask/GW/DNS and configure the required number of NICs into the appropriate portgroups.  The 4th script will just contain any PowerCLI cmdlets or ESX CLI commands to configure any VM’s already deployed as necessary, such as templates, which will then be used to build other VM’s and in theory, propagate the settings!

Scripts attached and I’d love PowerCLI guru’s to review and offer advice on how I could streamline or improve the scripts.  I’m a rookie to this so have probably written lines of code to achieve something a wizard could do in one line!ps-customesxi ps-menu ps-vcenter (renamed to .txt extension since WordPress wouldn’t let me attach .ps1 and I can’t be arsed to sort that out!).

vSphere ESXi Unattended Install & Follow-Up PowerCLI Script

This week I’ve been working on a means to automate the installation of ESXi with as much of the configuration completed as possible.  By this I mean joining the host to an AD for local auth, joining the host to a vCenter environment, building out local vSwitches and implementing as much of the hardening guide as possible.  This is the first time I’ve ever used the unattended install kickstart process and it has been slow but pleasing progress.  It would appear that almost all my requirements are possible, but may not be so easy through the kickstart process which is why I’ve also utilised the PowerCLI capabilities of vSphere to capture the impossible/difficult bits as a post deployment script.  All scripts are in early development and continue to progress as other parts of the infrastructure appear and allow additional functions to be possible.

My objective here is to ensure that all hosts are built identically by removing as much user intervention as possible and quick to deploy with little need to work through complex processes.  Yes, you can right a build guide with all the information and steps in minute detail, but steps can be missed for any number of reasons which poses a risk that each host could end up different, or missing a vital security configuration.  A mis-configured script could achieve the same result, but the speed of installation is important to me as it is possible other IT support staff will take over the infrastructure once deployment is complete and I want a good, clean, simple process for them to use should they need to deploy more hosts (which is likely too).

Installing from a USB key onto a server that had a local RAID1 array and USB Flash card, the first challenge was to get ESXi to install to the correct USB device, not onto the HDD and certainly not onto the install media USB (which it did the first time of trying!).  After some investigation into the kickstart script options I figured out how to achieve this objective but also how to prepare the local HDD for a VMFS datastore at the same time.  This was great because I wanted to rename the local datastore through an initial installation process anyway, so sorting out how to build it was taking me in the right direction.

Next came the network info for the management port, and I did have trouble here trying to use the %pre option I’ve seen used in so many online locations….it just complain about “/.pre” and falling over so I left that challenge for later opting for putting the IP details onto the command line.  I’m hoping to use a combination of DHCP/DNS and scripting to grab the correct network details from the network and apply that.  I also configured a number of vSwitch which were needed for the primary objective of this activity and set security etc.   With a few additional %firstboot configuration additions, the install was working nicely but I wanted more!

I don’t yet have a Windows AD, DHCP or DNS so we’re in the early phase of this project task but I’ve seen that it is possible to get a host to join an AD and also a vCenter from the kickstart process.  I’ve also seen people creating a local user account with shell access and setting perms but I decided to leave this out of this initial script for now.

With the ESXi host built and configured with it’s local datastore renamed, vSwitch configured and some hardening achieved, I moved to working on a PowerCLI script to complete some other actions needed for a clean and simple installation.  One big issue I had that I couldn’t find a way of doing during the unattended install, was to rename the default PortGroup used by the Management VMkernel.  I like to rename all PortGroups, which is easily done post-install via the VIClient, but this particular one was trouble for me.  So, one reason for PowerCLI because it is easy to do there!  Another cool task achieved at this stage was adding the host to vCenter and into the right cluster.  There will be several clusters operational so any automated (or semi-automated) process needs to be able to select the correct one without modifying code to do so.  PowerCLI did this with ease too.  While I was adding things to hosts and vCenters I thought why not add the local user and assign it to a role, propagating it!  Again, easy to do………..although I did have problems until I realised I’d spelt propagate wrong!!!

Lots more things to do with these scripts and more scripts to appear as I try to automate, ease the administration of the solution during deployment.  What I’d love to be able to do is automate the activation of a PowerCLI script from the kickstart install so it can be processed without user intervention until 100% completed.  I’d also like to figure out how to get my boot USB to auto-select the ks.cfg file without me have to SHIFT+O all the time.  I’ve tried adding runweasel=ks=usb:/ks.cfg to the BOOT.CFG kernelopts!

Kickstart cfg and PowerShell scripts attached, ps-il3-safetopost ks-il3-safetopost

Further Lab Enhancements

After the initial post regarding my home lab, not much has changed and to be honest, I’ve not done much with it.  Having passed the VCP I’m deciding whether to keep the vSphere 5.5 setup and work towards VCAP or rebuild with vSphere 6.0!  My current contract is based around 5.5 so that’s a tick in the ‘keep it’ box!  At present, the lab is using a single network alongside my general home devices, so it’s not a true reflection of a live environment as I’ve not got VLAN’s and don’t have any devices capable of internal IP routing without setting up another VM/Physical system to handle that.

Needing to learn Cisco’s IOS since that is the most common network vendor used all contracts I’ve applied/interviewed/accepted, I’ve decided to splash some business cash and procure a layer 3 Cisco switch to help further develop my lab and skills……two birds with one stone so to speak.  I wanted 1Gb capable connectivity on all ports (I’m not just learning for CCNA!), and therefore had to spend a bit more to get it.  Ebay had a few 24x 1Gb port switches and I opted for a 3750G-24TS-S which also comes with 4x SFP ports should I ever need to experiment with that!  It was actually cheaper that the base 3750G-24 switches though so I’m pleased with my purchase.  In all, £275, but I do need to get hold of a Cisco serial cable so perhaps my new client will have one spare!

With this new switch, arriving tomorrow, I’ll be placing it directly behind the BT home-hub and running the entire network from it with IP routing enabled.  I’ll VLAN off the general network (using the existing IP range), storage, vSphere Mgmt, vMotion and Lab networks.  It will be especially useful since I’m currently involved in low-level/system-level design with vSphere and Cisco switch/firewalls.

Virtual Earthrunner IT Ltd is Born

Having secured a contract and now just seeking closure on all ‘open’ opportunities, a Ltd company is needed to progress.  Having picked the name ‘Virtual Earthrunner IT’ a few weeks back it was an easy task of completing the accountancy application forms where the company will be set-up in the process.  I already have a logo design and business cards set-up so once a bank account is in place (also set-up by the accountants) I’ll be acquiring these low cost extras.  I’ve opted to start life as a Ltd company owner using Nixon Williams as my accountants.  They seem to offer a good service for £95+VAT per month and I’ve opted to use their address as the main company address (as advised by several website on the subject), at an additional cost of £5+VAT.  There is no contractual tie-in so I’m free to switch accounts at a later date if necessary but I’m going to leave things as they are until I get settled and get my head around it all.

Here’s hoping for a long and prosperous career as a business owner and contractor/consultant.

Planned Training Path

Well, still struggling to get a contract and getting frustrated with agencies at present.  Very few actually call you when you’ve applied and some never get back to you when your CV has been submitted to the client!  I need the feedback to establish why I’m being rejected!  I believe I have a strong enough experience/skill set so maybe I’m too good for the roles I apply for….maybe I don’t have the right experience/skill set!  Either way it’ll be nice to know so I can tailor my CV to suite.

Once I get some contract money coming in through my Ltd company I’ve got a plan for personal training.  To start with, I need to update my VCP and sadly I’ve slacked in revising for the exam due next week because I’m getting a bit down in the dumps for struggling to find work despite trying for at least 6 weeks.  Once my VCP is secured, I was planning on the following:

  • CCNA
  • MCSA Windows Server 2012
  • MCA SQL Server
  • VCP-DT
  • Prince 2
  • ITIL Foundation

I hope that by working on these and passing exams I’ll have a proven record of understanding in all the areas.  Currently I’m going on hands-on experience which, as a recruiter in my time at SSTL, I felt was far more valuable than a piece of paper.  However, the certificate does prove one to be capable of absorbing information and extracting it correctly in a formal examination environment, a personal capability rather than defined technical ability.

I’ve fingers in so many pies at the moment, surely one has to be ready for eating!  I just can’t believe that my CV is not getting me interviews all the time based on the feedback I get about it!