An interesting evening – “Boot logical drive is configured but missing or offline.”

October 5th, 2009

So I got a call from a customer on the way home Thursday night. They were doing some server maintenance, updating drivers and firmware for their HP DL380 G5 through HP SIM. Nothing unusual happened during the installs, but after the admin rebooted the server you got this error message, “Boot logical drive is configured but missing or offline.” I told him that I had never scene it before, but walked him through reinstalling the firmware from the HP Maintenance CD. After a reboot we got the same error message. This was the departments File and Print server and had an external shelf with a P800 RAID controller, so we needed to get this up and running. The OS was installed on a mirrored set of drives, so we tried failing one drive and rebooting. Same problem.

Next we called HP and they had us boot off a SmartStart CD and going into maintenance and wipe the RAID configuration and reboot. The tech felt if this didn’t work that we would have to reinstall the OS again. Also since our data was on a different set of disks he felt fine with the recommendation. The issue is that this was a small department with only the one server, which was also their backup server, so reinstalling and reconfiguring this server would take all night as the backup product didn’t have a bare metal restore.

I kept asking him, how does upgrading firmware effect disks, it doesn’t make sense. So I suggested we back rev the firmware on the RAID controller. The tech didn’t think this would do anything but I felt it couldn’t hurt. After booting off the maintenance CD we installed the previous firmware and rebooted. Poof… magically the server booted and our evenings were mostly spared.

One interesting thing is that windows lost one of the data partitions off the external cages and had to be fixed, but in the end everything worked out well. So long story short, if you have a HP RAID controller and are upgrading to their new 6.x firmware version and get the error above, quickly install the older version and cross your fingers. šŸ™‚

VMworld Follow Up

September 14th, 2009

After the VMworld presentation a few people asked me some questions and I wanted to address one of those now. The question was by an admin that doesn’t have the funding or extra hardware to setup a separate test network, but because he’s a good admin he still wants to do testing. So how does he accomplish this?

I came up with five potential solutions varying in cost and complexity to implement.

Option 1 –

The first option is one he said he didn’t have, but I wanted to mention anyway, and that is to dig up a piece of hardware and build a separate test environment. šŸ™‚ A possible environment could be a server you recently did a P2V conversion on and you could combine this with the free version of ESXi for an inexpensive solution.

Pros of this solution are having a separate environment so that a single small mistake of misconfiguring the network on the VM or physical switch will not cause conflicts. This is especially important when you are testing something as critical as an Active Directory upgrade which this admin wanted to do. Another pro is that you can keep the same VM name as production during testing to limit other errors.

Cons of this solution are the cost and complexity of setting up and maintaining this separate environment. The cons can be limited and I encourage you to try and create a separate test environment.

For those that still feel they cannot have this separate environment I created a visio diagram to give a visual aid for the next four options.

Private VM Setup

Option 2 –

The next best thing to the fully isolated test network is to use a dedicated vswitch with dedicated vmnics connected to a separate isolated physical switch (recommend non routed switch). This will definitely keep your test traffic isolated from your production traffic and somewhat limits you exposure to human errors (on the physical switch side).

Pros – can VMotion between hosts, communicate VM to VM, no advanced trunking or vlan tagging
Cons – A misconfiguration could put the test VM on the production network

Option 3 –

Use a dedicated vswitch with dedicated vmnics to a separate vlan on the same physical switch as production (recommend use a non routed vlan)

Pros – Can VMotion between hosts, communicate VM to VM, doesn’t require dedicated physical switches

Cons – shared production and test equipment, requires dedicated physical NICs, and a misconfiguration could put the test VM on the production network

Option 4 –

use a trunk on the same vmnics as production, but use a separate vlan for test (recommend non routed vlan)

Pros – Can VMotion between hosts, communicate VM to VM, requires least amount of equipment
Cons – Most complex design, have to use VLAN tagging, trunk to vSwitch, a misconfiguration a multiple locations could put the test VM on the production network

Option 5 –

use an isolated vSwitch on one host (vswitch with no attached VMnics)
Pros – VMs isolated from everything
Cons – VMs have to be on same host, no vmotion, drs, or communication between test VMs on different ESX hosts, a misconfiguration could put the test VM on the production network

Hopefully this helps. Any comments are welcome.

Private VM Setup

VMworld 09 Wrapup

September 4th, 2009

Well VMworld is over, but the work continues. I got some good questions at the end of the session that I wanted to respond to and am working on posts. For now I wanted to get the presentation up like I said I would. Please use for your reference, but remember these are my opinions and may not be best practices for your environment, so make sure to verify with your vendors!

VMworld 09 Presentation

Presentation Complete!!

September 2nd, 2009

I thought the presentation went very well and got some good feedback from people that stayed behind to ask questions. Hopefully the people that left also got something out of the presentation. I would love to hear any suggestions or comments that people might have. Who knows I could be back next year. šŸ™‚ Anyway I will continue to post more links and resources here and upload the presentation slides. So look for that soon, but I am crashing for now.

My PowerShell Healthcheck Script

September 1st, 2009

Like most admins I tend to “borrow” tools and scripts rather than “create” them. This script is no exception. I first got this script aboutĀ a yearĀ ago from Ivo Beerens ivo@ivobeerens.nl. Initially the script did some cool things, but I sought to improveĀ it and was able to get someĀ additional information about VMDK usage that I wanted.Ā Since that time many people have improved the script and othersĀ haveĀ come out with their own that I am sureĀ are better than mine. ButĀ since I am demoing this script today I thought I should post it as well. I modified the script to support a port shifted vcenter, report on a specific datacenter as I manage multiple ones and wanted a separate report, added VM disk usage info, shared .css location for formatting and added a section on top 10 lowest VM disk free space. Hope you find it useful.

CleanVMwareHealthcheckĀ 

VMworld Thoughts

September 1st, 2009

I don’t know what other people are experiencing, but this years VMworld has fallen really short after attending last year.Ā The firstĀ issue is allĀ the IT problems. My friends are teasing meĀ because “if a conference of IT people can get it to work then who can?”Ā I felt for the IT guys working the conference, but making things work is what we do. I had issues with the vSpehere bootcamp on Sunday, which was a total bust and waste of the day, only to get to my VCP4 exam yesterday to find out they are having slowness issues. My “timed” 90 minutes exam took well over 2 hours and I still had “30 minutes” left. Thankfully I passed that, but I was late for my lab andĀ not able to attend. My first session today was also bad as it turned into a vendor pitch and hadĀ nothing to do with theĀ title. It got me thinking about my presentation and hopefullyĀ it will live up to my own expectations.Ā Lets hope things improve!Ā 

Forgiveness?

September 1st, 2009

After looking at this blog setup I realize how bad it is and that I have little idea of how to fix it at the moment, so I ask for forgiveness. šŸ™‚ It will get better. I am going to try to post a few links to things I promised in the presentation so that there will be a few resources available initially that I can expand on rapidly. Thanks for you patience while I figure this WordPress thing out.

I Made it

September 1st, 2009

Hello Everyone,

Thanks for visiting my blog. I created this blog as a resource for my presentation at VMworld, so naturally I will be posting mostly technical information and links as it relates to virtualization and server administration. I had hoped to have many posts up by now, but lack of time and my never ending battle with procastination got the best of me. Anyway I hope this will be useful to you and me as well. Thanks,

Tony