Thursday, April 10, 2014

Heartbleed-ing on the inside

It is alarming to me how many vendors have not yet produced a patch for the OpenSSL issue, even days after it was released. Some vendors have taken the stance and said, "you should not have the management/configuration interface be public facing". This mindset follows the escargot model of security from the 1990s and is not an acceptable solution.

The idea of having just a hard and crunchy firewall perimeter while maintaining a soft and chewy inside is dangerous. Please remember that when a system in the organization is compromised, it can, and often does, give a remote user (threat actor) access to the inside network. When this happens the predictive and preventive security tools that you spent so much money on are not going to help you when all your trusted servers are effectively wide open to the internal network.

Thursday, February 6, 2014

Compile eepe r392 for MacOSX

Getting eepe to compile on a Mac is not too bad once you get everything in the right place. The versions of code I am running are as follows:
  • Mac OSX 10.7.5
  • xcode 4.2
  • Qt 4.8.2
I am going to assume you have some basic understanding of installing applications, downloading source code, and generally working with source code. Here are the steps:

Step 1: Install XCode if you have not already done so (you can get this from the App Store) and install Qt 4.8.x from  5.x might work as well, but I have not tried that.

Step 2: Make a directory in your home directory for source code, I call mine "workspace"

Step 3: Download the missing qextserialenumerator_osx.cpp file from .  I downloaded the following file: Save this file to the workspace directory in your home directory

Step 4: Open a terminal
> cd workspace
> unzip

Step 5: Check out the eepe source code
> svn checkout eepe

Step 6: Copy the missing files to the eepe source tree and build xcode project.  Unlike in Linux, qmake does not make a Makefile, but rather an xcode project.
> cp qextserialport-1.2rc/src/* eepe/src/
> cd eepe/src
> qmake

Step 7: From Finder open the following file "~/workspace/eepe/src/eepe.xcodeproj".  It should open in XCode if you have it installed correctly.  Now from the XCode menu select Product -> Archive.  This will build an archive version of the eepe code.

Step 8: After a few minutes the "Organizer" window should pop open and you should see your eepe project.  Control-Click and select "Show In Finder".  Then Control-Click the file "eepe 2-6-14 10.26 PM.xcarchive" (your file name will be slightly different based on date and time) and select "Show Package Contents".  Navigate down this tree to Products -> Applications and find the  Now drag this to your Desktop or the Applications directory and run the

Saturday, December 21, 2013

Configuring VMware ESX 5.5 from the command line

Over the past few months I have spent a lot of time building a large ESX and nested ESX infrastructure based on VMware ESX 5.5. As you do this, you quickly realize that configuring ESX from the UI is painful, especially when you need to make sure you have all of the ESX servers exactly the same.  Here are some tips and tricks that I have found to be very helpful.

  1. Enable SSH on your ESX server and setup certificate based authentication.  This will greatly ease your work as you can then pipe configuration commands through SSH, which in turn allows you to script the whole configuration (and yes, this all works with the free version of ESXi).  I can now perform all of the configuration for 100+ ESX servers in a few seconds. On the ESX server the public keys for your Linux servers go in a file called:
  2. I also like to change the motd, shell profile, and ntp.conf at the same time.  I just copy these files over.  The shell profile goes in a file called: /etc/profile.local

    My profile.local files looks like this:

    # profile.local

    PS1="[\u@\h]:\w-> "
    export PS1

    if [ "$TERM" != "dumb" ]; then
        alias ls='ls --color=auto'
        alias ll='ls -l -a --color=auto'

  3. Configure DNS and Hostname settings
    ssh root@x.x.x.x "esxcli network ip dns server add --server="
    ssh root@x.x.x.x "esxcli network ip dns server add --server="
    ssh root@x.x.x.x "esxcli system hostname set --host=esxserver01"
    ssh root@x.x.x.x "esxcli system hostname set"
  4. Configure NTP Settings
    Copy over a valid ntp.conf file to
    ssh root@x.x.x.x "esxcli network firewall ruleset set --enabled=true --ruleset-id=ntpClient"
    ssh root@x.x.x.x "chkconfig --add ntpd"
  5. License ESX
    ssh root@x.x.x.x "vim-cmd vimsvc/license --set xxxxx-xxxxx-xxxxx-xxxxx-xxxx"
  6. Setup any networking you need.  For my setup, I need to rename the first port group and create a new vswitch with a port group.  You also need to change the failover state as it defaults to non active.  This is how I did that.
    ssh root@x.x.x.x "esxcli network vswitch standard portgroup remove -p \'VM Network\' -v vSwitch0"
    ssh root@x.x.x.x "esxcli network vswitch standard portgroup add -p \'Trusted Network\' -v vSwitch0"
    ssh root@x.x.x.x "esxcli network vswitch standard add -v vSwitch1"
    ssh root@x.x.x.x "esxcli network vswitch standard portgroup add -p \'Client Network\' -v vSwitch1"
    ssh root@x.x.x.x "esxcli network vswitch standard uplink add -u vmnic1 -v vSwitch1"
    ssh root@x.x.x.x "esxcli network vswitch standard policy failover set -a vmnic1 -v vSwitch1"
  7. Reboot ESX server so all change take effect
    ssh root@x.x.x.x "reboot"
As you can see, once you setup certificate based authentication, you could easily script the above commands in bash, perl, python, etc and configure all of you ESX servers at once.  If you do this, I found that you need to add a sleep for 2 seconds statement between setting the DNS hostname and setting the DNS domain. 

Thursday, October 10, 2013

Upgrading ESXi from 5.1 to 5.5

I started upgrading my ESX servers tonight and here is the simple version of how to do that. Now if this was VMware's official howto documentation you would be on page 243 by now and would still need to read to page 500 before you got through it all.  Tech writers should not be paid by the word.

1) Shutdown all running VMs

2) Put ESX in maintenance mode:
> vim-cmd /hostsvc/maintenance_mode_enter

3) Enable outbound HTTP client:
> esxcli network firewall ruleset set -e true -r httpClient

4) List avaliable updates.  You want the standard one and it can takes FOREVER for it to complete, say more than 5 minutes with no feedback:
> esxcli software sources profile list -d | grep ESXi-5.5

5) Perform the upgrade:
> esxcli software profile update -d -p ESXi-5.5.0-1331820-standard

6) Reboot

7) Exit maintenance mode:
> vim-cmd /hostsvc/maintenance_mode_exit

Wednesday, May 8, 2013


If you have an LSI RAID controller in your ESX host and you would like the LSI RAID health information to show up in vSphere, under Health Status -> Storage, you will need to install the LSI VMware SMIS Provider.  Further, in order to perform CLI RAID commands on the ESX host for the LSI controller, you will need to install the MegaCLI vib package.

I am using the following packages on ESX 5.1 Update 1
VMWare SMIS Provider VIB - MR 5.6
MegaCLI 5.5 P1

And downloaded them from here:

Step 1: Download the vib files from  I found it easier to find them on the product page instead of the LSI download page.

Step 2: scp the vib files over to your ESX host and put them in the /tmp directory

Step 3: SSH to the ESX host and install the vib packages by running the following commands:
/tmp # esxcli software vib install --no-sig-check -v /tmp/vmware-esx-MegaCli-8.07.07.vib
/tmp # esxcli software vib install --no-sig-check -v /tmp/vmware-esx-provider-lsiprovider.vib

Step 4: Shut down any running VMs and Reboot ESX host

Step 5: Power up all your VMs you shut down in Step 4.  NOTE: It will take about 20 minutes for all of the data to show up in the Health Status section.  Also, I had issues with the SMIS Provider randomly stopping on ESX 5.1, and thus upgraded to 5.1 Update 1.

Some useful MegaCLI commands are:
cd /opt/lsi/MegaCLI

Controller information
./MegaCli -AdpAllInfo -aALL
./MegaCli -CfgDsply -aALL

Enclosure information
./MegaCli -EncInfo -aALL

Virtual drive information
./MegaCli -LDInfo -Lall -aALL

Physical drive information
./MegaCli -PDList -aALL

NOTE: If you want to run the MegaRAID Storage Manager from your Windows/Linux system and have it manage the RAID controller in your ESX host, you need to be in the same Layer2 VLAN.  The software uses multicast to find controllers that it can support.  I also found that if your systems are not in DNS, and just have IP addresses, then you MUST add entries for them in the /etc/hosts file or /Windows/system32/drivers/etc/hosts file. If you do not do this then the software will get confused and refer to every device it finds as a NULL string.

Upgrading ESX 5.1 to 5.1 Update 1

A week or so ago, VMware released 5.1 Update 1 (Build Number: 1065491).  This post will show you how to quickly and easily upgrade your system.  If you read VMware's documentation and are still confused, than this should help you.  It is just too bad that VMware has to make everything more complicated than it needs to be.

Step 1: Download zip bundle ( ) not iso from VMware's download site: When you get to the portal, select "ESXi (Embedded and Installable) from the dropdown. 

Step 2: Copy (scp) the update zip file to your datastore (/vmfs/volumes/datastore1)

Step 3: Shut down all running VMs on the host you want to upgrade

Step 4: SSH to the ESX host you are going to upgrade and run the upgrade command
esxcli software vib install --depot /vmfs/volumes/datastore1/

Step 5: When the upgrade finishes reboot the ESX host

Step 6: Start all of the VMs that you shutdown in Step 3

Step 7: Upgrade the vmware-tools for each VM that had older versions

Step 8: Upgrade your vSphere application

Tuesday, June 26, 2012

More things to worry about

RSA SecurID 800 tokens appear to be compromised along with other brands. This month has been crazy on the security exploit and malware side of the house. There has been a rash of new and very interesting exploits and some pretty fun new malware. Earlier this month we also saw the boat get rocked, per say, when we learned of an exploit for hypervisors. Now a hypervisor attack has always been the holy grail of theoretical attacks right behind hard tokens, but before this month it was just that, theoretical. If the hypervisor attack proves to be true and left un-patched, this could nullify a lot of cloud deployments. (reference)

And it even gets more scary. Today, once again, we get to worry, really worry. The objects we have come to rely on for real secure authentication, hard tokens, are proving to be vulnerable in under 9 minutes of hacking.  Today we learn that the RSA SecurID 800 along with a few other brands of tokens have been exploited. (research paper)