Wednesday, April 23, 2014

Creating Snapshots in Linux / MacOSX

Here is a script I wrote back in the 1999 time frame to perform daily, weekly, monthly snapshots of critical directories.  This scripts is designed for filesystems that support the notion of hard links, aka Linux and MacOSX. (The Windows filesystems do not support hard links and thus this will not work on Windows).

This script is usually called from /etc/crontab and requires that you have rsync installed on the system.  At the top of the actual script code is a section that you will need to update to specify where you want the backups to go and what directories you want to backup.


#!/usr/bin/perl
#
#
####################################################################
# Ideas taken from Mike's handy rotating-filesystem-snapshot       #
# utility, http://www.mikerubel.org/computers/rsync_snapshots/     #
# and from code written by Eric Ross                               #
####################################################################
#
####################################################################
# This program is not guaranteed to work at all, and by using this #
# program you release the author of any an all liability.          #
#                                                                  #
# You may use this program so long as this notice, disclaimer and  #
# comment box remain intact and unchanged.                         #
#                                                                  #
# Please send me all bugs and patches.                             #
#                                                                  #
# Written by Bret Jordan                                           #
# jordan at open1x littledot org                                   #
####################################################################
#
#
#
# Example /etc/crontab
# Every night at 1:01 make rsync copy and then give plenty of time before snapshots
# 01 1  * * *     root  /local/bin/snapshot rsync
#
# Every night at 2:01 make snapshots
# 01 2  * * *     root  /local/bin/snapshot snap 7 daily
#
# Every Sunday Morning at 3:01 make weekly snapshot
# 01 3  * * 7     root  /local/bin/snapshot snap 4 weekly
#
# First day of every month at 4:01 make monthly snapshot
# 01 4  1 * *     root  /local/bin/snapshot snap 12 monthly


use 5.8.8;
use strict;
use warnings;

our $VERSION = "0.02";
$VERSION = eval $VERSION;

my $sType = shift;
my $iHistoryLength = shift; 
my $sFreq = shift;


########################
# BEGIN THINGS TO UPDATE

my $sSnapDir = "/backups/snapshots";
my $sCurrentBackupDir = "$sSnapDir/backup.current";
my @aDirsToBackup = qw (/data/home/);

# END THIGNS TO UPDATE
########################


unless ($sType eq "rsync" || $sType eq "snap")
{ &ShowSyntax(); }


if ($sType eq "snap")
{
    unless ($iHistoryLength =~ /^\d+$/) 
    { &ShowSyntax(); }

    unless ($sFreq eq "hourly" || $sFreq eq "daily" || $sFreq eq "weekly" || $sFreq eq "monthly") 
    { &ShowSyntax(); }
}


sub ShowSyntax
{
    print "Please run command with ./snapshot <rsync snap=""> <historylength> <freq name="">\n";
    print "Example: ./snapshot 7 daily\n";
    print "Example: ./snapshot 24 hourly\n";
    die;
}

# Lets make sure we are running at root
unless ( $< == 0 ) { die "This program must be run as root\n"; }

# Do we have write access to the snap directory?
unless ( -w $sSnapDir ) { die "This user does not have write access to $sSnapDir\n"; }


if ($sType eq "rsync") 
{
    #----------------------------------------
    # Rsync Command
    #----------------------------------------
    #-a = archive mode which is -rlptgoD (no -H,-A,-X)
    #
    #-r = recurse in to directories
    #-l = copy symlinks as symlinks
    #-p = preserve permissions
    #-t = preserve modification times
    #-g = preserve group
    #-o = preserve owner
    #-D = preserve device files and special files

    foreach (@aDirsToBackup) { system("rsync -vax --delete $_ $sCurrentBackupDir"); }

    # Touch the directory so the date time is right
    system("touch $sCurrentBackupDir");
}
elsif ($sType eq "snap")
{
    # Step 1: Delete oldest snap shot and increase the age of the rest
    my $i = $iHistoryLength;
    while ($i > 0)
    {
    my $sCurrentDir = "$sSnapDir/backup.$sFreq.$i";

    # If index equals the max length of the history, then delete that directory
    if ( $i == $iHistoryLength ) { system ("rm -rf $sCurrentDir"); }
    else 
    {  
        my $j = $i + 1;
        my $sNewDir = "$sSnapDir/backup.$sFreq.$j";
        if ( -d $sCurrentDir ) { system ("mv $sCurrentDir $sNewDir"); }
    }
    $i--;
    }

    # Step 2: Make a hard link from the backup.current to backup.daily.1
    unless ( -d $sCurrentBackupDir ) { die "The current backup directory does not exist\n"; }
    system("cp -al $sSnapDir/backup.current $sSnapDir/backup.$sFreq.1");
}



Thursday, April 10, 2014

Heartbleed-ing on the inside

It is alarming to me how many vendors have not yet produced a patch for the OpenSSL issue, even days after it was released. Some vendors have taken the stance and said, "you should not have the management/configuration interface be public facing". This mindset follows the escargot model of security from the 1990s and is not an acceptable solution.

The idea of having just a hard and crunchy firewall perimeter while maintaining a soft and chewy inside is dangerous. Please remember that when a system in the organization is compromised, it can, and often does, give a remote user (threat actor) access to the inside network. When this happens the predictive and preventive security tools that you spent so much money on are not going to help you when all your trusted servers are effectively wide open to the internal network.

Thursday, February 6, 2014

Compile eepe r392 for MacOSX

Getting eepe to compile on a Mac is not too bad once you get everything in the right place. The versions of code I am running are as follows:
  • Mac OSX 10.7.5
  • xcode 4.2
  • Qt 4.8.2
I am going to assume you have some basic understanding of installing applications, downloading source code, and generally working with source code. Here are the steps:

Step 1: Install XCode if you have not already done so (you can get this from the App Store) and install Qt 4.8.x from http://qt-project.org/downloads.  5.x might work as well, but I have not tried that.

Step 2: Make a directory in your home directory for source code, I call mine "workspace"

Step 3: Download the missing qextserialenumerator_osx.cpp file from https://code.google.com/p/qextserialport/ .  I downloaded the following file: qextserialport-1.2rc.zip. Save this file to the workspace directory in your home directory

Step 4: Open a terminal
> cd workspace
> unzip qextserialport-1.2rc.zip

Step 5: Check out the eepe source code
> svn checkout http://eepe.googlecode.com/svn/trunk eepe

Step 6: Copy the missing files to the eepe source tree and build xcode project.  Unlike in Linux, qmake does not make a Makefile, but rather an xcode project.
> cp qextserialport-1.2rc/src/* eepe/src/
> cd eepe/src
> qmake

Step 7: From Finder open the following file "~/workspace/eepe/src/eepe.xcodeproj".  It should open in XCode if you have it installed correctly.  Now from the XCode menu select Product -> Archive.  This will build an archive version of the eepe code.

Step 8: After a few minutes the "Organizer" window should pop open and you should see your eepe project.  Control-Click and select "Show In Finder".  Then Control-Click the file "eepe 2-6-14 10.26 PM.xcarchive" (your file name will be slightly different based on date and time) and select "Show Package Contents".  Navigate down this tree to Products -> Applications and find the eepe.app.  Now drag this to your Desktop or the Applications directory and run the eepe.app

Saturday, December 21, 2013

Configuring VMware ESX 5.5 from the command line

Over the past few months I have spent a lot of time building a large ESX and nested ESX infrastructure based on VMware ESX 5.5. As you do this, you quickly realize that configuring ESX from the UI is painful, especially when you need to make sure you have all of the ESX servers exactly the same.  Here are some tips and tricks that I have found to be very helpful.

  1. Enable SSH on your ESX server and setup certificate based authentication.  This will greatly ease your work as you can then pipe configuration commands through SSH, which in turn allows you to script the whole configuration (and yes, this all works with the free version of ESXi).  I can now perform all of the configuration for 100+ ESX servers in a few seconds. On the ESX server the public keys for your Linux servers go in a file called:
    /etc/ssh/keys-root/authorized_keys
     
  2. I also like to change the motd, shell profile, and ntp.conf at the same time.  I just copy these files over.  The shell profile goes in a file called: /etc/profile.local

    My profile.local files looks like this:

    # profile.local

    PS1="[\u@\h]:\w-> "
    export PS1

    if [ "$TERM" != "dumb" ]; then
        alias ls='ls --color=auto'
        alias ll='ls -l -a --color=auto'
    fi

  3. Configure DNS and Hostname settings
    ssh root@x.x.x.x "esxcli network ip dns server add --server=192.168.0.11"

    ssh root@x.x.x.x "esxcli network ip dns server add --server=192.168.0.11"
    ssh root@x.x.x.x "esxcli system hostname set --host=esxserver01"
    ssh root@x.x.x.x "esxcli system hostname set --domain=mydomain.com"
     
  4. Configure NTP Settings
    Copy over a valid ntp.conf file to
    /etc/ntp.conf
    ssh root@x.x.x.x "esxcli network firewall ruleset set --enabled=true --ruleset-id=ntpClient"
    ssh root@x.x.x.x "chkconfig --add ntpd"
     
  5. License ESX
    ssh root@x.x.x.x "vim-cmd vimsvc/license --set xxxxx-xxxxx-xxxxx-xxxxx-xxxx"
     
  6. Setup any networking you need.  For my setup, I need to rename the first port group and create a new vswitch with a port group.  You also need to change the failover state as it defaults to non active.  This is how I did that.
    ssh root@x.x.x.x "esxcli network vswitch standard portgroup remove -p \'VM Network\' -v vSwitch0"
    ssh root@x.x.x.x "esxcli network vswitch standard portgroup add -p \'Trusted Network\' -v vSwitch0"

    ssh root@x.x.x.x "esxcli network vswitch standard add -v vSwitch1"
    ssh root@x.x.x.x "esxcli network vswitch standard portgroup add -p \'Client Network\' -v vSwitch1"
    ssh root@x.x.x.x "esxcli network vswitch standard uplink add -u vmnic1 -v vSwitch1"
    ssh root@x.x.x.x "esxcli network vswitch standard policy failover set -a vmnic1 -v vSwitch1"
     
  7. Reboot ESX server so all change take effect
    ssh root@x.x.x.x "reboot"
     
As you can see, once you setup certificate based authentication, you could easily script the above commands in bash, perl, python, etc and configure all of you ESX servers at once.  If you do this, I found that you need to add a sleep for 2 seconds statement between setting the DNS hostname and setting the DNS domain. 

Thursday, October 10, 2013

Upgrading ESXi from 5.1 to 5.5

I started upgrading my ESX servers tonight and here is the simple version of how to do that. Now if this was VMware's official howto documentation you would be on page 243 by now and would still need to read to page 500 before you got through it all.  Tech writers should not be paid by the word.

1) Shutdown all running VMs

2) Put ESX in maintenance mode:
> vim-cmd /hostsvc/maintenance_mode_enter

3) Enable outbound HTTP client:
> esxcli network firewall ruleset set -e true -r httpClient

4) List avaliable updates.  You want the standard one and it can takes FOREVER for it to complete, say more than 5 minutes with no feedback:
> esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep ESXi-5.5

5) Perform the upgrade:
> esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-5.5.0-1331820-standard

6) Reboot

7) Exit maintenance mode:
> vim-cmd /hostsvc/maintenance_mode_exit