Working with binary in Go (golang).

I have recently been working with raw IP packet generation in Go and came across a bit of code in the "golang.org/x/net/ipv4/header.go" file that I believe could use some explanation.  It uses a bit of binary manipulation which some people find confusing.  The code in question is on line 71 and can be found here.  Lets look at this line and walk through its explanation.

b[0] = byte(Version<<4 | (hdrlen >> 2 & 0x0f))

First off, in an IPv4 header, the first byte of data is actually broken up in to two 4 bit fields.  The first 4 bit field is the IP version and the second 4 bit field is the number of 32-bit words in the header.  The smallest header a valid IPv4 packet can have is 5 32-bit words or 20 bytes.

Second, network byte order is big-endian, meaning that the biggest bit values are first, thus:

128 (decimal) = 1000 0000 (binary big-endian)

Lets look at the first part of the code so we can understand this expression:

Version<<4

To figure this out we first need to see that on line #25 the Version constant is defined to be 4.  Thus this expression expands to 4<<4 which is to say 4 times 2, 4 times, or 4*2*2*2*2.  This gives a value of

64 in decimal or 0100 0000 in binary big-endian

Now if you were to look at just the 4 left most bits of that byte, you would see 0100.  And if that is all you had, that would equal 4, which is the IP version.  The reason for the calculations is to get the right bit in this byte set, so that when you break the byte up in to two 4 bit blocks, the values are in the right places.

Now lets look at the second part of the expression which is actually two expressions:

(hdrlen >> 2 & 0x0f)

We see on line #69 that hdrlen is computed, and has a minimum value of 20.  So for the case where hdrlen equals 20 the first part of this expression could be expanded to 20 >> 2 which is to say 20 divided by 2, 2 times, or 20/2/2.  This gives a value of:

5 decimal or 0000 0101 in binary big-endian

Now we have the "&" operator, which is a bitwise AND.  What the bitwise AND gives us, is all of the bits that are a 1 in both values.  So lets look at both sides of the bitwise AND in binary big-endian.

hdrlen >> 2 = 5 = 0000 0101
0x0f (hex) = 15 decimal = 0000 1111

Now lets line up the binary and do the bitwise AND operation, thus giving you just the bits where the values are 1.  This is a type of mask to make sure we ONLY populate the 4 right most bits.  Remember the 4 left most bits are part of the version not part of the header length and we do not want to accidentally have them in this answer.

0000 0101 (header length)
0000 1111 (mask)
==========&
0000 0101

The last thing to do is join the two values together.  Remember the code was:

b[0] = byte(Version<<4 | (hdrlen >> 2 & 0x0f))

To this we use the "|" operator, which is a bitwise OR.  The bitwise OR gives us all of the bits that have a value of 1 in either of the two values.  So the first value is the Version from up above and the second value is the header length that we just did. Lets line them up in binary and compute the bitwise OR.

0100 0000 (version)
0000 0101 (header length)
========|
0100 0101

Thus the first byte of the b[0] of the slice is:

0100 0101 binary big-endian or 69 decimal or 0x45 hex

You can see that if you broke that byte up in to two 4 bit fields you would have:

0100 = IP version 4
0101 = 5 32-bit word header (a header that is 20 bytes in size)

So all of this binary manipulation was done to make sure we could get the right bits in the right parts of the byte so that we can treat a single byte of data as two separate 4 bit fields.

JSON Support for TAXII 1.1

Today on the TAXII discussion list I released v1.00 of the JSON Message Binding Specification for TAXII 1.1.  APIs written in Go, for generating and consuming JSON based TAXII messages can be found here on Github.

Why I switched to Go (golang), the next great programming langauge

I have been asked a lot as of late, why I switched to writing code in Go.  The answer is pretty simple and to quote another developer;

• The language is modern, small, simple and quite strict. There's a minimalism here that I like - what you see is what you get. Some things that wouldn't even merit a warning in other languages (like unused variables) are errors in Go - your code won't even compile. I like the tidiness this promotes.
  
• Awesome concurrency. Go's concept of goroutines and channels is simple, beautiful and works well. This is essential for something like syncthing where there's a lot of stuff going on in parallel.
  
• Simple deployment. Go compiles to a single statically linked binary that you just need to copy to the target system and run. It's trivial to cross compile from one os/architecture into all others supported by the Go compiler.
  
• Modern standard library, "some batteries included". This includes an HTTP server, a clean (non-OpenSSL) crypto and TLS implementation, JSON and XML serializers, etc.
  
• Good enough performance. The Go compiler doesn't generate as fast code as the best C or C++ compilers out there, but it's still faster than interpreted languages.
  
• Tooling and community. Go does things somewhat differently than many other languages and this can be a bit of an acquired taste... But for example the existence and adoption of "go fmt" means there is no discussion about formatting or indenting - there is only one standard. "Go get" simplifies fetching and building, plus results in a standardized repo layout. Etc.
  
• I think it's a really nifty language to work with and IMHO, it is the next great system language.
  
• It has the backing of a fiscally stable company, Google. So if anything it will only increase in popularity.

Sublime Text 3 Auto Complete Theme / Color Changes

After recently switching from Eclipse to Sublime Text 3 for my development work, I found the need to change some of the theme and element coloring options. One of the elements I wanted to change, but could not find any documentation for was the auto complete pop-up window. 

In the following screenshot you can see a pop up window with a list of auto complete options.  Let me explain how you might change these.  First off, the file you will need to edit is this one: ~/Library/Application Support/Sublime Text 3/Packages/Theme - Default/Default.sublime-theme




To change the background for the pop up window you would edit this element:
{
  "class": "popup_control",
  "layer0.tint": [64, 64, 64, 255],
  "layer0.opacity": 1.0,
  "content_margin": [2, 2]
},
{
  "class": "auto_complete",
  "row_padding": [2, 1],

  // White background
  "layer0.tint": [255, 255, 255],
  "layer0.opacity": 1.0,
  "dark_content": false
},


To change the color of text item is the pop up  window you would edit this element:

{
  "class": "auto_complete_label",

  // color of options in pop up window
  "fg": [72, 72, 72, 255],

  // red, the text color that you have typed that matches
  "match_fg": [255, 0, 0, 255],
  "bg": [255, 26, 26],

  // color of the text in the row that is selected 
  "selected_fg": [72, 72, 72, 255],

  // black, color of the matched text in the row that is selected
  "selected_match_fg": [0, 0, 0, 255],   
  "selected_bg": [156, 185, 223, 255]
},


To change the row highlight color you would edit this element

{
  "class": "table_row",
  "layer0.texture": "Theme - Default/row_highlight_wide.png”,

  // Blue highlight for the selected option
  "layer0.tint": [33, 90, 184],
  "layer0.opacity": 0.0,
  "layer0.inner_margin": [1, 1]
},
{
  "class": "table_row",
  "attributes": ["selected"],
  "layer0.opacity": 1.0,
},

Tools for editing PCAP files

I wrote a new command line tool to rebase PCAP files and edit their layer2 and layer3 addresses. This tool is smart enough to edit corresponding ARP packets and understands 802.1Q tagged frames and Q-in-Q double tagged frames.  It should easily compile with Go v1.4 on MacOSX and Linux (it may also compile on Windows though I can not test that). You can get it on GitHub at: https://github.com/jordan2175/rewritecap

Working With Golang Array Slices

In the Go language a Slice is equivalent to the way a Perl/Python developer might think of an Array.  Yes, there is an actual Array under the covers of a Slice, but you cannot do much with them. So in Go, most people works with a Slice.  So if you are new to Go and coming from Perl/Python or the like, when you think you want an Array, you really want a Slice.  

Example

package main

import (
 "fmt"
)

func main() {
 
 // This is how you would declare and initialize a Slice at the same time.
 var aSliceOfStrings1 = []string{"a", "b", "c"}
 fmt.Println(aSliceOfStrings1[1])
 // This will print "b"


 // This is a shorthand way of declaring and initializing a Slice at the same time.
 aSliceOfStrings2 := []string{"c", "d",}
 fmt.Println(aSliceOfStrings2[1])
 // This will print "d" 

 
 // To create a slice but not initialize it you would use the make function.  This allows to define the
 // size and the optional capacity (defaults to length) at the same time.
 aSliceOfStrings3 := make([]string, 5, 10)
 aSliceOfStrings3[0] = "a"
 aSliceOfStrings3[1] = "b"
 aSliceOfStrings3[2] = "c"
 aSliceOfStrings3[3] = "d"
 aSliceOfStrings3[4] = "e"
 fmt.Println(aSliceOfStrings3[0])
 // This will print "a"


 // Append a Slice with more data
 aSliceOfStrings3 = append(aSliceOfStrings3, "f", "g", "h")
 fmt.Println(aSliceOfStrings3[6])
 // This will print "g"

 // Get the length and capacity of a Slice
 l := len(aSliceOfStrings3)
 fmt.Println("Length: ", l)
 // This will print "Length: 8"

 c := cap(aSliceOfStrings3)
 fmt.Println("Capacity: ", c)
 // This will print "Capacity: 10"

 // Copy an element of a Slice
 i := aSliceOfStrings3[1]
 fmt.Println(i)
 // This will print "b"


 // Copy part of a Slice in to another Slice, end needs to be what you want + 1 so if we want [2:4] we need to say [2:5]
 // You can also say [:] to mean the entire Slice. 
 // [:2] means the start of the Slice to the second element. 
 // And [2:] to mean from the second element to the end of the Slice.
 anotherSliceOfStrings1 := aSliceOfStrings3[2:5]
 fmt.Println("Length: ", len(anotherSliceOfStrings1))
 fmt.Println("Capacity: ", cap(anotherSliceOfStrings1))
 fmt.Println(anotherSliceOfStrings1[0])
 fmt.Println(anotherSliceOfStrings1[2])
 // This will print "Length: 3"
 // This will print "Capacity: 8" since we removed the first 2 elements of the original Slice
 // This will print "c" and "e"

 anotherSliceOfStrings2 := aSliceOfStrings3[:2]
 fmt.Println("Length: ", len(anotherSliceOfStrings2))
 fmt.Println("Capacity: ", cap(anotherSliceOfStrings2))
 fmt.Println(anotherSliceOfStrings2[0])
 fmt.Println(anotherSliceOfStrings2[1])
 // This will print "Length: 2"
 // This will print "Capacity: 10" the start of the Slice did not change so the capacity is the same
 // This will print "a" and "b" 

 anotherSliceOfStrings3 := aSliceOfStrings3[4:]
 fmt.Println("Length: ", len(anotherSliceOfStrings3))
 fmt.Println("Capacity: ", cap(anotherSliceOfStrings3))
 fmt.Println(anotherSliceOfStrings3[0])
 fmt.Println(anotherSliceOfStrings3[1])
 // This will print "Length: 4"
 // This will print "Capacity: 6" the start of the Slice changed so the capacity did as well
 // This will print "e" and "f"
}

Passing Slices to Functions
It's important to understand that even though a slice contains a pointer, it is really a value that under the covers is a struct value holding a pointer and a length. It is not a pointer to a struct. 

This matters when you pass a Slice to a function.  When you pass a Slice to a function you can modify the values in the Slice in the function and they will keep, however,  you can not modify the header (aka the length and capacity) as the header was passed by value not as a pointer.  If you want to modify the header, then you need to pass the Slice as a pointer not as a value.    You pass a Slice as a pointer by preceding it with an “&”.  Here is an example to replicate the "shift" function in Perl.

package main

import ("fmt")

var DEBUG int = 0

// This function tries to replicate the shift function in Perl by removing the first
// element of an slice and returning it
func Shift (pToSlice *[]string) string {
    sValue := (*pToSlice)[0]
    if DEBUG == 1 {
        fmt.Println("The slice before the shift: ",  *pToSlice)
    }
    *pToSlice = (*pToSlice)[1:len(*pToSlice)]
    
    if DEBUG == 1 {
        fmt.Println("The slice after the shift: ",  *pToSlice)
    }
    return sValue    
}


func main() {
    var a = []string{"1", "2", "3", "4"}
    for i := range a {
        fmt.Println(a[i])
    }
    // This will print 1, 2, 3, 4


    test := Shift(&a)
    fmt.Println("This is the shifted value: ", test)
    // This will print 1


    test2 := Shift(&a)
    fmt.Println("This is the shifted value: ", test2)
    // This will print 2
    
    for i := range a {
        fmt.Println(a[i])
    } 
    // This will print 3, 4
}