Tuesday, May 10, 2016

Postmortem culture

Culture

It happens, something goes wrong, the system goes down, it stops proceeding orders or serve ads or … situation becomes nervous.

I would say one of the main differences how organisation handles those critical situations is a pretty good indicator of it shape. I’ve seen a companies putting the employees under huge pressure during incidents and blaming after.

Then, I’ve worked for Google and… When you make mistake, the impact factor happens to be thousands QPS. You look at graph and see how some stats goes down or crazily spike. You rollback or do emergency release and it starts (ok, it’s more complex, but hey). You (in most cases more than you) are responsible for writing a postmortem.

What is a postmortem?

In short, it’s a document describing what happened. I know in many companies it looks different, many adopted it after some ex-Googler has joined them (can we call it Googleism? or Googleisation?). What is important in Google’ postmortems is it’s purpose and main pur aim of postmortem is: to learn and never repeat old mistakes.

That changes the perspective dramatically. Writing about your own fuckup is not trivial and writing in no-blame way is even harder. It’s not easy even for local stars (local genius theory). How to write postmortem? What to put in it? How it should look? Check out the links

What’s next?

The document usually should be created in next 24-48h after, unless the incident is very complex or not understood. Usually, the PM are open for comments, so everyone may ask a question for some details. When ready and the document went through some ‘peer-review’, it’s should be sent to all interested parties (mailing group, #slack, whatever) and it should be discoverable. Means, it should end up in bug tracker under the issue (because, you have an open issue about the outage, right?). It should be kept in some postmortems’ database. Thus, even if it’s not you who pushed a binary or write the code, you will learn. And in the future, you can always look for similar cases.

Wednesday, May 4, 2016

Read from file: Length-prefixed protocol buffers

File format

In short it’s binary encoded protobuf message prefixed with the size. For longer description please check Length-prefix framing for protocol buffers.

In most languages the file is read as stream. If you need to open and parse a file which holds lenght-prefixed protobuf messages in Python, you read internet and you find a code which read the whole file. It’s huge bottleneck. The hit in a performance between reading whole file then parsing byte after byte and using BufferedReader was in 1000x. Thus enjoy my little piece of code:

def ReadItm(fname, constructor, size_limit = 0):
    ''' Reads and parses a length prefixed protobuf messages from file. 
        The file MUST not be corrupted. The parsing is equivalent to parseDelimitedFrom.
    '''
    f = None
    if fname.endswith('.gzip'):
        f = gzip.open(fname, 'rb')
    else:
        f = open(fname, 'rb')
    reader = BufferedReader(f)
    bytes_read = 0
    while size_limit<=0 or bytes_read<size_limit:
        buffer = reader.peek(10)
        if len(buffer) == 0:
            break
        (size, position) = decoder._DecodeVarint(buffer, 0)
        reader.read(position)
        itm = constructor()
        itm.ParseFromString(reader.read(size))
        bytes_read = bytes_read + position + size
        yield itm
    f.close()

Thursday, April 7, 2016

Go: imagick image croping after rotation problem

I need to load image rotate it by some angle, then crop some part of it. Seems trivial, isn't?

I use: gopkg.in/gographics/imagick.v2/imagick package.

Load image
iwand := imagick.NewMagickWand()
defer iwand.Destroy()
if err := iwand.ReadImage(s); err != nil {
 log.Panicf("cannot open image %s", err)
}
Rotate it, fill new area with yellow
w := iwand.GetImageWidth()
h := iwand.GetImageHeight()
log.Printf("old size: %d,%d", w, h)
pwand := imagick.NewPixelWand()
pwand.SetColor("yellow")
if err := iwand.RotateImage(pwand, 45); err != nil {
 log.Panicf("problem with rotation: %s", err)
}
Calculate the position and size of the rectangle to cut
newW := calcSize(w, h)
newH := newW

w = iwand.GetImageWidth()
h = iwand.GetImageHeight()
log.Printf("new size: %d,%d", w, h)

x := int((w - newW) / 2)
y := int((h - newH) / 2)
Make the area red (with transparency), just for learning
dwand := imagick.NewDrawingWand()
pwand.SetColor("red")
pwand.SetOpacity(0.5)
dwand.SetFillColor(pwand)
dwand.Rectangle(float64(x), float64(y), float64(newW+uint(x)), float64(newH+uint(y)))
iwand.DrawImage(dwand)
Crop and save to file
log.Printf("w,h,x,y: %d,%d,%d,%d", newW, newH, x, y)
if err := iwand.CropImage(newW, newH, x, y); err != nil {
 log.Fatalf("problem with crop: %s", err)
}
iwand.WriteImage(d)

Input file
Original image (from: https://en.wikipedia.org/wiki/Linux)

Output file
Image rotated and cropped

Need to repage!
log.Printf("w,h,x,y: %d,%d,%d,%d", newW, newH, x, y)
iwand.ResetImagePage("")
if err := iwand.CropImage(newW, newH, x, y); err != nil {
 log.Fatalf("problem with crop: %s", err)
}
iwand.WriteImage(d)
Rotated and cropped - fixed
Full source: gist

Friday, March 11, 2016

Android developer road

So, I'm going through a quick Android developer course. The main target is to learn how to provide the best Android apps with minimal effort. Assumption is we all know Java ;-)

The start

A good starting point is:
http://developer.android.com/training/index.html

I don't recommend to read it all at the beginning. I would suggest:
Others as needed, Best Practices for Interaction and Engagement (http://developer.android.com/training/best-ux.html) also helps.

The above gives you a rough overview. The real stuff comes in API Guides (http://developer.android.com/guide/components/index.html). I do recommend to read the first three sections:
This gives you a high level overview of how the Android works.

The lifecycle

You basically should have it at your desk. You will use it often. It's been reported to have some inaccuracies but it's still better than the Google provided simplified version.



Codelabs

Google provides two interesting codelabs:
If you have to provide some early results quickly, skip above readings and just do the codelab.

Other must-to-know

Butterknife - http://jakewharton.github.io/butterknife/ - saves you lot of boilerplate with binding view elements to variables.
Android ContentProvider Generator - it will save you a lot of time on boilerplate code - https://github.com/BoD/android-contentprovider-generator
Mosby - http://hannesdorfmann.com/mosby - if you go with MVP this is must to use.
Dagger - dependency injection framework - http://google.github.io/dagger/users-guide

And when you consider which Android version to support, check:
Android stats Dashboard - gives an overview of the Android version distribution - http://developer.android.com/about/dashboards/index.html

Learning

Monday, February 8, 2016

No, because no

I need Math.Round from Go's standard library, poor me:


I know it's hard. But hey, most languages provide it.

Rob Pike:
Floating point numbers don't have digits.
The bar for being useful needs to be pretty high to be in the Go math package. We accepted most of what's in the C library but we're not going to take all the routines from other libraries. The Javascript round is a 1-liner: int(f+0.5).