Thursday, May 11, 2017

Tensorflow custom operation - problem with word2vec operation

This is just a quick note for myself. One beautiful afternoon I got a below error:

tensorflow.python.framework.errors_impl.NotFoundError: 
models/tutorials/embedding/word2vec_ops.so: 
undefined symbol: _ZN10tensorflow16ReadFileToStringEPNS_3EnvERKSsPSs
The distributed package of Tensorflow uses gcc 4.
If you compile from sources on more or less up to date Ubuntu or Debian, you probably will have gcc 5 or newer installed.
gcc 5 and gcc 4 ABI are not compatible, thus your Tensorflow and operations MUST be compiled with same ABI.
You have two choices either compile the Tensorflow with old ABI support by providing
-cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0"
flag to bazel build or ensure your operations are compiled with new ABI.
You can also force gcc5 to use old ABI (source):

TF_INC=$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_include())')
g++ -std=c++11 -shared word2vec_ops.cc word2vec_kernels.cc \
  -o word2vec_ops.so -fPIC -I $TF_INC -O2 -D_GLIBCXX_USE_CXX11_ABI=0

More could be found in Tensorflow doc: https://www.tensorflow.org/install/install_sources#build_the_pip_package

References:

Friday, January 27, 2017

To shadow or to not?

Long story short: it's easy to accidentally shadow a variable by using := operator in if statement. Be aware:

package main

import (
 "fmt"
)

func f() (int, int) { return 2, 2 }

func main() {
 var x = 1
 if x, y := f(); x == y {
  fmt.Printf("%d %d\n", x, y)
 }
 fmt.Printf("%d\n", x)
 x, y := f()
 fmt.Printf("%d %d\n", x, y)
}
The output is:

2 2
1
2 2
Run in Golang Play: https://play.golang.org/p/lymn2gl6Wi.

Saturday, July 9, 2016

JavaScript not evaluating all functions in condition

I’ve just been struck by a simple though nuance code optimization. I’ve met it probably thousands times. In fact, some style guides I’ve followed forbids to do what I’ve just done ;-)

First let me explain what we want to achive. We want to call 3 functions and early return if all 3 returned false. Does the below code look good for you?

if (!func0() && !func1() && !func2()) {
    return;
}

Unfortunately it’s wrong. The JavaScript execution engine will optimize it and if the func0 call returns true (negated by !) then the func1 and func2 won’t be called at all.

This phenomenon has a name and it’s called short-circut evaluation, quoting Wikipedia:

the second argument is executed or evaluated only if the first argument does not suffice to determine the value of the expression

This applies to third and further arguments. Thus the lesson is: never put function calls in logical statements, at least not functions you want guarantee to be executed.

Changed code:

var skip = !func0();
skip &= !func1();
skip &= !func2();
if (skip) {
    return;
}

I’m sure the world is full of this kind of bugs.

Tuesday, May 10, 2016

Postmortem culture

Culture

It happens, something goes wrong, the system goes down, it stops proceeding orders or serve ads or … situation becomes nervous.

I would say one of the main differences how organisation handles those critical situations is a pretty good indicator of it shape. I’ve seen a companies putting the employees under huge pressure during incidents and blaming after.

Then, I’ve worked for Google and… When you make mistake, the impact factor happens to be thousands QPS. You look at graph and see how some stats goes down or crazily spike. You rollback or do emergency release and it starts (ok, it’s more complex, but hey). You (in most cases more than you) are responsible for writing a postmortem.

What is a postmortem?

In short, it’s a document describing what happened. I know in many companies it looks different, many adopted it after some ex-Googler has joined them (can we call it Googleism? or Googleisation?). What is important in Google’ postmortems is it’s purpose and main pur aim of postmortem is: to learn and never repeat old mistakes.

That changes the perspective dramatically. Writing about your own fuckup is not trivial and writing in no-blame way is even harder. It’s not easy even for local stars (local genius theory). How to write postmortem? What to put in it? How it should look? Check out the links

What’s next?

The document usually should be created in next 24-48h after, unless the incident is very complex or not understood. Usually, the PM are open for comments, so everyone may ask a question for some details. When ready and the document went through some ‘peer-review’, it’s should be sent to all interested parties (mailing group, #slack, whatever) and it should be discoverable. Means, it should end up in bug tracker under the issue (because, you have an open issue about the outage, right?). It should be kept in some postmortems’ database. Thus, even if it’s not you who pushed a binary or write the code, you will learn. And in the future, you can always look for similar cases.

Wednesday, May 4, 2016

Read from file: Length-prefixed protocol buffers

File format

In short it’s binary encoded protobuf message prefixed with the size. For longer description please check Length-prefix framing for protocol buffers.

In most languages the file is read as stream. If you need to open and parse a file which holds lenght-prefixed protobuf messages in Python, you read internet and you find a code which read the whole file. It’s huge bottleneck. The hit in a performance between reading whole file then parsing byte after byte and using BufferedReader was in 1000x. Thus enjoy my little piece of code:

def ReadItm(fname, constructor, size_limit = 0):
    ''' Reads and parses a length prefixed protobuf messages from file. 
        The file MUST not be corrupted. The parsing is equivalent to parseDelimitedFrom.
    '''
    f = None
    if fname.endswith('.gzip'):
        f = gzip.open(fname, 'rb')
    else:
        f = open(fname, 'rb')
    reader = BufferedReader(f)
    bytes_read = 0
    while size_limit<=0 or bytes_read<size_limit:
        buffer = reader.peek(10)
        if len(buffer) == 0:
            break
        (size, position) = decoder._DecodeVarint(buffer, 0)
        reader.read(position)
        itm = constructor()
        itm.ParseFromString(reader.read(size))
        bytes_read = bytes_read + position + size
        yield itm
    f.close()