Make it work - IntelliJ IDEA and Linux (Xmonad)

IntelliJ (and few other Java programs) has a problem with window not appearing or being frozen. Simple solution:

$ wmname LG3D

More info:

• https://bbs.archlinux.org/viewtopic.php?id=142063 which suggest export _JAVA_AWT_WM_NONREPARENTING=1
• https://code.google.com/p/android/issues/detail?id=57675
• http://awesome.naquadah.org/wiki/Problems_with_Java

Spark Core without Internet and Cloud

I recently got a Spark Core. I don't find a need to connect my device to cloud. What's even more I don't want to do that. My home automation system is suppose to be cloud free. I want to make it a loose connection of Spark Core (in future Photon P0/P1) as controllers and data collectors, and Raspberry Pi as a brain device (exposing API or syncing with cloud of my choice).
First, one needs to remember that Spark is an embed device. System is linked with one's application. Every time one wants to update a program, one needs to recompile firmware (compile one's application as a compilation unit and link with the rest of firmware).

Installing firmware

The firmware repository is located here: https://github.com/spark/firmware Following instruction: https://github.com/spark/firmware#1-download-and-install-dependencies on Ubuntu means:

sudo apt-get install gcc-arm-none-eabi # wrong version
sudo apt-get install automake
sudo apt-get install dfu-util # wrong version
sudo apt-get install git

Do NOT do it at home. One needs to install gcc by:

sudo apt-get remove binutils-arm-none-eabi gcc-arm-none-eabi
sudo add-apt-repository ppa:terry.guo/gcc-arm-embedded
sudo apt-get update
sudo apt-get install gcc-arm-none-eabi

More about problems with gcc for arm can be read:

• https://answers.launchpad.net/gcc-arm-embedded/+question/253484
• https://bugs.launchpad.net/gcc-arm-embedded/+bug/1309060 SOLUTION
• https://community.spark.io/t/how-to-install-the-spark-toolchain-in-ubuntu-14-04/4139
• https://bugs.archlinux.org/task/39004
• http://askubuntu.com/questions/502815/should-arm-none-eabi-gcc-include-a-stdio-h

and dfu-util from source (a dfu-util build manual http://dfu-util.sourceforge.net/build.html):

sudo apt-get install libusb-1.0-0-dev
wget http://dfu-util.sourceforge.net/releases/dfu-util-0.8.tar.gz
tar -zxvf dfu-util-0.8.tar.gz dfu-util-0.8
cd dfu-util-0.8
./autogen.sh
# you need to have autoreconf, if error, try: sudo aptitude install dh-autoreconf
./configure
make
sudo make install

Reasons:

• gcc-arm-none-eabi is missing nano.specs and <cctype> header file used in core-firmware/inc/spark_wiring_character.h this will cause your compilation to throw errors like this:

  4.8 arm-none-eabi-g++: error: nano.specs: No such file or directory
  

• Ubuntu have dfu-util version 0.5, the most recent version is 0.8. In an old version running dfu-util -l will give you:

  Found DFU: [1d50:607f] devnum=0, cfg=1, intf=0, alt=0, name="UNDEFINED"
  Found DFU: [1d50:607f] devnum=0, cfg=1, intf=0, alt=1, name="UNDEFINED"
  

Notice "UNDEFINED". Using new version will print something more similar to:

Found DFU: [1d50:607f] devnum=0, cfg=1, intf=0, alt=0, name="@Internal Flash  /0x08000000/20*001Ka,108*001Kg" 
Found DFU: [1d50:607f] devnum=0, cfg=1, intf=0, alt=1, name="@SPI Flash : SST25x/0x00000000/512*04Kg"

But what's more, the old version is not able to flash Spark Core. Flashing and listing devices required to run it with root privilages (sudo).

Getting source

Note, that makefile for firmware assumes that all 3 projects are downloaded/cloned to the same directory: core-firmware, core-common-lib, core-communication-lib.

FDU mode

Flash it part of firmware manual requires you to put Core in FDU mode. If you wonder how DFU mode on Core looks: https://vine.co/v/MahhI1Fg7O6:

Google: "GoLang compute md5 of file"

Title of this post is my last Google query.
It got me there: https://www.socketloop.com/tutorials/how-to-generate-checksum-for-file-in-go. After reading I was surprise that Go doesn't have ReaderWriter which reads data from some object implementing io.Reader and writes data to some other object implementing io.Writer. I've grabbed the code and started cleaning it but keeping all file content in memory made me search.
I went to GoLang official doc and found: func Copy(dst Writer, src Reader) (written int64, err error)
Copy copies from src to dst until either EOF is reached on src or an error occurs.
To summarize:

• Go defines interface hash.Hash which is implemented in packages: crypto/hmac, crypto/md5, crypto/sha1, crypto/sha256, crypto/sha512.
• hash.Hash inherits io.Writer, this mean it has Write method.
• One should use io.Copy to copy data between reader and writer.

Below is the full source of main.go:

package main

import (
  "crypto/md5"
  "fmt"
  "io"
  "os"
)

func ComputeMd5(filePath string) ([]byte, error) {
  var result []byte
  file, err := os.Open(filePath)
  if err != nil {
    return result, err
  }
  defer file.Close()

  hash := md5.New()
  if _, err := io.Copy(hash, file); err != nil {
    return result, err
  }

  return hash.Sum(result), nil
}

func main() {
  if b, err := ComputeMd5("main.go"); err != nil {
    fmt.Printf("Err: %v", err)
  } else {
    fmt.Printf("main.go md5 checksum is: %x", b)
  }
}

$ go run main.go
main.go md5 checksum is: facd74ec8975d8fd84897fb352f8f87e

Google Drive API - searching

Part1

Search for an item by name

List files

Google Drive API is a REST API. One have a resources which can be created, changed or removed.
In Google Drive both files and directories are represented as a file. To look for a file with a specific name we have to use List method of a resource File.
Documentation: Files.List
When programming in Go, it's worth to look at implementation: Drive API in Go
One is interested in FilesService and search for a method List on it. The method returns FilesListCall which have methods allowing to set different query parameters.
Simple code getting all files:

func GetAllFiles(srv *drive.Service) ([]*drive.File, error) {
  return d.Files.List().Do()
}

According to the documentation, the List method by default returns all files on Drive limited by a maxResults parameter. One can change the limit by calling: d.Files.List().MaxResults(10). The default value is 100, and possible values are between 0 and 1000. If there are more files to list, the method returns a valid PageToken string in reponse which can be used in a following requests.
One can copy from documentation an example code in Go which handles PageToken:

// AllFiles fetches and displays all files
func AllFiles(d *drive.Service) ([]*drive.File, error) {
  var fs []*drive.File
  pageToken := ""
  for {
    q := d.Files.List()
    // If we have a pageToken set, apply it to the query
    if pageToken != "" {
      q = q.PageToken(pageToken)
    }
    r, err := q.Do()
    if err != nil {
      fmt.Printf("An error occurred: %v\n", err)
      return fs, err
    }
    fs = append(fs, r.Items...)
    pageToken = r.NextPageToken
    if pageToken == "" {
      break
    }
  }
  return fs, nil
}

Few words about results. The API returns []*drive.File. It's worth to take a look at a documentation: File and a code: type File struct in API source code.
The final code looks like: gist.github.com/orian/6a0d7883ca3678cb30ea

Search for a file with a specific name

Files don't have a name per se, the name shown in a drive.google.com is an attribute title of a resource File: File reference.
To search only files with a specific name we need to use q parameter. Go API allows to do that through Q(string) method. Example code below:

func FindFile(srv *drive.Service, name string) ([]*drive.File, error) {
  q := fmt.Sprintf("title = '%s'", name)
  return Files.List().Q(q).Do()
}

Search for a directory

The directory is a Google drive file with a special mimetype: 'application/vnd.google-apps.folder'. To search for a directory with a specific name we need to extend the previous code and a query parameter by "mimeType = 'application/vnd.google-apps.folder'".

func FindDir(srv *drive.Service, name string) ([]*drive.File, error) {
  q := fmt.Sprintf("mimeType = 'application/vnd.google-apps.folder' and title = '%s'", name)
  return Files.List().Q(q).Do()
}

Search for a directory knowing its parent id

If a name is not an identifier of file on Drive than what? FileId is an unique id given to each file on Google Drive. It's available as Id field of struct drive.File. If we look at search documentation: search parameters we can find parents property on which we can use operator in. E.g. when we have a folder id 1234 we can require a file to be in folder by writing '1234 in parents' as query.

func FindSubDir(srv *drive.Service, name, parentId string) ([]*drive.File, error) {
  subq := []string{
      "mimeType = 'application/vnd.google-apps.folder'", 
      fmt.Sprintf("title = '%s'", name),
      fmt.Sprintf("'%s' in parents", parentId),
  }
  q := strings.Join(subq, " and ")
  return Files.List().Q(q).Do()
}

Writing Google Drive upload application

This day has to come. I came back from holidays and have few thousands photos. Many of them to throw away but most of them to keep, share and print.
The photos were took by a standalone camera and backed up on a hard drive.
The tries of using Google Photos Backup for Mac OS, plus.google.com, drive.google.com were for me dissapointing. The first was stalled after few dozen of photos and no restart helped. The other ones also crashed and were rather slow.

After creating an app I was able to upload without significant problems over 2 thousand photos and counting.
The app works as follow:

• Authorize
• Find a directory on Google Drive, if not exist then create one.
• Get a list of all files from a destination directory.
• Scan a local directory to find files and for each:
• check if a name matches a pattern
• check if not already on Google Drive
• upload

Authentication & authorization

Authentication identifies your application and informs Google that you are you. To create simple authentication and authorization code follow: Google Drive API - Quickstart for Go
This bootstrap our efforts so we have working program which authenticats and authorizes itself with a help of user.

Caution! The quickstart example (I guess some dependencies) requires Go version 1.3. It doesn't work with 1.2 and earlier (default version in Ubuntu 14.04 package repo as August 2014).

Authorization gives your app a credential to act as a specific Google user. Google is recommending using OAuth2.0. The data you want to access is covered by a scope. In a case of Google Drive and accessing/modifying content it's 'https://www.googleapis.com/auth/drive'.

This is already done by a quickstart app from the above tutorial. The only think we want to modify is to cache an access token so we don't have to ask user for Drive permission every time.

Caching the user's access token

The OAuth2 library we are using already have a support for saving a token. There is a interface Cache and a simple implementation CacheFile:
https://godoc.org/code.google.com/p/goauth2/oauth#Cache

First, we will separate code responsible for a creating token and transport.

func GetNewToken() (*oauth.Token, *oauth.Transport) {
 // Generate a URL to visit for authorization.
 authUrl := config.AuthCodeURL("state")
 log.Printf("Go to the following link in your browser: %v\n", authUrl)
 // Read the code, and exchange it for a token.
 log.Printf("Enter verification code: ")
 var code string
 fmt.Scanln(&code)

 t := &oauth.Transport{
  Config:    config,
  Transport: http.DefaultTransport,
 }
 token, err := t.Exchange(code)
 if err != nil {
  log.Fatalf("An error occurred exchanging the code: %v\n", err)
 }
 return token, t
}

The example which tries to load a token from file and if cannot then request a new one may looks as follow:

var cache oauth.Cache = oauth.CacheFile("access_token.json")
token, err := cache.Token()
var t *oauth.Transport
if err != nil {
 log.Printf("Need a new token. Cannot load old one.")
 token, t = GetNewToken()
 cache.PutToken(token)
} else {
 t = &oauth.Transport{
  Config:    config,
  Token:     token,
  Transport: http.DefaultTransport,
 }
}

Full source code: gist.github.com/orian/96b5140b66363f4dee65

Parallel programming notes

OpenMP

Introduction to OpenMP - Tim Mattson (Intel):
https://www.youtube.com/watch?v=jfQLD2AGSvc&list=PLLX-Q6B8xqZ8n8bwjGdzBJ25X2utwnoEG&index=0
OpenMP - Quick Reference Card:
http://openmp.org/mp-documents/OpenMP-4.0-C.pdf

CUDA

Measuring performance of CUDA operations
http://devblogs.nvidia.com/parallelforall/how-implement-performance-metrics-cuda-cc/
Monitoring GPU usage

$ nvidia-sli -l -lms 1000
+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0            Not Supported                                               |
+-----------------------------------------------------------------------------+
Wed Sep 10 23:51:15 2014       
+------------------------------------------------------+                       
| NVIDIA-SMI 340.29     Driver Version: 340.29         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 770     Off  | 0000:01:00.0     N/A |                  N/A |
| 66%   80C    P0    N/A /  N/A |   1279MiB /  2047MiB |     N/A      Default |
+-------------------------------+----------------------+----------------------+

..to be continued..

Using AppEngine remote_api in development environment

Note: This is a follow-up of a previous post Locally modifying Go package. As today (May 17, 2014) the changes described there are neccessary for a below code work.

We will modify: datastore_info.go provided as example of remote_api usage in App Engine Go SDK. One should follow the steps in Go SDK doc on remote_api to enable it.

Local client

The trivial function to sign in in a dev server as an admin is below.

func clientLocalLoginClient(host, email string) *http.Client {
 jar, err := cookiejar.New(nil)
 if err != nil {
  log.Fatalf("failed to make cookie jar: %v", err)
 }
 client := &http.Client{
  Jar: jar,
 }
 local_login_url := fmt.Sprintf("http://%s/_ah/login?email=%s&admin=True&action=Login&continue=", host, email)
 resp, err := client.Get(local_login_url)
 if err != nil {
  log.Fatalf("could not post login: %v", err)
 }
 defer resp.Body.Close()

 body, err := ioutil.ReadAll(resp.Body)
 if resp.StatusCode != http.StatusOK {
  log.Fatalf("unsuccessful request: status %d; body %q", resp.StatusCode, body)
 }
 if err != nil {
  log.Fatalf("unable to read response: %v", err)
 }

 m := regexp.MustCompile(`Logged in`).FindSubmatch(body)
 if m == nil {
  log.Fatalf("no auth code in response %q", body)
 }

 return client
}

Connecting to localhost instead a real app requires modifying a main to use the clientLocalLoginClient function if host address points to localhost:

 is_local := regexp.MustCompile(`.*(localhost|127\.0\.0\.1)`).MatchString(*host)
 if !is_local && *passwordFile == "" {
  log.Fatalf("Required flag: -password_file")
 }

 var client *http.Client
 if !is_local {
  p, err := ioutil.ReadFile(*passwordFile)
  if err != nil {
   log.Fatalf("Unable to read password from %q: %v", *passwordFile, err)
  }
  password := strings.TrimSpace(string(p))
  client = clientLoginClient(*host, *email, password)
 } else {
  client = clientLocalLoginClient(*host, *email)
 }

The full source code can be found here: https://gist.github.com/orian/3f74c6add4e4f572e108

The above code can be invoked as follow:

$ goapp run datastore_stats.go -host=localhost:8080 -email=test@example.com

Exporting data to sample app

Sample application with enabled remote_api in Go and data exporter can be found on GitHub github.com/orian/gae-go-remote-api-example. The prerequirement is a configured Google App Engine Go SDK.
Getting and starting the app:

cd workspace/go
git clone git@github.com:orian/gae-go-remote-api-example.git
cd gae-go-remote-api-example
goapp server

This starts app and logs 3 crucial info:

INFO     2014-05-17 21:30:27,120 api_server.py:171] Starting API server at: http://localhost:55542
INFO     2014-05-17 21:30:27,132 dispatcher.py:182] Starting module "default" running at: http://localhost:8080
INFO     2014-05-17 21:30:27,133 admin_server.py:117] Starting admin server at: http://localhost:8000

In another terminal one can:

cd workspace/go/gae-go-remote-api-example/examples
goapp run export_data.go --data_dir data/ -host localhost:8080 -email test@test.com

The terminal output should look similar to:

2014/05/17 23:47:07 appengine: not running under devappserver2; using some default configuration
2014/05/17 23:47:07 App ID "gae-go-boilerplate"
Skip: 
Visited: data/data_item_0.json
Visited: data/data_item_1.json
filepath.Walk() returned    # ironically this is good

This means that data from files data_item_0.json and data_item_1.json has been opened successfully and exported. One can check on admin panel of dev server: http://localhost:8000/datastore?kind=DataItem

Locally modifying Go package

Long story - short:
I'm playing with Google App Engine - Go version. I've tried to use one of the provided libraries and found out it doesn't work as I've expected.
A appengine/remote_api Client doesn't allow to connect to localhost and custom port, only default :80. I found a place in code responsible for handling localhost connection: https://github.com/golang/appengine/blob/a5bf4a208e232b1d3d1c972da47afe05b2c5faa5/remote_api/client.go#L46

    url := url.URL{
        Scheme: "https",
        Host: host,
        Path: "/_ah/remote_api",
    }
    if host == "localhost" {  // here's the reason
        url.Scheme = "http"
    }

then open terminal, go to directory where main go_appengine package is unpacked

cd ~/Downloads/software/go_appengine
find . -name remote_api
vim ./goroot/src/pkg/appengine/remote_api

and replace the above line with:

    if regexp.MustCompile(`^localhost(:\d{1,5})?$`).MatchString(host) {

Check it here: http://play.golang.org/p/fMogPEfgc8
There's one more thing one has to do, install modified package so it's used:

goapp install ./goroot/src/pkg/appengine/remote_api/

After this, if one run's goapp run my_super_tool.go it will use modified code. Pull request to original project.

GCC and vectorization

#include<ctime>
#include<iostream>
#include<memory>

using namespace std;

class Clock {
 public:
  Clock() : start_time_(std::clock()) {}

  std::clock_t Now() const { return std::clock() - start_time_; }

  double NowSeconds() const {
    return static_cast<double>(std::clock() - start_time_) / CLOCKS_PER_SEC;
  }

  void Reset() { start_time_ = std::clock(); }

 private:
  std::clock_t start_time_;
};

void Func(int* a, int *b, int* c, int size) {
  for(int i=0; i<size; ++i) {
    *a = *b * (*c);
    ++a; ++b; ++c;
  }
}

void FuncForSIMD(int* __restrict__ a, int* __restrict__ b, int* __restrict__ c,
                 int size) {
  for(int i=0; i<size; ++i) {
    *a = *b * (*c);
    ++a; ++b; ++c;
  }
}

int main() {
  constexpr int size = 1000000000;
  unique_ptr<int> a(new int[size]);
  unique_ptr<int> b(new int[size]);
  unique_ptr<int> c(new int[size]);
  Clock clock;
  Func(a.get(), b.get(), c.get(), size);
  cout << "time without simd: " << clock.Now() << endl;
  clock.Reset();
  FuncForSIMD(a.get(), b.get(), c.get(), size);
  cout << "time with simd: " << clock.Now() << endl;
  return 0;
}

To compile we need to run:

g++ -std=c++0x simd.cc -o simd -O3

The output on my machine looks as follow:

time without simd: 2160000
time with simd: 390000

So it gives 5.5x faster execution time.
Great explanation is given here:
Demystifying The Restrict Keyword - http://cellperformance.beyond3d.com/articles/2006/05/demystifying-the-restrict-keyword.html

Arduino IDE and Linux

After connecting Arduino UNO to Linux powered PC I've encountered two simple problems but which required from me to search for few minutes, less linuxy users may not found a solution, so here it goes.

Not existing device

Board at COM1 is not available

and during uploading

Arduino: 1.5.5 (Linux), Board: "Arduino Uno"

/home/pawel/software/arduino/arduino-1.5.5/hardware/tools/avr/bin/../lib/gcc/avr/4.3.2/../../../avr/lib/avr5/crtm328p.o: In function `__bad_interrupt':
../../../../crt1/gcrt1.S:193: undefined reference to `main'

One may need to create a symlink:

sudo ln -s /dev/ttyACM0 /dev/ttyCOM1

Or one should choose a correct port in Arduino IDE Tools->Port menu.

Permissions

avrdude: ser_open(): can't open device "/dev/ttyACM0": Permission deniedioctl("TIOCMGET"): Inappropriate ioctl for device

where /dev/ttyACM0 is my Arduino serial connection.

1. check user/group of device
   

   ls -l /dev/ttyACM0
   crw-rw---- 1 root dialout 166, 0 Jan 21 21:18 /dev/ttyACM0
   

2. add user 'pawel' to group 'dialout'
   

   sudo usermod -a -G dialout pawel