Using AppEngine remote_api in development environment

Note: This is a follow-up of a previous post Locally modifying Go package. As today (May 17, 2014) the changes described there are neccessary for a below code work.

We will modify: datastore_info.go provided as example of remote_api usage in App Engine Go SDK. One should follow the steps in Go SDK doc on remote_api to enable it.

Local client

The trivial function to sign in in a dev server as an admin is below.

func clientLocalLoginClient(host, email string) *http.Client {
 jar, err := cookiejar.New(nil)
 if err != nil {
  log.Fatalf("failed to make cookie jar: %v", err)
 }
 client := &http.Client{
  Jar: jar,
 }
 local_login_url := fmt.Sprintf("http://%s/_ah/login?email=%s&admin=True&action=Login&continue=", host, email)
 resp, err := client.Get(local_login_url)
 if err != nil {
  log.Fatalf("could not post login: %v", err)
 }
 defer resp.Body.Close()

 body, err := ioutil.ReadAll(resp.Body)
 if resp.StatusCode != http.StatusOK {
  log.Fatalf("unsuccessful request: status %d; body %q", resp.StatusCode, body)
 }
 if err != nil {
  log.Fatalf("unable to read response: %v", err)
 }

 m := regexp.MustCompile(`Logged in`).FindSubmatch(body)
 if m == nil {
  log.Fatalf("no auth code in response %q", body)
 }

 return client
}

Connecting to localhost instead a real app requires modifying a main to use the clientLocalLoginClient function if host address points to localhost:

 is_local := regexp.MustCompile(`.*(localhost|127\.0\.0\.1)`).MatchString(*host)
 if !is_local && *passwordFile == "" {
  log.Fatalf("Required flag: -password_file")
 }

 var client *http.Client
 if !is_local {
  p, err := ioutil.ReadFile(*passwordFile)
  if err != nil {
   log.Fatalf("Unable to read password from %q: %v", *passwordFile, err)
  }
  password := strings.TrimSpace(string(p))
  client = clientLoginClient(*host, *email, password)
 } else {
  client = clientLocalLoginClient(*host, *email)
 }

The full source code can be found here: https://gist.github.com/orian/3f74c6add4e4f572e108

The above code can be invoked as follow:

$ goapp run datastore_stats.go -host=localhost:8080 -email=test@example.com

Exporting data to sample app

Sample application with enabled remote_api in Go and data exporter can be found on GitHub github.com/orian/gae-go-remote-api-example. The prerequirement is a configured Google App Engine Go SDK.
Getting and starting the app:

cd workspace/go
git clone git@github.com:orian/gae-go-remote-api-example.git
cd gae-go-remote-api-example
goapp server

This starts app and logs 3 crucial info:

INFO     2014-05-17 21:30:27,120 api_server.py:171] Starting API server at: http://localhost:55542
INFO     2014-05-17 21:30:27,132 dispatcher.py:182] Starting module "default" running at: http://localhost:8080
INFO     2014-05-17 21:30:27,133 admin_server.py:117] Starting admin server at: http://localhost:8000

In another terminal one can:

cd workspace/go/gae-go-remote-api-example/examples
goapp run export_data.go --data_dir data/ -host localhost:8080 -email test@test.com

The terminal output should look similar to:

2014/05/17 23:47:07 appengine: not running under devappserver2; using some default configuration
2014/05/17 23:47:07 App ID "gae-go-boilerplate"
Skip: 
Visited: data/data_item_0.json
Visited: data/data_item_1.json
filepath.Walk() returned    # ironically this is good

This means that data from files data_item_0.json and data_item_1.json has been opened successfully and exported. One can check on admin panel of dev server: http://localhost:8000/datastore?kind=DataItem

Locally modifying Go package

Long story - short:
I'm playing with Google App Engine - Go version. I've tried to use one of the provided libraries and found out it doesn't work as I've expected.
A appengine/remote_api Client doesn't allow to connect to localhost and custom port, only default :80. I found a place in code responsible for handling localhost connection: https://github.com/golang/appengine/blob/a5bf4a208e232b1d3d1c972da47afe05b2c5faa5/remote_api/client.go#L46

    url := url.URL{
        Scheme: "https",
        Host: host,
        Path: "/_ah/remote_api",
    }
    if host == "localhost" {  // here's the reason
        url.Scheme = "http"
    }

then open terminal, go to directory where main go_appengine package is unpacked

cd ~/Downloads/software/go_appengine
find . -name remote_api
vim ./goroot/src/pkg/appengine/remote_api

and replace the above line with:

    if regexp.MustCompile(`^localhost(:\d{1,5})?$`).MatchString(host) {

Check it here: http://play.golang.org/p/fMogPEfgc8
There's one more thing one has to do, install modified package so it's used:

goapp install ./goroot/src/pkg/appengine/remote_api/

After this, if one run's goapp run my_super_tool.go it will use modified code. Pull request to original project.