In English on oinume journal

Resolving 3rd party proto file on IntelliJ IDEA

Thu, 25 Jun 2020 09:00:00 +0900

I use Protobuf Support Plugin for syntax highlighting. However, it can't resolve 3rd party proto file like this:

In my case, the file which is not resolved is located in proto/third_party directory.

Solution

Open Project Structure (File -> Project Structure or Cmd + ; )
Choose the directory as Resource root

Finally, unresolved error was gone 🎉🎉🎉

Satisfying a large interface quickly in Go

Mon, 12 Nov 2018 09:00:00 +0900

Sometimes it's painful to safisfy a large interface in Go. Here is a simple answer for this, just embed interface on struct like bellow:

package main

import (
    "fmt"
)

type Foo interface {
    MethodA()
    MethodB()  
    MethodC()
    MethodD()   
}

type FooImpl struct {
    Foo
}

func (fi *FooImpl) MethodA() {
    fmt.Println("MethodA called")
}

func main() {
    foo := new(FooImpl)
    foo.MethodA() // Implemented
    foo.MethodB() // Not implemented, runtime error will happen
}

You can check the result on https://play.golang.org/p/0y8ICTWCfpy.

Generating an unpredictable random value in Go

Mon, 14 May 2018 09:00:00 +0900

There are a lot of examples to use math/rand. However, should use crypto/rand if you want to generate an unpredictable random value. That's because crypto/rand uses getrandom(2) if available, /dev/urandom otherwise on Linux.

As a real world example, UUID v4 uses crypto/rand internally. UUID v4) is randomly generated. An implementation of UUID v4 in Go uses crypto/rand internally.

Here is an example to generate random values with crypto/rand.

Generate random integer

func generateInt(n int64) (int64, error) {
    r, err := rand.Int(rand.Reader, big.NewInt(n))
    if err != nil {
        return 0, err
    }
    return r.Int64(), nil
}

Generate random string

func generateString(length int) (string, error) {
    b := make([]byte, length)
    _, err := rand.Read(b)
    if err != nil {
        return "", err
    }
    return base64.URLEncoding.EncodeToString(b), nil
}

Whole source code

package main

import (
    "crypto/rand"
    "encoding/base64"
    "fmt"
    "log"
    "math/big"
)

func main() {
    v, err := generateInt(1000)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("random int = %v\n", v)

    s, err := generateString(10)
    if err != nil {
        log.Fatal(err)
    }
    fmt.Printf("random string = %v\n", s)
}

func generateInt(n int64) (int64, error) {
    r, err := rand.Int(rand.Reader, big.NewInt(n))
    if err != nil {
        return 0, err
    }
    return r.Int64(), nil
}

func generateString(length int) (string, error) {
    b := make([]byte, length)
    _, err := rand.Read(b)
    if err != nil {
        return "", err
    }
    return base64.URLEncoding.EncodeToString(b), nil
}

go test in practice

Thu, 13 Apr 2017 09:00:00 +0900

This blog post describes basics and practical examples of go test. Go’s automated test mechanism is well designed and easy to use. I’ll show you some techniques of go test in this article.

What is `go test`

go test is a command to run automated tests of go packages.

Automated test must be defined in _test.go file
Automated test function must start with Test prefix

Example: stack

I implement a stack as real world example to explain go test. stack package provides stack.go. If you want to write tests for stack.go, create a file named stack_test.go. The source code is in GitHub repository.

Arguments to defered functions are evaluated when the defer executes

Mon, 23 May 2016 19:00:00 +0900

The arguments to the deferred function (which include the receiver if the function is a method) are evaluated when the defer executes, not when the call executes.

Effective Go - The Go Programming Language

I sometimes make a mistake that defered functions are evaluated when the function is called but it’s wrong. Here is an example.

package main

import (
        "fmt"
        "time"
)

type wrapped func()

func main() {
        wrapper(func () {
                time.Sleep(time.Second * 3)
                fmt.Println("Hello")
        })
}

func wrapper(f wrapped) {
        fmt.Println("wrapper() start: ", time.Now().UTC())
        defer instrument(time.Now().UTC())
        f()
        fmt.Println("wrapper()   end: ", time.Now().UTC())
}

func instrument(start time.Time) {
        fmt.Println("instrument():    ", start)
}

$ go run defer.go
wrapper() start:  2016-05-23 07:28:37.474091714 +0000 UTC
Hello
wrapper()   end:  2016-05-23 07:28:40.474272645 +0000 UTC
instrument():     2016-05-23 07:28:37.474139821 +0000 UTC

If defer functions are evaluated at call time, instrument() time must be 3 seconds later.

3 things I do after installing Nodejs

Sat, 09 Apr 2016 21:43:56 +0900

Installing latest npm

$ npm install -g npm@latest

Disabling npm’s progress bar

Disabling progress bar for faster npm install.

$ npm set progress=false

See Progress bar noticeably slows down npm install · Issue #11283 · npm/npm

Installing `npm-check`

npm-check checks outdated, incorrect, and unused dependencies.

$ npm install -g npm-check

www.npmjs.com

Go Tip: Don't take the address of loop variable

Sat, 09 Apr 2016 12:00:00 +0900

stackoverflow.com

If you take the address of loop variable, it may cause a bug which all values are same. You might expect following program prints 5 10 15 but it prints 15 15 15. That’s because loop variable v is initialized just once.

package main

import "fmt"

func main() {
    coll := []int{5, 10, 15}
    for _, v := range toPointers(coll) {
        fmt.Printf("%v ", *v)
    }
}

func toPointers(ints []int) []*int {
    pointers := make([]*int, len(ints))
    for i, v := range ints {
        pointers[i] = &v // XXX: mistake
    }
    return pointers
}

You can get expected result if you write like this:

First loaded configuration becomes default_server in nginx

Mon, 04 Apr 2016 12:00:00 +0900

How nginx processes a request

server {
    listen      80;
    server_name example.org www.example.org;
    ...
}

server {
    listen      80;
    server_name example.net www.example.net;
    ...
}

server {
    listen      80;
    server_name example.com www.example.com;
    ...
}

In this configuration nginx tests only the request’s header field “Host” to determine which server the request should be routed to. If its value does not match any server name, or the request does not contain this header field at all, then nginx will route the request to the default server for this port. In the configuration above, the default server is the first one — which is nginx’s standard default behaviour.

Sorting processes by memory usage

Mon, 28 Mar 2016 14:26:03 +0900

$ ps aux --sort -rss

Sorted by ascending order if you don't add -.

alvinalexander.com

Making an enviroment to learn ES6 with babel

Mon, 21 Mar 2016 22:57:27 +0900

This is a just memo for me who is a beginner of front-end development.

First of all, install nodejs v4.4.0.

$ brew install homebrew/versions/node4-lts
$ node -v
v4.4.0

Put package.json into a current directory. DO NOT forget to add babel-preset-es2015.

{
  "name": "hello",
  "version": "1.0.0",
  "engines": {
    "node": "4.4.0",
    "npm": "3.8.2"
  },
  "devDependencies": {
    "babel-cli": "^6.0.0",
    "babel-preset-es2015": "^6.6.0"
  }
}

Run npm install

Parsing MySQL's URL in Python3

Mon, 14 Dec 2015 00:32:58 +0900

Just add urllib.parse.uses_netloc.append("mysql") if you want to parse URL such as mysql://root:pass@localhost/demo.

#!/usr/bin/env python

import urllib.parse
urllib.parse.uses_netloc.append("mysql")

if __name__ == "__main__":
    url_str = "mysql://root:pass@localhost/demo"
    url = urllib.parse.urlparse(url_str)
    print("URL={0}".format(url_str))
    print("host={0}, user={1}, password={2}, database={3}".format(
        url.hostname, url.username, url.password, url.path[1:]))

$ python3 url_parse.python
URL=mysql://root:pass@localhost/demo
host=localhost, user=root, password=pass, database=demo

Sending pull-request only with terminal and keyboard

Wed, 28 Oct 2015 09:00:00 +0900

Requirements

Mac
Terminal
hub command (brew install hub)

How to send pull-request

Make a new branch.

$ git checkout -b new-cool-feature

Editing source code and commit.

$ git commit -a

Push to remote and copy correct git push command into clipboard.

$ git push 2>&1 | grep git | pbcopy

Paste clipboard into terminal.

$ git push --set-upstream origin new-cool-feature

Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 324 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
To git@github.com:example/example-project.git
 * [new branch]      new-cool-feature -> new-cool-feature
Branch new-cool-feature set up to track remote branch new-cool-feature from origin.

Send pull-request by hub.

Logging into dmm.com by Selenium

Mon, 26 Oct 2015 08:50:00 +0900

I use Selenium to log into dmm.com because dmm.com requires JavaScript enabled browser. Here is a sample code. You can see Firefox is launched and automatically logging into dmm.com if you use Firefox instead of PhantomJS.

login_dmm.py

#!/usr/bin/env python
# encoding: utf-8

import os
from selenium import webdriver

driver = webdriver.PhantomJS()
#driver = webdriver.Firefox()
driver.get("https://www.dmm.com/my/-/login/=/path=DRVESVwZTldRDlBRRFdIUwwIGFVfVEs_")
login_id = driver.find_element_by_id("login_id")
login_id.send_keys(os.environ["DMM_LOGIN_ID"])
password = driver.find_element_by_id("password")
password.send_keys(os.environ["DMM_LOGIN_PASSWORD"])
driver.find_element_by_tag_name("form").submit()
print(driver.page_source)

driver.quit()

Run

$ brew install phantomjs
$ pip install selenium
$ DMM_LOGIN_ID=your_id DMM_LOGIN_PASSWORD=your_password python ./login_dmm.py

Get auto increment values from MySQL information schema

Wed, 21 Oct 2015 09:00:00 +0900

Information schema is metadata about MySQL. We can obtain useful information from it. For example, I get current auto increment values by following SQL.

SELECT
  t.table_name, t.auto_increment
FROM
  information_schema.tables AS t
WHERE
  t.table_schema = 'your_db'
ORDER BY t.table_name;

+-------------------------------------+----------------+
| table_name                          | auto_increment |
+-------------------------------------+----------------+
| blog_categories                     |              4 |
| blog_posts                          |              4 |
| blog_post_categories                |              4 |
| blog_post_comments                  |              1 |
| images                              |              7 |
+-------------------------------------+----------------+

INFORMATION_SCHEMA database has much more useful tables like COLUMNS, GLOBAL_VARIABLES and so on. It's good to use it when you want to know much about MySQL.

What I read, watched, listened recently

Sat, 03 Oct 2015 23:04:09 +0900

dotGo 2014 - Andrew Gerrand - 5 things I love

Andrew Gerrand is a Googler and worked for Go. I watched this video just to learn English but it was good that he was describing Go's feature very well.

www.youtube.com

Golang UK Conference 2015 - Andrew Gerrand - Stupid Gopher Tricks

Another Andrew's talk. Slide

www.youtube.com

Twelve Go Best Practices - Francesc Campoy

Francesc Campoy is also a Googler and worked for Go. Best practices from Googler is worth to read. Slide

Duplicating a MySQL table schema and data

Fri, 04 Sep 2015 20:31:19 +0900

Duplicate foo table into foo_20150904

/* Create a new table from foo's schema */
> CREATE TABLE foo_20150904 LIKE foo;

/* Insert whole data into a new table */
> INSERT INTO foo_20150904 SELECT * FROM foo;

Detecting duplicated code in Golang with CPD and Jenkins

Sun, 30 Aug 2015 00:40:00 +0900

CPD (Copy Paste Detecor)

Detecting duplicated code is a good way to make source code clean. Since I couldn't find how to detect duplicated code in Golang, I sent a pull request that CPD(Copy Paste Detector) is able to accept Golang. CPD is a software included in PMD.

And there is a useful Jenkins plugin, DRY. It visualizes how much duplicated code exist. I'll show you how to make it visible with Jenkins.

Make VirtualBox's network faster with paravirtualized network adapter

Tue, 28 Jul 2015 08:30:00 +0900

I develop applications with Vagrant and VirtualBox and feel frustrated about slowness of VirtualBox. Since I found virtio (paravirtualized network adapter), I tried it out on my application.

How to enable virtio with Vagrant

Add following lines into your Vagrantfile.

v.customize ["modifyvm", :id, "--nictype1", "virtio"]
v.customize ["modifyvm", :id, "--nictype2", "virtio"]

Results

I ran time vagrant provision with old and new settings and virtio-net is 1.2x faster than 82540EM (default network adapter).

vagrant provision 14.85s user 12.49s system 5% cpu 8:20.59 total with Intel PRO/1000 MT Desktop (82540EM)
vagrant provision 14.63s user 12.62s system 6% cpu 7:00.70 total with virtio-net

Building Vagrant box from VirtualBox OVF

Mon, 29 Jun 2015 06:30:00 +0900

VirtualBox Builder (from an OVF/OVA) - Packer by HashiCorp www.packer.io

I realized that Packer can build Vagrant box from VirtualBox OVF file not only from ISO image. Building Vagrant box takes much time to run an installer so I'm very happy to skip the step.

I created Ansible installed Vagrant box from another box to try.

Run 'vagrant box add oinume/ubuntu-14.04-jp'
Downloaded box is stored in ~/.vagrant.d/boxes/oinume-VAGRANTSLASH-ubuntu-14.04-jp/1.0.1/virtualbox and box.ovf is in the directory.
Specify the path of .ovf like below in template.json and run 'packer build template.json'
'ubuntu-14-04-x64-jp-ansible-virtualbox.box' created.

Mocking a HTTP access with http.Transport in Golang

Mon, 13 Apr 2015 09:00:00 +0900

I've investegated a way to mock HTTP request and response when writing a code to access to a HTTP API with http.Client. Finally, I found that I can do it by just implementing RoundTrip method of http.RoundTripper.

Here is an example of mocking and proxying HTTP request and response. Running with go run main.go -mock mock, you can get mocked result and with go run main.go -mock proxy, you can get proxied result with httptest.Server.

Changing bower package cache directory

Thu, 26 Feb 2015 23:30:00 +0900

bower caches packages in ~/.cache/bower. How can I change the directory? bower's document says "Add the following in .bowerrc" but it's totally wrong.

"storage": {
  "cache" : "~/.bower/cache",
  "registry" : "~/.bower/registry"
}

I try the configuration and run bower install but nothing changed. I finally found out correct configuration in bower/config repo.

"storage": {
  packages: ".bower/packages",
  registry: ".bower/registry"
}

Why the document of famous software such as bower is wrong?

Recent open source activities

Sat, 08 Nov 2014 20:19:51 +0900

pmd

https://github.com/pmd/pmd/pull/44

CPD (Copy Paste Detector) supported GoLang.

go-xorm

https://github.com/go-xorm/xorm/issues/166

Suggested and implemented "soft delete" for xorm. My pull-request was immediately merged! (very impressive)

goose

https://bitbucket.org/liamstask/goose/pull-request/43/support-mysql-driver

Added mysql driver support. My pull-request hasn't been merged yet. I contacted to the author but he didn't respond.

Make Javas VM's GC log human readable with -XX:+PrintGCDateStamps

Sat, 27 Jul 2013 23:00:00 +0900

I use Java VM's option -XX:+PrintGCTimeStamps in order to output time in GC log of Java VM , Just like this.

TODAY=`date "+%Y%m%d-%H%M%S"`

JAVA_OPTS="-server -Xms512m -Xmx512m -Xmn256m -XX:PermSize=256m -XX:MaxPermSize=256m \
 -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:+UseParNewGC \
 -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=32 -XX:TargetSurvivorRatio=85 \
 -verbose:gc -Xloggc:/usr/local/tomcat/logs/gc.log.$TODAY \
 -XX:+PrintGCDetails \
 -XX:+PrintGCTimeStamps \
 -DJENKINS_HOME=/var/lib/jenkins \
"

But, the output is relative time from starting time of Java VM. So it's difficult to read. However, we have an another option -XX:+PrintGCDateStamps !! It is an option to output absolute time!! (Available from Java6 u6)

ruby-lint - Static syntax checker for Ruby

Sat, 27 Jul 2013 22:47:00 +0900

Recently I found static syntax checker for Ruby called ruby-lint. We haven't had such a syntax checker for Ruby, so I'm very happy to find it. (Although it's alpha quality)

Installation

$ gem install ruby-lint

Try

Prepare following ruby code.

#!/usr/bin/env ruby
# -*- coding: utf-8 -*-
a = 'hoge'
puts "hello" + b

ruby-lint it!!

$ ruby-lint hoge.rb
hoge.rb: error: line 4, column 15: undefined local variable or method b
hoge.rb: warning: line 3, column 0: assigned but unused local variable a

I cheer an author of ruby-lint. This is a great software!