Blog (ruby) - Fred Wu's Tech

Conference Talk: Adaptable Human @ RubyConf China 2019

ifredwu@gmail.com (Fred Wu) — Sat, 14 Sep 2019 08:05:09 +0000

Seven years ago in 2012 I spoke at RubyConf China 2012. It was a technical talk on how to become a better developer, if you’re interested you can check out the video recording.

Seven years later in August 2019, I headed back to Shanghai again to speak at this year’s RubyConf China. And this time around, it was a non-technical talk on how to develop one’s career. Check out the video recording below:

video: https://www.youtube.com/watch?v=MBczdO7RgNo

And, here are some photos of yours truly. :)

]]>

I Accidentally Some Machine Learning - My Story of A Month of Learning Elixir

ifredwu@gmail.com (Fred Wu) — Sat, 23 Jul 2016 18:33:29 +0000

About a month ago I was in-between jobs - I had two weeks to rest up, recharge and get ready for my new job. So I thought, I should use those two weeks to learn something new.

Years ago I briefly looked into Elixir when it was first released to the wild, at the time I wasn’t interested in picking it up due to its syntax similarity to Ruby, despite their vastly different underlying semantics. I love Ruby, and it’s been my weapon of choice for the past 6-7 years, so when it came time for me to learn something new, I naturally wanted to learn something a bit more different than Ruby, syntax-wise.

Fast-forward a few years, I am more mature and open-minded, and are now in a position to welcome Elixir and to embrace the Ruby-like syntax as well as the functional programming mindset with open arms.

Learning Elixir

Given the strong influence of Ruby in Elixir, I can’t help but get nostalgic about learning it by reading another great book by Dave Thomas: Programming Elixir.

Dave Thomas’ original Programming Ruby was instrumental in the success of the Ruby programming language and its communities in the west, and it certainly has helped me to greatly widen my exposure to not only the wonderful world of Ruby but object-oriented programming in general as a PHP developer.

Learning Elixir has been really fun, not only does it share the same developer-friendliness championed by Ruby, it also does so with remarkable strength in concurrency thanks to BEAM (Erlang’s VM).

Many people are drawn to Elixir due to its Ruby influence, its functional and immutability nature, and its Actor-based concurrency model. I would however like to call out one of my favourite features of Elixir that sometimes gets overlooked, and that is how you could write unit tests using ExUnit.DocTest.

Elixir Makes Writing Unit Tests Effortless

Python programmers have been writing doctests for years, but Ruby due to not having an equivalent standard library, has never had writing doctests taken off in its community. I’m glad Elixir has.

Take a look at the code snippet below:

defmodule Stemmer.Step0 do
  @doc """
  ## Examples

      iex> Stemmer.Step0.trim_apostrophes("'ok")
      "ok"

      iex> Stemmer.Step0.trim_apostrophes("o'k")
      "o'k"

      iex> Stemmer.Step0.trim_apostrophes("'o'k'")
      "o'k"
  """
  def trim_apostrophes(word) do
    word
    |> String.replace_prefix("'", "")
    |> String.replace_suffix("'", "")
  end
end

The three test cases given will actually be tested by ExUnit, provided you ask it to do so as below:

defmodule Stemmer.Step0Test do
  use ExUnit.Case, async: true

  doctest Stemmer.Step0
end

You may think having doctests is not a big deal, what I can say is that I am really enjoying the fact that unit tests can be written with minimal friction and high visibility (as they sit with the implementation, rather than within a big test suite with a sea of files). Besides, not all of us practice TDD religiously or 100% of the time, and when I don’t or can’t do TDD, having doctests ensures I don’t forget adding test cases. :)

Toy Robot in Elixir

The best way to learn something new is to practice it as you learn it. Instead of diving straight into a complicated system, or to write a hello world program, I thought I would dig out a code test and re-implement it in Elixir.

The code test is the rather infamous Toy Robot test. As an interviewer, I must have reviewed a few hundreds of these tests, most of which in Ruby. I know what a good OO solution looks like, so naturally I was looking forward to “rethink” the problem and have a crack at it using Elixir.

Here is the result: https://github.com/fredwu/toy-robot-elixir

As a learner, I wanted to practice as many language features as possible, so included in the Toy Robot test code I tried:

Different data structures such as Struct and List
Pattern matching
Data piping (|>)
Agent for managing states
Macros for validation rules

I kept the test code small and nimble on purpose to illustrate the readability of a highly expressive language. I may not have exercised all the language features (such as supervisors and protocols), but at that point I felt I could start the Elixir journey with a big smile on my face already.

Learning Phoenix

I have to admit, a big part of the reason that got me started on learning Elixir is to build a side project. Initially I wanted to simply use Ruby and Rails because I can be very productive using them, but in the end I decided on learning Elixir and Phoenix, because most side projects fail, and when they do, how much learning can you extract out of those experiences? My answer to this question, is to learn a new language, a new framework and most importantly a new programming paradigm to drastically increase the amount of learning I could gain.

But please, before you jump on Phoenix, learn Elixir properly first! Over the years I’ve seen far too many cases of people jumping on Rails without understanding the language features of Ruby…

To learn Phoenix, the Programming Phoenix book is pretty much the bible on this subject aside from the official guide, and it is written by Chris McCord, the author of Phoenix, Bruce Tate, and Jose Valim, the author of Elixir.

As an experienced Rails developer, it only took me a day or two to go through the book (I largely skimmed the chapters on Channels as I don’t intend to build a real-time app).

Now, as a Ruby developer, the biggest pain point when working on Rails or other sizeable Ruby MVC projects for me, is ActiveRecord. I got so frustrated to the status quo I even attempted to fix it.

Fortunately, in Elixir and Phoenix we have Ecto. One of my favourite features of Ecto is the concept of Changeset.

Take a look at the code snippet below:

defmodule MySecretApp.User do
  use MySecretApp.Web, :model

  schema "users" do
    field :username, :string
    field :email, :string
    field :password, :string, virtual: true
    field :encrypted_password, :string

    has_one :profile, MySecretApp.Profile

    timestamps()
  end

  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:username, :email])
    |> validate_required([:username, :email])
    |> validate_length(:username, min: 3, max: 20)
    |> validate_format(:username, ~r/\A[a-zA-Z0-9_]+\z/, message: "alphanumeric and underscores only")
    |> validate_format(:email, ~r/@/)
    |> unique_constraint(:username)
    |> unique_constraint(:email)
  end

  def creation_changeset(struct, params \\ %{}) do
    struct
    |> changeset(params)
    |> password_changeset(params)
  end


  defp password_changeset(struct, params) do
    struct
    |> cast(params, [:password])
    |> validate_required([:password])
    |> validate_length(:password, min: 8, max: 200)
    |> encrypt_password
  end

  defp encrypt_password(changeset) do
    case changeset do
      %Ecto.Changeset{valid?: true, changes: %{password: password}} ->
        put_change(changeset, :encrypted_password, Comeonin.Bcrypt.hashpwsalt(password))
      _ ->
        changeset
    end
  end
end

The changset/2 function can be used when a user needs to be updated, whereas the creation_changeset/2 is to be used only when a user is first created. Sure, you can achieve similar result in Rails by using custom validators, but the fact that this practice is enforced by the library and the framework, is encouraging.

One other thing that I’ve seen in larger Rails apps, is the leaky abstraction of view-level logic, they typically sit in controllers, helpers (which are globally available) or worse, models. Phoenix in this case follows what Hanami, Trailblazer and many other frameworks do: it introduces a “view model” layer.

Something along the lines of:

defmodule MySecretApp.UserView do
  use MySecretApp.Web, :view

  def full_name do
    "#{title} #{first_name} #{last_name}"
  end
end

In a nutshell, Phoenix is just like Rails, but less magical (in a good way) and faster. :) Like the Elixir eco-system, most libraries and concepts feel like their ruby/rails counterparts, but more refined.

There are a few popular Hex packages if you’re familiar with their ruby counterparts:

	Elixir	Ruby
Code analysis	credo	rubocop
Testing	espec	rspec
Browser testing	wallaby	capybara
Test coverage	excoveralls	simplecov
Test factory	ex_machina	factory_girl
Authentication	guardian	devise

And the list goes on and on… Be sure to check out Awesome Elixir for more community curated Elixir libraries.

Learning Machine Learning

As I started building the foundation of my side project, you know, the usual user management and session management, etc, etc, by chance I came across mentions of Bayesian inference.

One thing led to another, very soon I started looking into varies different machine learning algorithms such as Naive Bayes and Random Forest. These are all subjects I had never come across or even heard of before, as I have no strong mathematics, statistics or computer science background. I was however, intrigued and inspired nonetheless, and wanted to employ some machine learning in my side project.

And the best way to understand and learn about machine learning algorithms? You guessed, is to write one! After some research, I came to the conclusion that Naive Bayes is one of the simplest to implement, is great for text classification which is useful for my side project, and has good accuracy given its simplicity and fast speed.

Introducing Simple Bayes

So, after a few train rides to and from work, on my newly purchased Macbook (yes, the one with only a single USB-C port), I built a Naive Bayes library: Simple Bayes.

As of writing, this is the only Naive Bayes library in Elixir that supports the following features:

Naive Bayes algorithm with different models
- Multinomial
- Binarized (boolean) multinomial
- Bernoulli
No external dependencies
Ignores stop words
Additive smoothing
TF-IDF
Optional keywords weighting
Optional word stemming via Stemmer

Let’s see it in action shall we? :)

SimpleBayes.init
|> SimpleBayes.train(:ruby, "I enjoyed using Rails and ActiveRecord for the most part.")
|> SimpleBayes.train(:ruby, "The Ruby community is awesome.")
|> SimpleBayes.train(:ruby, "There is a new framework called Hanami that's promising.")
|> SimpleBayes.train(:ruby, "Please learn Ruby before you learn Rails.")
|> SimpleBayes.train(:ruby, "We use Rails at work.")
|> SimpleBayes.train(:elixir, "It has Phoenix which is a Rails-like framework.")
|> SimpleBayes.train(:elixir, "Its author is a Rails core member, Jose Valim.")
|> SimpleBayes.train(:elixir, "Phoenix and Rails are on many levels, comparable.")
|> SimpleBayes.train(:elixir, "Phoenix has great performance.")
|> SimpleBayes.train(:elixir, "I love Elixir.")
|> SimpleBayes.train(:php, "I haven't written any PHP in years.")
|> SimpleBayes.train(:php, "The PHP framework Laravel is inspired by Rails.")
|> SimpleBayes.classify("I wrote some Rails code at work today.")
# => [
# ruby: 0.20761437345986136,
# elixir: 0.08101868169313056,
# php: 0.019047884912605735
# ]

As you can see, provided with reasonable training data, Naive Bayes can work extremely well.

Introducing Stemmer

Something I discovered as I was building Simple Bayes, is something called stemming. Let’s see another example:

SimpleBayes.init(stem: false)
|> SimpleBayes.train(:apple, "buying apple")
|> SimpleBayes.train(:banana, "buy banana")
|> SimpleBayes.classify("buy apple")
# => [
# banana: 0.057143654737817205,
# apple: 0.057143654737817205
# ]

Oops, the probabilities of the sentence “buy apple” of being apple and banana are the same, that’s not good, as we know “buying” and “buy” should mean the same thing. This is where stemming comes in handy. Let’s enable stemming and run the example again:

SimpleBayes.init(stem: true)
|> SimpleBayes.train(:apple, "buying apple")
|> SimpleBayes.train(:banana, "buy banana")
|> SimpleBayes.classify("buy apple")
# => [
# apple: 0.18096114003107086,
# banana: 0.1505149978319906
# ]

Stemming in this case correctly identified “buy” and “buying” as the same thing (they both have the stemmed root of “buy”).

How did I include stemming in Simple Bayes you wonder? Let me introduce you to Stemmer - as of writing this is the only Porter2 stemmer available in Elixir. :)

Elixir and BEAM may not be known for their raw performance, but compared to a similar Porter2 stemmer implemented in pure ruby, my Elixir version runs about five times faster (under 5s to stem 29000+ words, compared to 25s with the ruby version).

The Learning Continues…

I’ve only been writing Elixir for a month, and there are still heaps to learn and to practice. So far though, I have to say my experience with both Elixir and Phoenix has been fantastic - I get the same satisfying and pumped feeling I got when I was learning Ruby and Rails years ago.

If you haven’t checked out Elixir, I strongly encourage you to do so, after all, the world seems to be moving in the concurrency direction at an increasingly rapid speed, including Ruby 3x3. :)

]]>

Protip: Faster Ruby Tests with DatabaseCleaner and DatabaseRewinder

ifredwu@gmail.com (Fred Wu) — Wed, 18 Sep 2013 06:50:00 +0000

Please also see this blog post on tweaking your ruby GC settings.

I use and love DatabaseCleaner, although historically I had never paid too much attention on the performance of its varies cleaning strategies - I’d always used truncation.

We use Postgres, and after digging around and finding out the difference between DELETE and TRUNCATE, I ended up improving our test suite speed by about 30-40% simply by tweaking the cleaning strategies.

RSpec.configure do |config|
  config.before :suite do
    DatabaseCleaner.clean_with :truncation
    DatabaseCleaner.strategy = :transaction
  end

  config.before do
    if example.metadata[:js] || example.metadata[:type] == :feature
      DatabaseCleaner.strategy = :deletion
    else
      DatabaseCleaner.strategy = :transaction
      DatabaseCleaner.start
    end
  end

  config.after do
    DatabaseCleaner.clean
  end
end

Essentially, we want to truncate the DB only once before the whole suite runs to ensure a clean slate DB, then we only want to use deletion on Capybara tests, everything else should just use transaction which is the fastest strategy.

Now, as a bonus, I have just discovered @amatsuda’s DatabaseRewinder which is a lightweight alternative that supports only ActiveRecord. It offers comparable performance with a much similar API.

RSpec.configure do |config|
  config.before :suite do
    DatabaseRewinder.clean_all
  end

  config.after do
    DatabaseRewinder.clean
  end
end

By the way, we also use parallel_tests to scale our test suite to multiple processes, even on Travis CI and Wercker.

Hooray to faster tests! :)

]]>

Protip: Ruby Devs, Please Tweak Your GC Settings for Tests!

ifredwu@gmail.com (Fred Wu) — Fri, 06 Sep 2013 11:13:37 +0000

It was made apparent to me that many ruby devs either aren’t aware or couldn’t be bothered to tweak their ruby garbage collector settings.

Well, if you are using MRI, please start tweaking your GC settings. Here’s what I use (on my 15” Macbook Pro Retina):

export RUBY_GC_MALLOC_LIMIT=90000000
export RUBY_FREE_MIN=200000

Not only can you tweak it for your local dev machine, you can also tweak Jenkins, Travis CI, Wercker and other CI solutions, making instant speed gain for your test suite!

Here’s what we get:

            Before After
Local:      8m22s 5m52s
Travis CI:  10m10s 6m0s
Wercker:    8m45s 6m24s

YMMV depending on your system and available RAM.

]]>

Gotchas in the Ruby Sequel Gem

ifredwu@gmail.com (Fred Wu) — Wed, 21 Aug 2013 12:39:00 +0000

I haven’t really used Sequel much therefore I am definitely a newbie. However, after days and nights of frustration, endless debugging and some search-fu during the development of Datamappify, I have finally arrived at the conclusion that Sequel is a capable library, as long as you are aware of the gotchas.

Gotcha 1: Always use “`select`“/“`select_all`“, or your data records will mysteriously have wrong IDs!

In ActiveRecord, joining an associated model couldn’t be simpler:

Post.joins(:author)

In Sequel, despite having a similar API for models to declare associations and their corresponding primary and foreign keys, you cannot do a join without specifying the keys:

Not good:

Post.join(:authors)
# or
Post.join(Author)

Better:

Post.join(:authors, :id => :author_id)

You would think the version above works - it doesn’t. Even worse, the above example will give you incorrect data - the IDs of the Post records will now contain the IDs from their corresponding Author records! This is because upon a join, Sequel merges attributes from both models into a single hash.

The correct version:

Post.join(:authors, :id => :author_id).select(:posts __id, :posts__ title, :posts__body)
# or
Post.join(:authors, :id => :author_id).select_all(:posts)

Gotcha 2: Always call “`all`“ at the end of the chain, or the chain will present data in a different format.

In ActiveRecord, all of the below examples return an ActiveRecord::Relation collection:

Post.where(:title => 'Hello world')
Post.joins(:author)
Post.includes(:author)

And indeed, calling first on any of them returns an object of class Post (assuming the result collection is not empty).

Post.where(:title => 'Hello world').first.class #=> Post
Post.joins(:author).first.class #=> Post
Post.includes(:author).first.class #=> Post

In Sequel, the below examples all return a Sequel::DataSet collection:

Post.where(:title => 'Hello world')
Post.eager(:author)
Post.eager_graph(:author)

But let’s see what we get from calling first.class on them:

Post.where(:title => 'Hello world').first.class #=> Post
Post.eager(:author).first.class #=> Post
Post.eager_graph(:author).first.class #=> Hash

Huh? Last one is a Hash? It turns out, if you call all at the end of chains to convert them to Arrays, then the returned collections are consistent:

Post.where(:title => 'Hello world').all.first.class #=> Post
Post.eager(:author).all.first.class #=> Post
Post.eager_graph(:author).all.first.class #=> Post

]]>

The Future of Computing, The Future of Computer Programmers - An Interview with Yukihiro "Matz" Matsumoto

ifredwu@gmail.com (Fred Wu) — Sat, 29 Jun 2013 12:04:00 +0000

A while ago I translated an interview with Matz done by a Chinese book publisher. The interview and the translation were well received, so this time I am translating another interview with Matz, done by Ito, the editor-in-chief from Japanese website Engineer Type. Since I don’t read Japanese, the translation is based on Turing Book’s Chinese translation.

The Chinese translator has done a great job translating the interview, but there are still many words and sentences lack sufficient context and therefore are difficult to grasp. I have put in many hours translating the text as well as doing researches to ensure the final article is readable. I hope you will enjoy it! :)

Ito: Thank you for doing an interview with us, Matz. I have just finished reading your latest book The Future of Computing, could you perhaps talk about the future of programming and software programmers in general?

Matz: Hmmm, this is difficult to answer… but thanks for reading my book!

Ito: In the book you’ve shared your thoughts on the past, present and future of different programming languages and software design patterns. Would you like to talk about the current state of the software industry? And is there going to be another paradigm shift in software development?

Matz: As discussed in my book, to predict the future of a high tech industry such as computing is not particularly difficult. I believe in the foreseeable future the computing industry is still going to advance based on Moore’s law. Although, it is possible that in the next year or two quantum computers become a practical reality, in that case it will change everything! *chuckles* On a serious note, according to Moore’s law, the cost of computing will decrease and the performance and capacity of computing will increase - this basic principle is unlikely to change. One thing I did notice in recent years is that due to the advancement in computer hardware, the software industry is subtly changing too.

Software Development in the Era of Multi-core and Cloud Computing

Matz: It was about twenty years ago (in 1993) I invented the Ruby programming language, yet it still runs surprisingly well on modern computers.

What this means is that in the past twenty years the computing environment which the software runs on did not see any fundamental changes. In recent years, we started seeing computing power being shifted from having higher CPU frequencies to being distributed over more CPU cores. And that means software needs to move in that direction too.

Matz: Software has not seen major changes for years.

Ito: And this is covered in the last chapter from the book, right?

Matz: Yes. Similarly to multi-core, cloud computing is advancing in the same direction. The future of computing is all about utilise multiple CPUs or computers effectively.

Ito: So, how does that change software development?

Matz: In the past ten years or so we have been seeing more and more things happen on the Internet, and the Internet is an amazing application platform for extension and distribution. Compared to software engineers working on mainframe computers, web developers are naturally more familiar with the concepts of multi-core and cloud computing.

Ito: After interviewing many web and mobile startups, we realised that the number of software engineers working in PaaS and cloud computing have been increasing rapidly.

Matz: Absolutely. And I do believe that “not needing to purchase and own dedicated hardware” is going to be the mainstream. The idea and thought process of “not owning” is not only important for software development, but also important for business development.

“Owning” Becoming a Liability, Not Asset

Matz: In the past, “owning” was seen as the source of vitality of a corporation - those who own high performance mainframe computers were able to do business transactions in high volume whereas those who do not were not able to compete.

These days the landscape is changing - those corporations who do not “own” expensive hardware have more competitive edge. Let’s say it takes five years to break even from the expensive investment of servers, during that time those machines are being put in use to realise their full potential and to justify their cost. It may appear to have saved the business cost but it is not, simply because the value of the hardware decreases as each day passes by.

To put simply, we are now entering the era of “owning” being a liability rather than asset. If you had the most advanced hardware, software engineers were able to develop efficiently. On the contrary, if you didn’t, then you might want to get used to the hours-long waiting for the code to compile. *chuckles* The rise of cloud computing platforms like Heroku is making “owning” a thing of the past.

Also, “not owning” has several advantages on the development as well as the commercial front. For instance, it allows many startups to rise. In the past, in order to start a new business you would need capital for purchasing servers and/or renting servers in a data centre. These days, to get started on a platform like Heroku couldn’t be easier, for example on Heroku you could start with just one dyno for free. This new way of developing software significantly reduces costs and risks.

Years ago I read an essay called Ramen Profitable by Y Combinator’s founder Paul Graham. “Not owning”’s flexibility and agility contribute a great deal to it. And this trend has now grown beyond just relevant to startups, in fact in the recent years many large corporations have begun adapting this approach too.

In the United States, corporations like Disney and Best Buy are indeed utilising Ruby, Rails and Heroku to rapidly grow their internal infrastructure in a cost-effective fashion. What was once considered competitive edges to venture capitalists, like “rapid development” and “development flexibility” are now also possible for these giant corporations.

Ito: What about the giant corporations in Japan?

Matz: I have never worked in a big corporation so I can’t tell where they are heading. People have been optimistic, though as an observer I am concerned.

Real Benefits of Innovation in Cloud Computing Not to Be Overlooked

Ito: What makes you concerned about software development?

Matz: The traditional approach of developing software is still the norm. For example, some corporations, even though use Amazon Web Services, still rely on system administrators to handle their infrastructure. It is too common to see a software development team consists of over a dozen people.

This in my opinion defeats the purpose and forfeits the benefits of “not owning” servers. There are simply too many of these case studies whereby only on the surface of cloud computing is explored and understood.

I have to say I am disappointed by some of the so-called “private clouds” owned by large corporations. The advantage of cloud computing is to utilise multiple computers in the cloud, but those private clouds are essentially their internal data centres. Isn’t that the same as owning a bunch of servers?

Matz: Many companies barely scratch the surface of emerging technologies.

Ito: Indeed, it is too often to see the real benefits of emerging technologies overlooked or misunderstood. Anything else that makes you concerned about the future?

Matz: Nowadays the speed of development has always been a priority from big development projects in B2B to small development projects in many startups. Yahoo! Japan even coined a term “爆速化” (explosively high speed) to indicate the importance of development speed in the ever more competitive and engaging markets.

Looking at things this way, those so-called “system integrators” are becoming obsolete. Should they just give up what they do or continue? I don’t know, but I do know that the gap between them and engineers who have the capability and skills to create real value is increasing.

Career Longevity of Software Engineering

Ito: Who are those engineers who have the capability and skills to create real value?

Matz: The ones who would put in effort to create software or systems from a prototype to a final product. And this has nothing to do with whether they work in web or system integration, or whether it’s consumer oriented or corporate oriented.

Matz: Unbalanced skill combination leads to a gloomy future.

Ito: Do you mean the engineers who are capable from design to implementation?

Matz: Yes. Speaking of which, software developers have to know more than just system design - they cannot survive without knowing how to code. Just like in life, you cannot survive without being down to earth. *chuckles*

Despite the fact that it is pointless to have someone doing only the system design, and not the development as well, the System Integration industry is still going strong in Japan - and it is in fact an industry with high profit margin.

Even if the system designers came up with questionable specifications, or if the programmers were sloppy so the software was terrible to use, users would still use it despite whining. Flaws are easily glossed over under high profit margins.

But just as discussed before, as development speed increases, profit margins would undoubtedly become smaller. Flaws are therefore harder to gloss over.

In my opinion, if things don’t change, those run-of-the-mill software engineers might not survive in five years. Worse, the junior to mid-level to senior programmer corporate ladder is going to collapse.

Say, you wouldn’t want to start a VHS rental shop when DVDs were on the rise, would you?

Difference Between Those Who Control Their Destiny to Those Who Don’t

Ito: Do you have any advice for those who do not wish to be in a “gloomy future”? What can they do?

Matz: To innovate and to create new things, I suppose.

It’s not all doom and gloom. Even though many ageing technologies have been or are being replaced by the web, jobs will not disappear overnight. I think many software developers will still be employed in those jobs.

Having said that, it is always good to create new things or even invent new programming languages.

Ito: What are these “new things”?

Matz: I see three types of new things.

First of all, new services. If you can create a new service, or a service that offers superior user experience - it would be an innovation.

Secondly, new technologies. To come up with technologies better than the existing ones - and this is what I have been doing.

mruby was released earlier this year on Github.

Or, another way is to invent new algorithms.

The three ways I mentioned have different difficulties, but they share the same goal - to create something that hasn’t existed yet. Those who keep working on these kinds of challenges are the true outstanding software engineers.

The ones who do not challenge themselves to create new things are often falling behind - they learn a hip new language today and try a new web framework tomorrow, but still lack the foresight to invent and to improve.

Of course, it is important to learn and try new things, but if you see them as your ultimate goal then you will lose control of your destiny. I believe that the ones who do not get boggled down in every new trendy thing will ultimately be happier.

Software Development is a Punch to Deficiency

Ito: Here is a sharp question: be a follower rather than an inventor is always easier and perhaps makes more money too. What makes you keep inventing?

Matz: My standard answer would be “because writing and running new programs make me happy”. But the real reason is because I don’t like deficiency.

There are people who have different opinions and thought processes, I would often come up with questions like “why was it done this way” or “this will be too hard to use”.

Matz dislikes deficiency, so he invented the ruby programming language.

Ito: True, but all products more or less reflect their producers’ preferences, right?

Matz: Absolutely, and I am not saying that this is bad or anything. I just hate to point fingers at other people’s preferences - if you don’t like something, make your own! This is a basic trait of a good software engineer, and is what makes open source sustainable.

In open source projects, all the source code is publicly available therefore it is very easy to see how a program is designed. As long as you have ideas on how to improve and optimise the design, you are welcome to do so.

Now it is an entirely different story for certain things in the society. *chuckles* At least in software development, we can rely on our skills and knowledge to improve and to change. If it’s your own creation, it can be adjusted and adapted to suit the ever changing needs.

This is same for Ruby - I like programming languages and more importantly I like improving programming languages myself, and that’s why I still work on Ruby till this day.

Software Development, One of the Rare Careers that Could Make a Change on Your Own

Matz talking about developer happiness, wearing his “Ruby City MATSUE“ polo shirt.

Matz: I think I have the right personality for developing software. Only the software industry can tolerate my carefreeness - am I too arrogant for saying that? *chuckles*

In all honesty, software development is one of the rare careers that could bring positive changes to the society on your own. It’s a wonderful occupation that brings happiness and fulfillment!

Ito: Many people would predict their software future based on theories, but Matz you always use “happiness”.

Matz: That’s right. Because only you can control your own destiny. It doesn’t matter if you were told to do things in a way just becasue “Matz said so” - ultimately, I cannot be responsible for your destiny. You should make your own decisions.

I would still say things like “the future might look like this”, but these are just my personal opinions.

And this is the same even for today’s discussion - if someone thinks he does not agree with what Matz has said, he should follow his own decision and the path he chooses.

Exploring the Future: “You” Are the Only Constant

Ito: Having read The Future of Computing, I remember you talked about the inception and development of varies programming languages. But we all know that the IT industry is moving in a rapid pace, it is difficult to rely on history to guide us through to the future. If multi-core and cloud computing are only just the beginning of a paradigm shift, why did you write about the things happened in the past?

Matz: Technologies progress just like a pendulum clock.

Matz: People see things differently - and I believe the IT industry is progressing in a manner similar to the swing motion of a pendulum clock.

As more and more new programming languages, techniques and frameworks pop up, software development related technologies are progressing whilst seeking for balance.

So, how does “the most balanced from the past” become “the most balanced right now”? Think about how pendulum clock swings and in the past how technologies have emerged - you could then predict roughly what would constitute “the most balanced in the future”.

Use “centralised computing vs distributed computing” as an example, in the past there was usually only one centralised mainframe computer, later on to increase the processing capability commodity server farms were utilised, and now we are moving towards cloud computing.

There is no point to look at a particular past event. If you wanted to predict a technology in the future, knowing what has contributed to the balance of a past technology’s rise and fall is going to help.

Human’s ability is one of the factors too, because we have limited capability as a language designer it is useful to look at what others have done to cater for our ability, and therefore improve and evolve the technology.

In the book I briefly talked about Dart and Go. As a programming language inventor I find it really fascinating to explore the thought processes behind those language designers. And it has helped me to gain a deeper understanding of human behaviour.

Ito: I was going to ask why it is so important to study the past, now I know.

Matz: I mentioned this in the beginning - computing has not seen major changes for years.

Programming languages invented over fifty years ago are still in use today, and Ruby has been around for twenty years now. This proves that computing is progressing slower than what a lot of people believe.

On that note, there are many past cases whereby focuses were put on what was cool and new without understanding why. Compared to those “follower” software developers, the ones who command and understand the principles and theories behind changes and progresses have a much longer career longevity.

If you are a software developer who wants a longer career longevity, please read The Future of Computing! *chuckles*

Ito: Thank you Matz for talking to us today!

]]>

Datamappify - A New Take on Decoupling Domain, Form and Persistence in Rails

ifredwu@gmail.com (Fred Wu) — Thu, 27 Jun 2013 12:09:00 +0000

This post is about the ruby library we are building - Datamappify, please go check it out.

At Locomote we are building a relatively large web application using Rails. Before we began to lay the foundation, we knew very well that if we wanted the project to be maintainable we had to architect the system with extra care and attention. More specifically, we can’t rely on simply using ActiveRecord which combines behaviour and persistence as our domain models.

We began our search for something that would help us decouple our application from the domain layer down to the form handling. We’ve found a couple of gems that are close to what we were after - Curator, Minimapper, Edr and later on Reform. They are all wonderful gems but unfortunately none of them has everything we need.

Here are the things we need:

Decouple domain logic and data persistence.
Decouple models and forms.
Support ActiveRecord (or at the very least, ActiveModel) so we can still use many of the awesome gems like Devise, SimpleForm and CarrierWave.
Support attributes mapped from different data sources (e.g. remote web services from third-party vendors).
Support lazy loading so that attributes stored on remote data sources will not get triggered upon loading the entities.

All things considered, we bit the bullet and started working on Datamappify.

Before I go into details and examples, here is an extremely simplified overview of Datamappify’s architecture:

As you can see, Datamappify is loosely based on Repository pattern and Entity Aggregation pattern.

Datamappify has three main design goals:

To utilise the powerfulness of existing ORMs so that using Datamappify doesn’t interrupt too much of your current workflow. For example, Devise would still work if you use it with a UserAccount ActiveRecord model that is attached to a User entity managed by Datamappify.
To have a flexible entity model that works great with dealing with form data. For example, SimpleForm would still work with nested attributes from different ORM models if you map entity attributes smartly in your repositories managed by Datamappify.
To have a set of data providers to encapsulate the handling of how the data is persisted. This is especially useful for dealing with external data sources such as a web service. For example, by calling UserRepository.save(user), certain attributes of the user entity are now persisted on a remote web service. Better yet, dirty tracking and lazy loading are supported out of the box!

You can read more about Datamappify in the project’s README. For this blog post I will focus on how we use Datamappify in our Rails application. We are still early days in our application development and Datamappify still has quirks and issues, but I am hoping this post will illustrate some of the key benefits of Datamappify.

Getting Started

Getting started with using Datamappify is really easy. We simply include it in the Gemfile:

gem 'datamappify'

To keep things organised, we put entities and repositories in their respective directories under app:

Entities

The reason we wanted Datamappify to utilise existing ORMs like ActiveRecord, is so that we could still use gems like Devise.

So, we have a UserAccount model that handles authentication:

class UserAccount < ActiveRecord::Base
  devise :database_authenticatable,
          :recoverable, :rememberable, :trackable, :validatable
end

UserAccount has one and only one purpose - user account authentication. Other user behaviours would be contained in either the User entity itself or service objects. Speaking of the User entity, it looks like this:

class User
  include Datamappify::Entity

  attr_accessor :user_account

  attribute :account_username, String
  attribute :account_email, String
  attribute :first_name, String
  attribute :last_name, String
  attribute :activated, Boolean, :default => true

  attributes_from Contact, prefix_with: :work

  validates :first_name, presence: true
  validates :last_name, presence: true

  references :agency

  def full_name
    "#{first_name} #{last_name}"
  end
end

We include Datamappify::Entity in the User class to make it an entity. We set the :user_account accessor is so that we could attach the UserAccount object onto the entity.

The attribute DSL is from Virtus - we get attribute type coercion for free, awesome!

attributes_from is a DSL provided by Datamappify - it essentially “mounts” all the attributes from another entity, in this case the Contact entity, which looks like this:

class Contact
  include Datamappify::Entity

  attribute :email, String
  attribute :phone_number, String
  attribute :fax_number, String

  validates :phone_number, presence: true
  validates :fax_number, presence: true
end

All the attributes and validation rules from Contact are now “mounted” on User. attributes_from Contact, prefix_with: :work is equivalent to:

attribute :work_email, String
attribute :work_phone_number, String
attribute :work_fax_number, String

validates :work_phone_number, presence: true
validates :work_fax_number, presence: true

Validation DSL is provided by ActiveModel::Validations.

references :agency is a convenient DSL provided by Datamappify. It is equivalent to:

attr_accessor :agency

attribute :agency_id, Integer

Now that we have entities, we need a way to retrieve and store them. For that we need repositories.

Repositories

Here is the user repository:

class UserRepository
  include Datamappify::Repository

  for_entity User

  default_provider :ActiveRecord

  map_attribute :account_username, 'ActiveRecord::UserAccount#username'
  map_attribute :account_email, 'ActiveRecord::UserAccount#email'

  map_attribute :work_email, 'ActiveRecord::Contact#email'
  map_attribute :work_phone_number, 'ActiveRecord::Contact#phone_number'
  map_attribute :work_fax_number, 'ActiveRecord::Contact#fax_number'
end

Similarly to an entity, we include Datamappify::Repository to make the class a repository. We specify for_entity to link the repository to an entity, and default_provider to use a specific data provider for unmapped attributes.

Unmapped attributes are the ones not specified in map_attribute, in this case they are first_name, last_name and activated. Unmapped attributes are actually automatically mapped by Datamappify, so the user repository essentially does this:

map_attribute :first_name, 'ActiveRecord::User#first_name'
map_attribute :last_name, 'ActiveRecord::User#last_name'
map_attribute :activated, 'ActiveRecord::User#activated'

The first argument of map_attribute is the name of the attribute from the User entity (e.g. :first_name).

The second argument is a string containing three things:

ActiveRecord is the data provider.
::User is the ActiveRecord model class.
#first_name is the ActiveRecord attribute from the model class.

Even though the User entity is a representation of a user on the domain level, the underlying data structure does not necessary have to be. As you can see from the user repository example, we are mapping :account_username and :account_email to the UserAccount ActiveRecord model we’ve seen before. And we have a bunch of contact details attributes mapped to the Contact ActiveRecord model.

The database schema therefore looks like this:

create_table "contacts", force: true do |t|
  t.string "email"
  t.string "phone_number"
  t.string "fax_number"
  t.integer "user_id"
end

create_table "user_accounts", force: true do |t|
  t.string "username", default: "", null: false
  t.string "email", default: "", null: false
  t.string "encrypted_password", default: "", null: false
  t.string "reset_password_token"
  t.datetime "reset_password_sent_at"
  t.datetime "remember_created_at"
  t.integer "sign_in_count", default: 0
  t.datetime "current_sign_in_at"
  t.datetime "last_sign_in_at"
  t.string "current_sign_in_ip"
  t.string "last_sign_in_ip"
  t.integer "user_id"
  t.datetime "created_at"
  t.datetime "updated_at"
end

create_table "users", force: true do |t|
  t.string "first_name"
  t.string "last_name"
  t.boolean "activated"
  t.integer "agency_id"
end

Note that because the Contact entity is mounted on the User entity, we need a foreign key user_id in the contacts table to link them.

Data providers

Because we are allowed to specify a data provider (i.e. ActiveRecord) for each attribute, we can map attributes to entirely different data providers! For instance, we could have:

map_attribute :first_name, 'ActiveRecord::User#first_name'
map_attribute :last_name, 'ActiveRecord::User#last_name'
map_attribute :activated, 'Sequel::UserActivation#activated'

Now the activated attribute is mapped to the UserActivation Sequel model. This powerful feature would allow us to develop data providers that communicate with remote web services. :)

Repository APIs

Datamappify provides a bunch of APIs for retrieving and storing data. These APIs are being developed as needed during our application development.

For instance, to find a particular user entity by ID:

user = UserRepository.find(1)

To get all the users:

users = UserRepository.all

To search for users:

users = UserRepository.find(:first_name => 'Fred', :activated => true)

To save a user:

UserRepository.save(user)

To delete a user:

UserRepository.destroy(user)

There are some limitations for certain APIs, you can read more about them here.

When saving an entity, because Datamappify has an internal pool for tracking dirty attributes, only those dirty attributes will get saved.

Also, Datamappify supports attribute lazy loading, all we have to do is to tell our entity to become lazy aware:

class User
  include Datamappify::Entity
  include Datamappify::Lazy
end

Once an entity is lazy aware, a repository will inject the lazy loading mechanism onto the entity when it retrieves such entity (via find).

Both repositories and entities support inheritance, again, you may read more about them in the project’s README.

Putting Things Together With A Controller

Remember we have the UserAccount for authentication? So how does that work? Well, here’s the piece of code in our application controller:

class ApplicationController < ActionController::Base
  before_filter :authenticate_user_account!

  private

  def current_user
    unless current_user_account.blank?
      @current_user ||= UserRepository.find(current_user_account.user_id)
    end
  end

  helper_method :current_user
end

Not too different from the normal Devise workflow hey? ;)

Forms

Web applications typically have lots of forms - ours is no different. It turns out, Datamappify can help build form objects too!

Here’s the view portion of one of our forms (we use Slim templates):

# our standard form layout

= simple_form_for submission_target, url: submission_path, signed: true do |f|

  = yield(f)

  .form-actions
    .pull-left
      - if archivable? && resource.persisted?
        = link_to 'Archive', '#', method: :delete, class: 'btn-archive'

    = f.submit 'Save'
    = link_to 'Cancel', cancel_path

# actual form content

= render layout: 'forms/resource' do |f|
  h3 Account Details
  .row-fluid
    .span4 = f.input :account_type, collection: BankAccount::ACCOUNT_TYPES.invert
    .span4 = f.input :account_code
    .span4 = f.input :account_name
  .row-fluid
    .span4 = f.input :account_number
    .span4 = f.input :bank_name
    .span4 = f.input :bank_branch
  .row-fluid
    .span4 = f.input :bank_bsb, label: "Bank BSB"
    .span4 = f.input :activated, as: :boolean

  h3 Branch Details
  .row-fluid
    .span4 = f.input :branch_address_line_1, label: "Address 1"
    .span4 = f.input :branch_address_line_2, label: "Address 2"
    .span4 = f.input :branch_address_line_3, label: "Address 3"
  .row-fluid
    .span4 = f.input :branch_state, label: "State"
    .span4 = f.input :branch_postcode, label: "Postcode"
    .span4
      = f.input :branch_country_code, label: "Country" do
        = f.country_select :branch_country_code, ['au', 'us', 'gb'], value: 'au'
      - f.add_signed_fields :branch_country_code
  .row-fluid
    .span4 = f.input :branch_phone_number, label: "Phone number"
    .span4 = f.input :branch_fax_number, label: "Fax number"

  h3 Merchant Details
  .row-fluid
    .span4 = f.input :merchant_number
    .span4 = f.input :merchant_name
    .span4 = f.input :merchant_address
  .row-fluid
    .span4 = f.input :merchant_biller_code, label: "Biller code"
    .span4 = f.input :merchant_bpay_method, collection: Merchant::BPAY_METHODS.invert, label: "BPAY method"

  h3 Payment Details
  .row-fluid
    .span4 = f.input :payment_number
    .span4 = f.input :payment_merchant, label: "Merchant description"
    .span4 = f.input :payment_reference_type, collection: PaymentAccount::REFERENCE_TYPES.invert, label: 'Reference type'

As you can see, it’s a form containing details from bank account, branch address, branch contact information, merchant and payment. And indeed we do have bank_accounts, addresses, contacts, merchants and payment_accounts tables in our database. Yet, the form still remains flat-structured and submits via simple_form_for.

This is thanks to our BankAccount entity and BankAccountRepository repository:

class BankAccount
  include Datamappify::Entity
  include Concerns::Archivable

  ACCOUNT_TYPES = {
    'general' => 'General',
    'trust' => 'Trust'
  }

  attribute :account_type, String
  attribute :account_code, String
  attribute :account_name, String
  attribute :account_number, String
  attribute :bank_name, String
  attribute :bank_branch, String
  attribute :bank_bsb, String
  attribute :activated, Boolean, default: true

  attributes_from Address, prefix_with: :branch
  attributes_from Contact, prefix_with: :branch
  attributes_from Merchant, prefix_with: :merchant
  attributes_from PaymentAccount, prefix_with: :payment

  references :agency

  validates :account_type, Mos::Validation.in(ACCOUNT_TYPES)
  validates :account_code, Mos::Validation::STANDARD_TEXT
  validates :account_name, Mos::Validation::STANDARD_TEXT
  validates :account_number, Mos::Validation::STANDARD_TEXT
  validates :bank_name, Mos::Validation::STANDARD_TEXT
  validates :bank_branch, Mos::Validation::STANDARD_TEXT_OPTIONAL
  validates :bank_bsb, Mos::Validation::BSB
  validates :merchant_bpay_method, Mos::Validation.presence_depend_on(:merchant_biller_code)
  validates :payment_reference_type, Mos::Validation.presence_depend_on(:payment_number)
end

class BankAccountRepository
  include Datamappify::Repository

  for_entity BankAccount

  default_provider :ActiveRecord

  map_attribute :branch_address_line_1, 'ActiveRecord::Address#address_line_1'
  map_attribute :branch_address_line_2, 'ActiveRecord::Address#address_line_2'
  map_attribute :branch_address_line_3, 'ActiveRecord::Address#address_line_3'
  map_attribute :branch_state, 'ActiveRecord::Address#state'
  map_attribute :branch_postcode, 'ActiveRecord::Address#postcode'
  map_attribute :branch_country_code, 'ActiveRecord::Address#country_code'

  map_attribute :branch_phone_number, 'ActiveRecord::Contact#phone_number'
  map_attribute :branch_fax_number, 'ActiveRecord::Contact#fax_number'
  map_attribute :branch_email, 'ActiveRecord::Contact#email'
  map_attribute :branch_website, 'ActiveRecord::Contact#website'

  map_attribute :merchant_number, 'ActiveRecord::Merchant#number'
  map_attribute :merchant_name, 'ActiveRecord::Merchant#name'
  map_attribute :merchant_address, 'ActiveRecord::Merchant#address'
  map_attribute :merchant_biller_code, 'ActiveRecord::Merchant#biller_code'
  map_attribute :merchant_bpay_method, 'ActiveRecord::Merchant#bpay_method'

  map_attribute :payment_number, 'ActiveRecord::PaymentAccount#number'
  map_attribute :payment_merchant, 'ActiveRecord::PaymentAccount#merchant'
  map_attribute :payment_reference_type, 'ActiveRecord::PaymentAccount#reference_type'
end

Pretty clean and straightforward. What do you think?

Datamappify - Keeping Your Domain Layer Sane

Datamappify’s concept isn’t new, but there simply isn’t anything in the ruby community that solves everything Datamappify tries to solve.

I hope that this post has not only introduced you to Datamappify, but has also made you think about how to make your application’s domain layer more decoupled.

Thanks for reading, and if you have any feedback for Datamappify please get in touch with me!

]]>

ActiveRecord and DB Migration Ate My Model Attributes!

ifredwu@gmail.com (Fred Wu) — Wed, 27 Feb 2013 00:12:00 +0000

Update: You might also want to check out reset_column_information.

So a few days ago we started seeing the following errors on our Jenkins builds (swapped with fictional model and attribute names):

NoMethodError:
  undefined method `attack_power=' for #

attack_power is a new attribute we recently added to the Ironman ActiveRecord model.

I was baffled, as the table column is clearly there but ActiveRecord couldn’t see it.

This weird behaviour is confirmed by debugging the model:

Ironman
# => Ironman(id: integer, created_at: datetime, updated_at: datetime)

Ironman.column_names
# => ["id", "created_at", "updated_at"]

And by debugging the schema:

ActiveRecord::Base.connection.structure_dump
# => "CREATE TABLE `ironmans` (\n `id` int(11) NOT NULL AUTO_INCREMENT,\n `created_at` datetime NOT NULL,\n `updated_at` datetime NOT NULL,\n `attack_power` int(11) NOT NULL,\n PRIMARY KEY (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;\n\n"

So apparently somehow ActiveRecord ate our attack_power attribute!

After some struggling and head-scratching, finally I’ve discovered that during the migration if we call the Ironman class, then any subsequent attribute changes to the class will not be recognised by ActiveRecord.

It turns out, because we run our test suites by invoking rake tasks:

Rake::Task['spec:something'].invoke
Rake::Task['spec:something_else'].invoke

And we manually reset our database beforehand too within the same Rakefile:

Rake::Task['db:reset'].invoke

ActiveRecord will cache its attributes as soon as a model class (Ironman) is called during the migration.

The fix? Simple! Simply use another process to run the database reset:

system 'rake db:reset'

]]>

"Become A Better Developer You Can" - Video of My RubyConf China 2012 Talk

ifredwu@gmail.com (Fred Wu) — Mon, 03 Dec 2012 11:44:00 +0000

It was the first time I presented in front of 400+ people so I was really nervous - I blasted through the talk in under 30 minutes even though I was supposed to talk for 45 minutes, oh well. ;)

video: https://www.youtube.com/embed/DeBsmdDmB9A

Also, my slides are available too.

P.S. If you’re located in China, you may view the talk at Railscasts China.

]]>

An Interview with Yukihiro "Matz" Matsumoto

ifredwu@gmail.com (Fred Wu) — Sun, 25 Nov 2012 06:45:00 +0000

Update: Please check out this new interview with Matz, done by Engineer Type.

A week ago I went to Shanghai, China to attend and to give a talk at RubyConf China. The day before the conference’s first day a bunch of us were invited to a VIP dinner where we met with Matz and got to play with a device running MRuby. And I heard that earlier on that day Matz was ‘adopted’ by a book publisher to do an interview.

I have found the interview (in Chinese), and found it to be really useful. So I translated it to English. Hope more people will like it. :)

On November 16th, 2012 - a day before RubyConf China, a Chinese book publisher Turing Book has done an interview with Matz on his new book The Future of Computing as well as about a few topics interested to Chinese readers. The interview was conducted by The Future of Computing’s Chinese translator Zi Heng Zhou.

Zhou: Mr. Matsumoto’s new book The Future of Computing is published earlier this year and the Chinese version is being translated by yours truly, and that is going to be released some time next year in China. Your last book The World of Code has received high praises amongst the readers in China, so what are the differences between the last and the new book?

Matz: The World of Code has 14 topics in total, and each topic covers the basics - they covered more breadth than depth. The new book on the other hand has one set topic - thoughts on the emerging technologies in the future, therefore the ground covered would be narrower and more in depth than the last book. On top of that, the new book discusses several things by timescale, such as the history and the changes since the invention of computing, and computing’s impact on our future lives. Therefore it’s the thoughts of both the past and the future. The computing world is changing rapidly, and this book’s purpose is to discuss the direction of computing heading into the future.

Zhou: Speaking of the history of computing, you have touched on a few things about Moore’s law in the book?

Matz: Moore’s law describes the rule of changes over the history of computing hardware. The book discusses not only the changes on computing itself, but also the changes on its surrounding environment.

Zhou: On the topic of the evolution of programming languages, Paul Graham said in The Hundred-Year Language that the main branches of the evolutionary tree pass through the languages that have the smallest, cleanest cores. In the new book you seem to hold a different opinion, can you tell us why? And what’s your take on the evolution of programming languages?

Matz: Paul loves Lisp, and Lisp perfectly matches the characteristics of the programming languages described in his essay, and so Paul reckons the programming languages in a hundred years from now will look like Lisp. In reality though, Lisp has been around for over 50 years, and to be honest it isn’t one of the mainstream programming languages. In my opinion this may have been because most programmers don’t find Lisp charming enough. In other words, there is a gap between the so-called “smallest, cleanest cores”, “beautiful” languages and the expectation of programmers. It would be understandable if Lisp’s charm had not been accepted by everyone in a year or two, but for 50+ years it hasn’t reached the mainstream, could it be because it fundamentally does not match our expectation? There is a huge difference between human friendly languages and languages that have smallest, cleanest cores , and I am afraid the gap between them might not close even in a hundred years. As for what future programming languages should look like, I think they should have a runtime model similar to that of Lisp and be easily understandable by humans. All of a sudden, Ruby looks a lot closer to that, doesn’t it?

Zhou: Mr. Matsumoto you are the father of Ruby. We know that during the design of a programming language there might be a lot of different choices to be made, e.g. dynamically typed vs statically typed, prototype based vs class based, etc. When you were designing Ruby, what was the most difficult choice you made?

Matz: Before Ruby I actually have designed another language when I was in university, and that language was statically typed, similar to Eiffel. I liked statically typed languages, but the language I designed during university was more for academic purposes. Several years later, when I wanted to design a language as a tool I could use, I preferred to have it dynamically typed as I think they were more practical. So I designed Ruby. And I think it was the correct decision - it might not have been the most difficult decision, but it certainly was the biggest decision. Now that the decision of being a dynamic language is made, languages such as Smalltalk, Lisp and to an extent Perl have all had influences on Ruby. One of the features of Ruby is “mixins”, and mixins were not very common at the time Ruby was created. But because I don’t like multiple inheritance, I always believed that there must be an easier way to achieve similar results, so I designed mixins in Ruby.

Zhou: Looking back, is there anything in Ruby you wish you did differently?

Matz: In the beginning my goal was to replace Perl as my tool therefore I have borrowed many ideas from Perl, such as using the dollar sign ($) to indicate special variables. Looking at it now it seems to be a little bit too much and too similar to Perl. There are a few other things but mainly I think it is too similar to Perl. Back then before the ruby idioms were formed, there were many things that were borrowed from Perl - nowadays I think many of them weren’t necessary thanks to the ruby and rails idioms.

Zhou: Many believe Ruby’s popularity was because of Ruby on Rails, and you agreed with this in the book. So what do you think made Rails so successful?

Matz: The first and foremost, it benefitted from the rapid growth of the web - almost every software development platform is eyeing on the web field. What was developed using the traditional Client-Server architecture can now be implemented on the web. The number of applications that can be developed for the web has grown, and that’s important. Secondly, Ruby is optimised for easier development and higher development productivity. I think these two reasons combined, is what made Rails so successful.

Also, Ruby has many powerful features such as meta-programming and extendability via monkey patching. Through these features, basic classes can be enhanced, and DHH created Rails using these powerful features of Ruby. For people who have never touched Ruby, for example those Java programmers who like “inflexible” languages, they will be like, “Huh? You can do that?” - and I think that’s also one of the reasons why Rails is so popular.

Zhou: It is said that DHH was going to create a web framework using PHP, but eventually moved to Ruby?

Matz: Yes, and I think the reason for that might be the limitation of meta-programming in PHP. After Rails was released there were a bunch of PHP frameworks came out that are inspired by Rails, such as Symfony and CakePHP. PHP however does not have many of the powerful features found in Ruby, and purely from the development perspective I still believe Rails is the more powerful one out of the bunch. By the way, I actually met with DHH before, in Denmark. Back then he hadn’t started learning Ruby yet, perhaps that had a small impact on him too.

Zhou: Ruby is being used widely for the web, but one hot topic amongst Chinese readers is the development direction of Ruby outside of the web. Any thoughts?

Matz: True, many web based projects use Ruby, for instance web frameworks like Rails and Sinatra, etc. The world of programming however is far beyond simply web, and I have always wanted Ruby to break out from the web to other areas. In the near future, I would like to see Ruby being used in three fields:

1.

Scientific computing. In this field, languages like Python, R and Matlab are popular choices. I hope Ruby will soon be one of them. In fact, we are developing a project called SciRuby, and we hope it will help us with the adoption in scientific computing.

2.

High performance computing. This is somewhat related to scientific computing - to use super computers for doing computational work. Compared to C++, Ruby indeed is very slow, and that’s why people think Ruby isn’t suitable for doing high performance computing. In University of Tokyo a research student is working on an academic research project that compiles Ruby code to C code before compiling the binary code. The process involves techniques such as type inference, and in optimal scenarios the speed could reach up to 90% of typical hand-written C code. So far there is only a paper published, no open source code yet, but I’m hoping next year everything will be revealed.

3.

Embedded systems. For micro devices such as cell phones, medical equipment and robots, Ruby isn’t really suitable due to its memory use and APIs. Therefore we need something that’s more suitable for these embedded systems where memory consumption has to be low and APIs have to be optimised. And that’s what MRuby is designed for.

Zhou: Twitter is mainly built using Rails, but I read a news recently about the traffic surge during the US president election period and Twitter is now migrating to other platforms to help with the scaling. What’s your view on this?

Matz: There are many reasons I believe. First of all, nobody could have predicted Twitter’s huge success - who would’ve thought a service that essentially provides 140 characters of text for blogging to become one of the largest social networks. During their rapid growth they have added a lot of new features, and I think Ruby has contributed to that - it allowed many new features to be thought and implemented in very short amount of time. Twitter could not have anticipated the traffic it is now handling, so it’s fair to say that they have now hit the limit of their architecture designed from way back.

To solve the issue, Twitter needs to design a new architecture from the ground up in order to handle the ever increasing traffic. Although, Ruby can still be used for rewriting the architecture (chuckles). More seriously though, when a platform hits its limit, there are a few ways of solving it such as rewriting the architecture and changing to another language/platform, etc. I believe the most effective way is to rewrite the architecture, and this is exactly what Twitter’s been doing. During the rewriting process the Twitter engineers wanted to take on more challenges so they picked Scala. Because Scala is a compiled language it has great performance, so it is a fine choice for the new architecture.

My opinion is that when your system is still in its growing stage, it is far more important to have the ability to react quickly to changes, and that’s what a highly flexible language such as Ruby offers. Once your system reaches to a point of maturity, stability and success, then to have a new architecture that saves on resources makes sense. Twitter only chose to use Scala for its core components, the web front-end and many of their internal tools are still using Ruby. As a matter of fact, I paid Twitter a visit last month and talked to many of their engineers there - Ruby is still in great use (chuckles).

Zhou: Mobile platforms become hotter and hotter in recent years due the increased use of smartphones, tablets and other mobile devices. As far as programming language is concerned, Android uses Java and iOS uses Objective-C. What about scripting languages like Ruby, how do you see them fit in this picture?

Matz: It is no doubt that so far in order to develop for Android you use Java, and for iOS you use Objective-C. But that also creates a huge barrier because you would need to rewrite your application if you wanted to port from one platform to another. Projects like PhoneGap and Titanium try to solve this very issue, via languages such as JavaScript and Lua to offer cross-platform compatibility. For Ruby, there is a project called Rhodes - you can use Ruby to write applications that will run on iOS, Android and Blackberry.

In the old days, mobile devices are much less powerful than PCs, and if applications can’t run in full speed then they are useless. Nowadays though, mobile devices have gotten significantly faster, and the use of cross-platform frameworks becomes far more practical than ever before. I think this way of developing for mobile platforms will become more and more popular. And by the way, we spoke about MRuby before, and using MRuby for iOS and Android is already underway, hopefully more applications will get developed using MRuby in the near future.

Zhou: You mentioned MRuby, and the topic of your keynote presentation tomorrow is Ruby 2.0. Could you talk about MRuby and Ruby 2.0’s highlights a little bit?

Matz: As we discussed earlier, MRuby is designed to run on embedded devices, it does not have everything from Ruby. As a result, many devices which traditionally can’t use Ruby, such as vendor machines, controllers and robots can soon utilise MRuby. Perhaps in a few years time, we could see Ruby used in televisions and cars.

Ruby 2.0 indicates two messages. Firstly, by next year it will be 20 years since the development of Ruby started. What better version number can there be to celebrate this special moment! Contrary to what it might seem like, in terms of the changes, from 1.9 to 2.0 is not as big as from 1.8 to 1.9. In fact, our goal for 2.0 is to be fully backward compatible with 1.9. Your 1.9 applications should run just fine on 2.0.

Secondly, Ruby 2.0 does offer a few new features. For instance, use monkey patching to add new or replace features has global effects and it might cause conflicts and bugs. In order to avoid situations like this, Ruby 2.0 limits the scope of monkey patching via Refinements. As bigger teams and bigger projects are adopting Ruby, scoped monkey patching will increasingly show its importance. There are many other features in Ruby 2.0 that are designed to cater for bigger teams and projects.

Zhou: As far as I know most of the mainstream programming languages are from America and Europe - though there are Lua from Brazil and Ruby from Japan. You too mentioned this in your book and you said this feels “lonely”. So what is the cause of this, and what can we do about it?

Matz: Well, for Lua you can include it in Europe/America too because Brazil is part of South America (chuckles). In the south eastern Asia region though there is only Ruby, and it is lonely. Europe and America still remain the most powerful regions as far as programming languages go. Asia, although has massive population, does not compete in this regard, that indeed feels lonely.

I am not sure about other countries, but at least in Japan there are many people working on programming languages, unfortunately other than Ruby none of them are well known. If more people are interested in programming and designing programming languages, there bound to be one or two that’ll break out, right? There is another hurdle in Japan - language. Most Japanese people only speak Japanese and they cannot speak English well. Funnily enough there are programming languages written entirely in Japanese. ( Zhou: In China there also are programming languages written entirely in Chinese.) In China too? I knew it! No matter how interesting these programming languages are, they will never influence anyone beyond the ones in their own country.

On a side note, I once received an email from an American. He said that you are Japanese, but Ruby looks like English because it’s written in English, why isn’t there any Japanese-written languages? I replied saying that there are, you just don’t know them, and even if you did, you wouldn’t be able to use them.

In Japan, more and more people are interested in programming, maybe because both online and in my books I always talk about how fun programming can be. Many people are now taking on the challenge of designing new programming languages. Out of these new languages, even only 0.1% of them ever get any success, I think it’s a win. I don’t know how many people want to take on the same challenge in China, Korea and other countries in Asia, but if people could look beyond programming languages are created for us, we just passively accept them, and think to create a new programming language can also be fun, then I am sure some of them will succeed.

Talking about open source projects, not many of them are from Japan, China and Korea, and I think this could be an entry point for many. There are many reasons why this is the case though, for example English is hard to learn… ( Zhou: And GitHub is also difficult to use?) Haha, is GitHub usable in China? ( Zhou: It is, it is…) Oh, that’s not too bad then. But, China’s Great Firewall still has a huge impact, many resources can’t be accessed here, right? ( Zhou: That’s right, for instance the Go programming language’s website is blocked.) Ah really? Is it because it’s made by Google? (Chuckles.) In any case, I think there are still many difficulties to face. Also, in Japan many programmers still spend most of the time at work (to put food on the table), it’s very difficult for them to contribute to open source projects. Ten years ago nobody cares about open source in Japan, but nowadays people start to realise the importance of open source, and the number of open source projects is growing. I believe China will soon follow this pattern as well, I am looking forward to it.

In the beginning no one knows what will succeed. When I started with Ruby I could not possibly have predicted its success. So I think for a programming language, timing is really important - and you’ll never know until you tried. I think in China there might also be languages that emerge from the right time that will eventually be a global success.

Zhou: In the book when you were talking about Dart, you mentioned that the ecological environment is really important for a programming language. So what do we need to do to ensure a good ecological environment?

Matz: From a user’s perspective, the most important thing is what benefits a programming language can offer. Before Rails came along, many Ruby users including myself, believe Ruby is a user-friendly language, and that’s the reason we like to use it. After Rails was born, more and more people like to use Ruby because making websites using Rails is really productive. So I think if you could communicate to your users what the benefits are by using your new programming language, it will help with its adoption and increase the chance of success.

Zhou: Thank you very much Matz! Do you have anything else to say to Chinese programmers?

Matz: In the book The World of Code I have mentioned that programming will likely to develop and evolve in the form of open source. New generations of programming languages and software are likely to emerge as open source projects. People are happy to freely use the software developed by others, and to take from that, and work on your own open source projects and make a little impact on the world - then you would become a world class software engineer. The software engineers I met in China are all very hard working and enjoying learning. I hope more of these people will step up and take on the challenge, and become world class programmers who will make a decent impact.

]]>