Fred Wu (@fredwu)

Quick comparisons of Viture Pro and Xreal Air 2 Pro

2024-06-09T10:08:39.269242Z

I’ve had the Xreal Air 2 Pro for a while. I ordered the Viture Pro on the 23rd May, it shipped on the 28th May, and finally got delivered on the 9th June. I’m in Australia.

Packaging

The unboxing experience has been stellar, the packaging is one level above Xreal’s.

Case

I do like Xreal’s case not needing a zipper, but the utility of Viture’s case is better, with lining for both the glasses and the cable, whereas Xreal’s is simply a plain case without any inserts.

Build Quality

I would say on par, both are very well built.

Styling

This is highly subjective. I’ve seen many many reviews on both. For someone with a small head, I thought the Viture would look better on me… hmmm… not quite, I think I still prefer the Xreal. Having said that it’s not a big deal for me, I’d be mostly wearing them at home, and perhaps on flights.

Wearing Comfort

This one goes to the Xreal, by a large margin. Xreal’s leg ends are soft, bendable and very thin, making putting on and off the glasses a breeze. The Viture on the other hand, has hard leg ends, which requires more force to bend the whole legs outwards when putting on and off.

When the glasses are on my face, I can feel the plastic legs clamping on me with the Viture, whereas with the Xreal I almost don’t feel anything.

Screen Brightness

Xreal Air 2 Pro technically has lower nits (500 vs 1000 perceived), but in reality I couldn’t tell much of a difference. Viture Pro might appear ever so slightly brighter. Although, interestingly, the Viture Pro’s lowest brightness is lower than Xreal’s (although with significant contrast drop).

Also, Viture Pro has OSD to display brightness and audio levels when you adjust them, Xreal doesn’t.

Sound Quality

Disappointingly, to my ear, the Xreal Air 2 Pro sounds better, it is slightly louder, and has significantly better clarity.

Image Quality

This might be the most disappointing part, and I’m really hoping a new firmware will address this. As is, on the latest 0603 firmware, the Viture Pro has a significant green/yellow tint - like, not a little bit, but very very significant. This makes movie watching (my primary use case) a no-go.

Electrochromic Screens

Very similar between the two. Xreal has three levels, Viture Pro has two, but realistically only the 0% and 100% are useful anyway. One small thing I noticed is that when you push the button, Xreal changes the level immediately, Viture Pro has half a second delay, no biggy.

Screen Edge Clarity

I used to own a pair of Xrea Air (the 1st gen), and from memory, I think that one has the best screen edge clarity.

The Xreal Air 2 Pro notoriously has blurry edges, which I can confirm. After swapping to using the smallest nose pads, I was able to make the edges clear enough, but it still has a slight colour shift.

The Viture Pro is better, however the edges I would say still have a little bit of colour shift. Although, can’t say it effects my use case at all. I’ve tried to look at text, and they are fine.

Myopia Adjustments

This is one of the main reasons I bought the Viture Pro despite already owning the Xreal. As I’m a contact lens wearer, I’d like to use the glasses in bed without using contacts.

Obviously the Xreal doesn’t have this functionality. However, this is another major disappointment that I’m hoping a firmware could fix. When the myopia dials are set to 0, the screen is actually blurry!! I thought it was my eyes at first, but nope, I had to dial them both up slightly in order for the screen clarity to equal that of Xreal’s.

Verdict

As you can probably tell, I am very disappointed. As is, the Viture Pro is unusable with the significant green tint. And the usability is impacted by the miscalibrated myopia adjustments.

I’m hoping both of these issues will be fixed soon via firmware updates.

P.S. Tested both glasses on both macOS (Macbook Pro) and Samsung Dex (S24 Ultra).

Update a day later

So last night I used the Viture Pro in bed to test two things: comfort and myopia dials.

Myopia Adjustments

The myopia dials work. But as I suspected, it’s miscalibrated. As mentioned before, when I’m wearing contacts (new prescription, also tested with Xreal), I have to slightly dial them up to get a clear picture. Now when I’m in naked eyes without contacts, both of my eyes are -4.75, but I have to dial them all the way to the end (-5) to get the most clear picture. This indicates to me that the dials are miscalibrated.

In Bed Comfort

Comfort unfortunately is a step down compared to the Xreal. Mostly due to the rigid legs that stick out longer compared to the Xreal - they would push into the pillow. Of course, if you don’t plan on wearing them in bed this won’t effect you. Or if you have a larger head, it probably will be okay too.

Audio Control

Another thing I noticed is that Viture Pro’s volume control is separate to the system. When connected to the Mac, it is connected as a DisplayPort device type, whereas Xreal is connected as a USB device type. So when changing volume, Xreal directly changes the system volume whereas with Viture Pro, you can either change the volume on both the system and on the glasses or only on the glasses depending on the host device (on a Mac, you can’t change the system volume, on an Android you can). Personally I prefer the Xreal’s approach.

Colour Tint

Finally, I’ve taken some comparison shots of the colours between my Macbook Pro, Xreal Air 2 Pro and Viture Pro (this is also sent to Viture support).

Upon first glance they look very similar. But please zoom in and take a closer look. I’ve used the same camera and manually set the white balance to ensure they are consistent across the devices.

The colour shift is way more pronounced and is very noticeable in real life.

I think I’ve got a better comparison. The backgrounds are all close to black in real life, and pay no attention to the blurriness it’s just my camera’s focus. Again, set with the same white balance, you can now clearly see the yellowish tint on the Viture Pro. In real life the Macbook Pro and Xreal screens are very very similar.

]]>

Three Versions of The Three Body Problems, Thoughts from A Native Mandarin Speaker

2024-05-26T07:51:38.204617Z

Warning: Minor spoilers on the characters, but no spoilers on the story.

As a casual Sci-Fi enjoyer who mostly enjoys them through movies and TV shows, I’ve only occasionally read Sci-Fi novels. But I loved Andy Weir’s The Martian (loved the movie first, then went back to reading the novel) and Project Hail Mary. Therefore, since discovering Netflix’s 3 Body Problem not too long ago, I realised I had to start from the origin, and follow the thread from there, especially since I am a native Mandarin speaker.

The original novel, 三体, would translate to simply Three Body. However, the English translation of the book was published as The Three-Body Problem, and the Netflix adaptation 3 Body Problem.

Within the span of two months, I consumed, in order:

The original Chinese trilogy, which includes Three Body, Three Body II: The Dark Forest and Three Body III: Death’s End, in audio book format, totalling a whopping 85+ hours.
The Chinese TV series adaptation produced by Tencent Video, which consists of 30 episodes covering the first book, totalling 20 hours. You can watch the full series here on Youtube by one of the official publishers.
The Netflix TV series adaptation, which consists of 8 episodes covering the first book and parts of the second book, totalling 8 hours.

For the Chinese TV series, there is also a 26-episode Director’s Cut version, which apparently has better pacing by removing some original sub-plots and tightening the overall story flow. I actually didn’t mind the slower pace and the original sub-plots so I stuck with the original 30-episode version.

I’m going to rate all three versions, but as a benchmark, I’d give The Martian (both the novel and the movie) a 10/10, and Project Hail Mary also a 10/10. This is from the perspective of someone who grew up in China then spent more than 20 (and counting) years living in and breathing the Western culture.

TL;DR: Ratings Summary

| Version | Overall | Story | Main Characters or Cast | Minor Characters or Cast | Cinematography | Music / Soundtrack | VFX | Audio and SFX | — | — | — | — | — | — | — | — | — | Novel | 9 | 9 | 10 | 9 | N/A | N/A | N/A | N/A | Tencent | 9 | 9 | 10 | 6 | 10 | 9 | 7 | 5 | Netflix | 7 | 7 | 8 | 7 | 7 | 7 | 6 | 7

Story

| Version | Rating | Commentary | — | — | — | Novel | 9 | A very captivating story and with a world building that’s mostly grounded to reality. | Tencent | 9 | A condensed version that is faithful to the original novel (1st book), with some added minor characters and sub-plots. The story-telling focuses more on character building to push the story forward, just like in the novel. | Netflix | 7 | A “re-engineered” story with a much smaller, mishmashed ensemble, much faster pace and less convincing world building. It is the fast food version - not necessarily inferior, but much less nuanced and is produced for mass consumption by the Western audience. Remember, it speeds through the entire book 1 and part of book 2 in merely 8 hours, compared to the 20 hours of the Tencent adaptation for only book 1.

Main Characters or Cast

| Version | Rating | Commentary | — | — | — | Novel | 10 | The believable and multi-faceted characters is definitely a highlight of the series, especially from the lead characters of each book. | Tencent | 10 | Superb acting from the main cast. Shi Qiang is a highlight, the portray of this version of Shi Qiang adds more humour and lightens much of the heaviness of the story beats. Wang Miao has a calm mannerism and deep and soothing voice. Both the young and old Ye Wenjie are portrayed with consistency, and nuances that beyond words. | Netflix | 8 | The acting is decent for what the show is. Due to the fast pace nature, there is only so much that can be done by the otherwise fine cast. The characters themselves however, take a major hit. Many characters from the novel are combined. Worryingly, Ye Wenjie has been portrayed as simply bitter, missing the nuances entirely as a result of the many hours of character building in the novel and the Tencent adaptation.

Minor Characters or Cast

| Version | Rating | Commentary | — | — | — | Novel | 9 | Like the main characters, each minor character comes with a believable background and motivation, and helps with the story and world building. | Tencent | 6 | Unfortunately, this is a weak point of the Chinse adaptation. Whilst the characters themselves are fine, the cast unfortunately is a big hit and miss, with a few of them definitely miscast. Wei Cheng in particular, has been reduced to a one-dimensional maths nerd with a heavy Shandong accent unnecessarily. Pan Han’s portray is also wooden and uninspired. To top it off, a super minor character with only a minute or two of screen time has an extremely bizarre, over-the-top acting style that nearly ruined the entire scene. Luckily, most of the other cast ranges from okay (e.g. Shen Yufei) to superb (e.g. Chang Weisi). | Netflix | 7 | Acting from the minor cast is fine, nothing remarkable but nothing terrible either.

Cinematography

| Version | Rating | Commentary | — | — | — | Tencent | 10 | It’s a masterpiece of an art that left a huge mark on me. The cinematography is simply breathtaking. Even more, each episode includes a bunch of photo montages during credits. | Netflix | 7 | It’s mostly a standard sci-fi look and feel, it looks good but nothing too inspiring here.

Music / Soundtrack

| Version | Rating | Commentary | — | — | — | Tencent | 9 | A collection of finely composed soundtracks covering a wide range of emotions. I was moved by many of them. Check them out here. | Netflix | 7 | As a fan of Ramin Djawadi and his work from Game of Thrones and Westworld, this is a fine collection of soundtracks, albeit a tad forgettable compared to the aforementioned masterpieces.

VFX (Visual Effects)

| Version | Rating | Commentary | — | — | — | Tencent | 7 | Some VFX are a bit janky. Though in most cases the producers have hidden them well. The “in-game” CG and animation are very janky but that can simply be attributed to them being “in-game” therefore to be expected, besides, they are actually quite charming. | Netflix | 6 | This surprised me. I thought for sure Netflix would have the budget for higher quality VFX. In most cases this is true, unfortunately this is let down by some key moments having very questionable VFX and/or artistic choice.

Audio and SFX (Sound Effects)

| Version | Rating | Commentary | — | — | — | Tencent | 5 | Another unfortunate weakness of the Tencent adaptation. Many sound bites (such as the eery music) were simply lifted from other TV shows like Dark which takes me out of the immersion. What was also baffling, is the audio mixing - in more than one occasion, lines from different takes are stitched together with vastly different audio levels and uniformity. | Netflix | 7 | Like the cinematography, there is nothing wrong or special here. It’s good.

Extra: Languages

As someone who speaks both Mandarin and English, it’s been an interesting experience.

The Netflix version is easier to consume in this regard. Other than the character Mike Evan’s borderline gibberish Mandarin, for the most part I only needed subtitles for the few lines of French and Spanish that didn’t really impact the story. Although, Ye Zhetai speaking with a heavy southern Chinese accent is a bizarre casting choice.

The Tencent version on the other hand is much more difficult to consume. A few episodes in I had to rely on subtitles to make out both the Mandarin and the English spoken by the non-native speakers.

Final Thoughts

I liked the original novel trilogy and loved the cinematography of the Chinese TV adaptation. I probably would’ve liked the Netflix adaptation more if I had no point of reference from the other two.

If you’ve watched the Netflix version and are curious about the source material - read the novel(s) or watch the Chinese TV adaptation.

]]>

Introducing Rizz.farm - An AI-Assisted Lead Generation Tool for Reddit

2024-01-06T13:10:47.679944Z

After having soft launched Persumi in 2023, I quickly found myself wanting to explore better ways to promote the product. Unfortunately, most of digital marketing products are focused on three main areas:

Keyword bidding, e.g. Google Ads or Facebook Ads
Cold outreach, e.g. Apollo
Search Engine Optimisation (SEO)

They each has its own pros and cons. As an indie developer without a huge budget, none of these tools work effectively.

The Challenges

Keyword bidding and targeted ads are expensive

Bidding wars drive up the cost of targeted ads, sometimes so significantly they price out many small businesses.

Users are increasingly using adblockers, making targeted ads ineffective.

Cold outreach are rarely answered

Highly sought-after leads are bombarded with cold emails every day, making the tactic ineffective.

Most cold emails end up in users’ spam bin, never to be seen again.

Search Engine Optimisation is a lost cause

SEO’ed content are being produced hastily, making them low quality and untrustworthy.

Sophisticated users have learnt to ignore highly SEO’ed content from untrusted sources.

The Epiphany

Reflecting on my own behaviour in choosing a product or service, I realised that over the years I have developed a good sensory to weed out the noise. Namely, I am very skeptical with highly ranked websites, I always use an adblocker (uBlock) to block annoying ads, and both my personal and work email inboxes are overflowed with cold emails I never read them.

So how do I do research, say when I’m buying a new TV, or subscribing to a new software package? I read user reviews. And not just questionable reviews on Amazon or review sites, but on Reddit, where (mostly) real users discuss and review products and services.

Of course, like any other place, Reddit can be filled with bots and fake content too. However, in my experience these are easily distinguishable, especially paired with content’s upvotes and user’s post history. With Reddit’s popularity, this makes it a highly desirable place for people to both look for answers, as well as to provide answers.

So I asked myself, what if there was a smart lead generation tool to help me promote my product on Reddit? What should the tool be capable of?

The Solution

In order to really stand out, and provide tangible value, I’ve focused on the following areas for the lead gen tool.

Warm leads over cold leads

By searching for and engaging with relevant content, the leads the tool finds for you are warm and targeted.

Many of the leads are already highly ranked on search engines, further boosting the content’s reach.

Personalised and public

Content the tool produces for the user are context-aware and personalised, making them highly relevant and trustworthy.

Even better, the content are made available publicly, boosting the user’s reputation and reach even further over time.

Infinite scale at a fixed cost

The tool finds leads and suggest responses for the user 24/7 non-stop, at a fraction of the usual marketing costs.

The pricing is fixed and transparent, so the user can scale their business without worrying about blowing the budget.

The Bottom Line

Ultimately, I want a lead generation tool that really provides value to not only its user (the marketer), but its user’s users (the target audience). Instead of focusing on selling, it should focus on helping people. This is where Rizz.farm comes in.

How Rizz.farm Works

With Rizz.farm, machine learning is put to good use. It’s not just another AI wrapper, it actually leverages many of the latest AI innovation to really take lead generation to the next level.

Smart search

Rizz.farm widens the search radius to find the most relevant or highly ranked content for the user. It may find leads a simple Google search couldn’t. Behind the scenes, keywords are expanded and multiple search queries are submitted to search engines.

Continuous monitor

Rizz.farm continuously monitors the social media platforms (Reddit to start off with) to find new leads 24/7, delivering them right to the user’s inbox. Whether they are highly ranked popular posts or the latest post in a particular subreddit, Rizz.farm is able to quickly deliver them to the inbox without fuss.

Leads inbox, as simple as Gmail

Leads are automatically sent to the user’s inbox, where they can review and decide whether to engage with them. It’s as simple as using email.

“Rizz Score”

As mentioned earlier, the tools should focus on helping people, rather than hard selling. To make it easier for the user, Rizz.farm calculates a score based on how relevant, helpful and spammy the generated or user edited response is. This ensures it always publishes high quality content for the target audience, making it a win-win.

Self-Learning AI

Last but not least, by using state of the art AI technologies that adapts to the users, Rizz.farm is able to help them automate and scale their lead generation and reputation building with ease.

Taking It For a Spin

As soon as I’ve deployed Rizz.farm to production, I started using it to promote Persumi and dare I say I am very happy and impressed with its results. Not only does it find relevant leads right away, it drafts up responses that are super helpful and relevant to the OP’s questions or posts, and is very subtle in pushing the “promotion” agenda. I know a lot of subreddits shun or outright ban self-promotion, so it’s extremely important to have helpful posts.

If you are looking for an affordable way to promote your product, service, event or anything really, please give Rizz.farm a try. It comes with 7 days of free trial and 30 days of no-question-asked money back guarantee. For a limited time, you can also get 50% off for six months using the coupon code LAUNCH2024.

Happy lead generation and growth hacking everyone!

]]>

Comparison of AI OCR Tools: Microsoft Azure AI Document Intelligence, Google Cloud Document AI, AWS Textract and Others

2023-09-17T03:44:24.872189Z

For a project I’m working on, we are looking at different AI OCR (Optical Character Recognition) options that would allow us to import documents with various layouts and extract relevant data from them with high enough accuracy. Due to the nature of these documents and the information contained in them, it is paramount that there would be an easy way for us to train the AI models using our documents.

Without revealing exactly what types of documents we are working with, as they’re commercially sensitive, the basic premise is to:

Import a document that might be one or more pages, the document itself may be high quality PDF, or low quality scanned documents that may or may not be skewed
Each page may or may not contain relevant information we want to extract
Relevant information may be structured in different ways, but usually in a tabular form (but each “line item” can either be horizontal or vertical)

In our case, a traditional OCR solution wouldn’t work as the documents often contain heaps of irrelevant data, and for the relevant data, it needs to understand the context in order to work out which data elements to extract, and how they relate to each other. Therefore, a machine learning based OCR solution that could adopt to our documents is highly desirable.

The success of the project is measured by the end results of imported data:

How much data is detected and recognised
How accurate the recognised data elements are
At a minimum, 80%+ accuracy is needed in order to save time in manual data entry, and to reduce/avoid incorrect data from being checked by a human operator before entering the system

Available OCR Tools

There are a number of OCR tools available, for our project the minimum criteria are:

AI powered solution so it can be further developed and tweaked by us for our documents
A UI solution for data training, preferably easy to use so that non-technical team members can help with data training too
A quick and easy way to test and verify trained models
A solution that is compliant with the ISO standards relevant for us
Pay by usage pricing model that’s not cost prohibitive

The tools I have discovered and considered were:

AWS Textract
Microsoft Azure Document Intelligence (formerly known as Azure Form Recognizer)
Google Cloud Document AI
Rossum.ai
Super.ai
Eden.ai

All six options are AI based OCR solutions. Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP) are the obvious “big three”, having their own AI models, whereas Super and Eden are aggregators that farms AI calls to providers like AWS, Azure and Google, and provides a middle layer that could potentially make the end result better (or worse). It’s unclear what model Rossum uses.

After some consideration, Rossum, Super and Eden were all taken off the table due to:

Rossum and Super not having transparent pricing
Eden being an API aggregator built by a small team, it’s unclear on its usefulness and longevity

With the longevity of the company and product in mind, as well as transparent pricing and commercial support, AWS, Azure and Google remain to be the leading choices.

The Big Three: AWS, Azure and Google

AWS Textract

Out of the big three, AWS is our preferred vendor due to its market leading position as a cloud service provider, as well as our existing projects already using AWS.

However, upon closer inspection, it appears Textract does not allow custom data training. Textract is provided “as-is” using Amazon’s pre-trained models, it does not allow customers like us to provide their own training data to improve on the detection and recognition of different types of documents.

AWS’ machine learning blog has published an article on “end-to-end intelligent document processing solution“ in 2020, however it does not explain in detail exactly what the capabilities are, and the solution requires a complex architecture that needs significant amount of time and effort to set up and maintain.

Azure Document Intelligence

Azure’s offering, formerly known as Form Recognizer, is an end-to-end solution that offers custom data labelling and training.

Google Cloud Document AI

Google’s offering, similar to Azure’s, is also an end-to-end solution that offers custom data labelling and training.

Comparison: Azure vs Google

With the initial assessment done, we are left with Azure and Google, so let’s dive into the deep comparison of the two platforms.

We will compare the two platforms in several aspects:

The ease of initial set up, i.e. how quickly can we get up and running right away
Whether a “base” pre-trained model is provided for documents with high enough accuracy so we save time in customisation and training
The detection and recognition – how much data is accurately detected and recognised
The custom training process, how easy is it to label custom data
How fast or slow the training is
The end result – provided with the same training dataset, which platform offers better results

The Initial Setup

Microsoft Azure

Azure’s initial set up is relatively straightforward. There is a four-step dialog window to set up the required Azure resources, etc. After which, documents can simply be uploaded for immediate consumption - labelling and training, which will be touched on in later sections.

Google Cloud

Google works slightly differently in this case. To set up a project it’s very simple, there is literally only the “Processor name” to fill out. However, it does require a separate step to set up the storage bucket required.

Also, instead of allowing files to be uploaded within the same UI like Azure, Google requires files to be uploaded to the cloud storage bucket first, then imported into Document AI.

Auto Labelling Data

Microsoft Azure

“Auto-labelling” refers to the AI OCR’s ability to do the initial labelling automatically, to save time and effort in the manual labelling work. It can be seen as the starting point of training the AI model.

In this case, the Azure experience is significantly better. It allows files to be uploaded, then auto-labelled by “current”, “unlabelled” or “all” documents. This gives the flexibility to users (i.e. us) to choose when a document should be auto-labelled – as it does take a bit of time, sometimes if a document is not very well structured and would require more manual labelling anyway, then it’s best to skip the auto-label step.

Google Cloud

Google on the other hand, has a more cumbersome process. We first need to upload the files to the storage bucket, as mentioned before. Then, we select the path/folder of the uploaded files, and the “auto-labelling” is enabled or disabled for the entire import. Also, sometimes the import takes a long time, whereas on Azure it’s always instant.

More on the accuracy of auto-labelling later.

Text Detection and Recognition

A big part of OCR is obviously text detection and recognition. Even without training, a good OCR solution should detect and recognise clear text effortlessly.

During our initial research, we’ve found this article that compared the text recognition accuracy between AWS, Azure and Google. Their finding was that Azure and Google were comparable with a slight edge to Azure (Azure performed better on 2 out of 3 documents, and Google performed better on 1 out of 3 documents), both were way ahead of AWS in terms of accuracy.

Our own testing has somewhat mirrored their experience. We found that in general, Azure performed better in the number of detected text elements, as well as the correctness of these text.

On Azure, it’s very rare for text to be undetectable, whereas on Google, we’ve found several instances where the text is very clear, yet Google was unable to detect the text at all.

Similarly on accuracy, Azure has rarely detected clear text incorrectly, whereas on Google, a lot of times units such as x10*9/L get recognised incorrectly despite all instances appearing similar in their appearance (i.e. very clear), and sometimes dashes won’t get recognised (e.g. 10.0 - 12.50 gets recognised as 10.0 12.50).

Custom Labelling

During training, a big portion of time is spent on custom labelling, meaning we look for and assign text to our pre-defined data elements such as name and occupation - these are just fictional examples.

Overall, Azure and Google each has its strengths and weaknesses. Let’s go through them in detail.

Auto Layout

On Azure, there is a “Run layout” step that would recognise all the text elements, as well as tabular data in a given document.

Once “run layout” is performed, all the recognised text are highlighted in yellow, giving a quick and easy overview of the usable data elements.

Google on the other hand, does not offer a similar function, therefore labelling needs to be performed with more effort, more on this later.

Schema vs Schema-less

One key difference between Azure and Google’s solutions, is that the Azure solution is schema-less whereas the Google solution is based on defined schemas.

On Azure, we can add a “field”, and then assign the field a type (string, number, etc). A field can then be renamed or re-assigned with a different type at any time.

There is a special field called “table field”, we can create tabular data using this field type.

Google works very differently. Instead of adding fields on the fly, Google requires a schema (a.k.a. field definitions) to be created first. Once a field is created, and data trained, it cannot be edited or deleted.

One advantage of Google’s approach is a more definitive structure of OCR’ed data. When creating a label, not only do we have to choose a type (similar to Azure), we also can choose its occurrence logic, see the screenshot below.

Despite this, we found that the schema-less approach on Azure offers far better versatility without being confined to a pre-defined schema, making training new revisions of models far easier.

Labelling Data

Most of the time spent on training, is the manual labelling of data elements on the documents. Azure’s approach in this case is significantly faster and more accurate, compared to Google’s.

Microsoft Azure

On Azure, because of the “run layout” step, all text elements are already detected. So labelling them as part of the test results is very simple, you click on the yellow elements to select the text you want, then click on the table cell on the right to assign them.

However, if somehow Azure detects the data incorrectly, there is no way for us to provide a correction. In this case, we could choose to either skip the cell, or simply assign the cell with the incorrect data – if more training data is provided over time then these edge cases will not have a big impact on the final accuracy of the model.

In the screenshot above, sensitive data elements are blacked out for the purpose of this blog post.

Google Cloud

On Google, this process is significantly more involved. Due to not having a “run layout”-like step, we need to draw over the text to select it, and there’s no way to tell whether Google has detected the text before using the tool, so sometimes you’ll draw over some text and get an error saying, “Cannot create labels with empty values”.

Again, in the screenshot above, sensitive data elements are blacked out for the purpose of this blog post.

And because we can’t simply click on the highlighted text already detected, like on Azure, this makes the whole process significantly longer as we have to carefully draw over the text to ensure the boundary is accurate (for training purpose), sometimes we have to zoom in and reposition the document for accurate drawing.

The schema assignment is also less intuitive compared to Azure’s. Instead of having a table to easily see tabular data, Google’s schema is more like a series of key-value pairs, making glancing at assigned data much harder, and the key-value elements are ordered alphabetically, instead of logically like on Azure, making assigning data unnatural.

The one advantage of Google’s approach though, is that when the recognised data is incorrect, we can manually override it with our own correct data. However, after using both solutions for a while, we’ve found that Google’s has a much higher error rate compared to Azure’s to begin with.

Auto-Label Accuracy

In order to assess the auto-labelling accuracy, we have trained several of our documents with varying quality and layout. We then uploaded a new document with a layout that has not been trained.

The results are very telling.

Microsoft Azure

Azure has pretty much labelled things perfectly - all the data elements were detected correctly and within the correct contexts. The second comment block is also detected correctly.

In this case, the Azure auto-label has only missed the first comment block.

The screenshot above is blurred to protect the sensitive nature of it.

Google Cloud

Google’s result unfortunately falls short of expectation.

As seen in the screenshot below, Google has missed several data elements, and incorrectly detected elements that should not be part of the result. Again, the screenshot is blurred to protect the sensitive nature of it.

This suggests two possible things about Google’s OCR solution:

The default text recognition is poor, therefore it’s missing data elements even though they are clearly presented, in the same way as other elements
The neural network for language models is poor, therefore it incorrectly detected text that shouldn’t be part of any result

Auto-Label Result Verification

Microsoft Azure

Once Azure’s performed the auto-labelling, it’s very easy to check the results, as seen in the blurred screenshot below, we simply look at the table to ensure the correctness. However, as mentioned before, we can only correct detected regions of text, not the content itself.

Google Cloud

On Google, it is more involved. As seen in the blurred screenshot below, it requires a lot of scrolling to find the auto-labelled data. Alternatively, we can hover over the highlighted text to look at the auto-labelled text, although in this case, the pop-up gets in the way of other data which makes it an annoyance at times.

Curiously, despite having a schema, and setting certain elements to “required once” (per schema), Google still went ahead and falsely detected multiple results per schema.

But as mentioned before, Google does offer the ability to manually correct the text content, which is a big plus.

Data Training Speed

We have found training to be significantly faster on Azure.

With about 45 documents, it takes about 30 minutes to train on Azure.

Google takes about twice as long, about an hour to train, and another several minutes to deploy the trained model. Azure does not need a separate deployment step.

Interestingly, on Google, training a model requires having enough test data. So documents need to be split into training and test groups. It is unclear whether the test data is used for actual training.

There is also a required number of labelled elements (10 minimum, 50 recommended) before a model can be trained. Azure has no such limitations.

Data Regions and Compliance

While both Azure and Google are compliant to many standards (ISO, GDPR, etc), Google’s Document AI can only be hosted out of their US or EU regions. Whereas Azure has no such limitation and can be hosted in any of their available regions.

This has a side effect on the performance of UI operations. Azure is very quick and snappy due to it being deployed in our local AU region, whereas Google due to it being deployed in the US region, is a little slow every time you open and close a document for example.

Conclusion

With the in-depth analysis done, it is no surprise that in the end we went with Microsoft Azure’s Document Intelligence for our AI OCR needs. I hope these findings are useful to other people too.

]]>

How I Built a Mostly Feature-Complete MVP in 3 Months Whilst Working Full-Time

2023-08-09T09:28:12.135201Z

A few weeks ago I soft launched an MVP - you are looking at it right now.

In this post I’ll talk about the features, the tech stack and the globally distributed infrastructure behind building this MVP, and of course, with a sprinkle of learnings too.

The “MVP”

Deciding what makes up an “MVP” is always interesting - I’ve heard some saying if the product isn’t embarrassing you’re releasing it too late, and also if the product is embarrassing, you’re not gonna make it.

For me, I’ve always had the idea of building all the essential features as part of the MVP, with one or two “hero” features that would differentiate the product from the competitions, and then build out more premium features over time.

If you can’t already tell, Persumi is a content creation platform with some social networking features. It may sound bland, but what I believe makes it stand out, is the desire of putting the focus back onto the content, rather than the VC-fuelled, ever increasing appetite for more ads and user hostile features.

The Long Nights and Weekends

There is no magic beans for productivity - especially when I have a full time job. Working on a side hustle means giving up on almost all social and entertainment activities. It’s not for everyone, but I didn’t mind it too much. Being an introvert definitely helped - I was happy to see night by night the MVP gradually taking shape to become more and more real.

Looking back, I spent about three months to build out most of the MVP, then another week or two on infrastructure, and another week or two for polishing, all whilst having a full time job.

It’s been a journey, I’m glad that it “only” took me 3-4 months to get to this stage, as initially I estimated for a 6+ months MVP build.

The Features

With all that in mind, I’ve set out to build the essential features that make a blogging and social networking platform:

Short form content like a tweet
Long form content like a blog post or a book chapter
RSS feeds
Communities similar to forums and sub-reddits
Direct messaging between users
A voting (like/dislike) system
A bunch of CRUD glue pieces to make all these things work

The “Hero” Features

Beyond these seemly unremarkable features, I’ve also had in mind two key features that would differentiate the platform from the rest:

The “persona” concept, whereby each user is allowed to have multiple personas to hold different content or topics of interest, e.g. a persona for professional stuff, a persona for gaming stuff and a persona for travel stuff, etc
AI generated audio content for text (also known as Text-to-Speech)

These two “hero” features are what drove me to build Persumi in the first place. Together, they solve some very real pain points for me, namely:

Following specific topics of interest from people is difficult, with the algorithms taking over people’s home feeds, there are simply way too much noise, thanks to VC-fuelled “user engagement” metrics
Content consumption on the go (e.g. during commute or during workouts, etc) is becoming more and more prevalent, but the traditional platforms haven’t adopted to this new lifestyle other than shoving short form content down our throats

There’s also a third “hero” feature: the Aura system. Unlike the upvote/downvote or like/dislike buttons in many social platforms that only serve the algorithm to push more content to you, Persumi’s Aura system keeps track of user’s content quality over time, and would punish the low quality content and promote high quality content using visual cues - lower quality content has a much lower contrast making them easy to ignore. In the age of social media, self-curating content becomes essential to keep a platform healthy, engaging and usable.

The Non-MVP Features, a.k.a. The Future

There are many features that didn’t make the MVP cut, most of these are value-added features that will eventually make their way into paid subscriptions - if Persumi gains enough traction to attract users who don’t mind paying for premium features.

A prime example of such paid features is ones that help users monetise their content, e.g. ad revenue sharing and paid subscribers (like Patreon).

I also have the ambition of building out Persumi’s features so it can eventually compete against the likes of LinkedIn and Tinder.

Wouldn’t it be better for the world to have a platform like Persumi that doesn’t focus on dark patterns and exploiting users? 😉

The Tech Stack

Over the past decade or so I’ve mainly worked with two tech stacks: Ruby and Elixir. So naturally, Persumi was going to be built using one of them.

After some consideration, I’ve decided to go ahead with Elixir, the main reasons were:

Elixir and Erlang/OTP support distributed systems out of box
I’ve been writing more Elixir than Ruby lately, so I’m more productive in Elixir
I really wanted to try and use LiveView in production
I prefer Phoenix’s application architecture more than Rails’

On top of Elixir, I’ve decided early on a few other things to go with it:

Tailwind for CSS
Postgres for database, preferably a serverless option
A search engine
An easy to maintain infrastructure that doesn’t cost an arm and a leg

Elixir

I first discovered Elixir in 2014 while I was still actively involved in the Ruby and Rails communities, but it was two years later that I had the opportunity to really dive into it. I built a few open source libraries to help me learn Elixir and OTP:

Crawler - a high performance web scraper.
OPQ: One Pooled Queue - a simple, in-memory FIFO queue with back-pressure support, built for Crawler.
Simple Bayes - a Naive Bayes machine learning implementation. Hey, I was doing machine learning before it was mainstream! 😆
Stemmer - an English (Porter2) stemming implementation, built for Simple Bayes.

With a decent amount of experience building Phoenix web apps over the years, I unfortunately never had the opportunity to use LiveView…

I guess that’s finally changed now.

LiveView has been amazing, not only does it drastically reduce the amount of front-end code you have to write, make the entire web app feel super snappy, but it also virtually eliminates any front-end and back-end code/logic duplications. It is such an awesome piece of technology that improves both the user experience and the developer experience.

Petal Pro

During my initial tech research, I came across Petal Pro which is a boilerplate starter template built on top of Phoenix. It handles things like user authentication which almost every web app needs, but is somewhat tedious to build.

Petal Pro isn’t free but it ended up saving me so much time. I also started contributing small bug fixes and features to it too. If you are about to build something in Phoenix, check it out!

Tailwind CSS

As I kept progressing my career, there have been fewer and fewer opportunities for me to write front-end and CSS code. Last time I rebuilt my blog, I used Bulma - that was 2019. Since then Tailwind has gained a lot more traction, so I wanted an excuse to try finally give it a shot.

There is a debate on how many new things you should try for building your MVP - the more you have to learn, the slower your MVP progresses. That said, given CSS is reasonably straightforward, I figured it wouldn’t slow me down too much, if anything, Tailwind’s flexibility might just eventually make up any time lost in learning.

I’m happy to report that it is indeed true - by using Tailwind, it became significantly easier for me to customise my components and elements. I can see why it became so popular. It’s not for everyone, but I like it.

Postgres

Choosing Postgres as a database was a no brainer, given how popular and versatile it is. I did briefly consider NoSQL options like DynamoDB but quickly wrote them off as I needed an RDBMS to get things off the ground quickly, and the DB is unlikely to be the bottleneck for a long time anyway.

In Elixir, the Ecto library works wonders for Postgres.

Later in the post I’ll touch on how I deploy and run Postgres in production.

Search Engine

For a search engine, my requirements were:

The ability to search across multiple fields of a schema
The ability to rank them
The ability to have typo tolerance, word stemming and other similar language features to make search more intuitive
The ability to search multiple languages, including CJK (Chinese/Japanese/Korean) characters
Simple to run
Cheap to run

Using Postgres’ full text search was probably going to be the simplest but it doesn’t offer all the functionalities I need without a bunch of set up and manual SQL queries so I didn’t pursue it.

Elasticsearch on the other hand, offers good search functionalities but takes a bit of effort to set up and maintain, and can be costly to run.

After doing some more research, I found the following three options that would fit my needs:

Both Meilisearch and Typesense are open source, with commercial SaaS offerings, whilst Algolia is SaaS-only.

It’s been an interesting journey. I started with Typesense as I liked what I read, but I quickly discovered that it doesn’t search Chinese characters properly.

I then turned to Meilisearch. I especially liked the fact that they offered a generous free tier SaaS to get you off the ground running. Spoiler: during my implementation they did a bait and switch and removed the free tier.

At the time the Elixir support for Meilisearch wasn’t up to date, so I ended up contributing to a community library to add the features I needed.

Curious timing, after Meilisearch removed their free tier, I discovered that even though they officially support searching for Chinese characters, the implementation wasn’t perfect. I found some edge cases where characters weren’t detected properly, making the search results unreliable.

So, my last hope was Algolia. Despite them being the more expensive option out of the three, it does offer a free tier. It turns out, their search results for Chinese characters were much better than Meilisearch’s. Luckily, re-implementing the search from Meilisearch to Algolia didn’t take too much effort, it was pretty much done in one night.

Infrastructure

Early on during the development I’d already determined I wanted to try Fly and Neon, for web and DB, respectively.

I am in no way associated with either company, I was curious about Fly due to its tie-in with the Elixir community (Phoenix Framework’s author Chris McCord works there), and Neon due to its serverless nature.

Globally Distributed Infra

With Fly, the infrastructure automatically becomes globally distributed as soon as I started provisioning servers in more than one region. As of the time of writing, Persumi is deployed to US West, Australia and EU.

Despite being simple to use, making Fly work initially actually took quite a bit of finessing due to its incomplete official documentation and flakiness. Some of the services were having issues during the course of my MVP development. Worse, they don’t report (or sometimes even acknowledge) the issues unless they are region-wide outages. To this date, I believe their blue/green deployment strategy which was recently introduced, is still buggy, I often have to use their rolling deployment strategy instead. Deployment logs were provided to Fly but I think they’re too busy with other things…

Still, I’m sticking with them for now due to the ease of use after the initial hurdle, and their globally distributed infrastructure without asking for my kidney.

To augment Fly’s web servers, I also use Cloudflare’s CDN as well as R2 to serve asset files and audio files.

Funny tangent, initially I used Bunny for asset files and CDN, as I misread Cloudflare’s terms and thought I couldn’t serve audio files from Cloudflare. Bunny worked okay but their dashboard for some reason was painfully slow - not a good look for a CDN company. Like the search engine switch, it didn’t take me too long to switch over to Cloudflare.

Serverless Postgres

There are a few options to run Postgres:

Run on a standard server for maximum portability, but it requires more server maintenance overhead
Run on AWS RDS/Aurora or a similar managed service, easy but can be costly
Run on a serverless option such as Aurora Serverless or Neon

For my use case, I think option 2 or 3 are better fitting. As I mentioned earlier, I started the experiment with Neon.

Neon worked well initially, until I started deploying Fly instances in multiple regions. Due to Neon being only available in one region (I chose US West), and I live in Australia, the round trips between Fly’s Australian instance and Neon’s US instance were a show stopper - especially when complex DB transactions were involved. Actions sometimes took seconds to complete, yikes.

Despite Fly not offering a managed Postgres service, I ended up trying it anyway due to its distributed nature. After incorporating Fly Postgres in the app, all DB operations immediately became more responsive. Paired with LiveView, it feels like running the application locally.

The current Persumi infra looks like:

1 x Fly instance in US West, always on
1 x Fly instance in Australia, auto-shutdown when there’s no traffic
1 x Fly instance in Netherlands, auto-shutdown when there’s no traffic
1 x Fly Postgres writer instance in US West, always on
1 x Fly Postgres read replica instance in Australia, always on
1 x Fly Postgres read replica instance in Netherlands, always on

With this setup, I think I’m quite happy with the cost and scalability balance - it costs ~$20/m to run, with the potential of both vertical and horizontal scaling with ease.

The Missteps

The search engine and CDN swaps mentioned earlier certainly took away some of my time, but they were nothing compared to a major misstep I encountered.

And that was: the choice of how machine learning is done.

Let me explain.

Machine Learning, and Inference

Even before I started the first line of code, I already painted a picture in my head on the machine learning needed: a TTS (text-to-speech) model that I could run inference locally on the instance.

The reason being I believed it was the more flexible approach to gradually improve the inference and therefore the end result by training my own AI models over time.

Given I didn’t want to rent expensive GPU instances, I opted for fast TTS models that could do near real-time inference on CPUs. I used Coqui TTS.

The resulting out-of-box audio wasn’t great, but I kept pressing on.

The show stopper came when it was time to deploy everything onto Fly. Due to Fly’s architecture (they deploy small-ish Docker images, < 2GB each, onto their global network), I struggled to keep the Docker image file small enough to be able to deploy. With Coqui TTS, I would need Python and all the dependencies that resulted in a Docker image around 4-5GB in size.

With my tunnel vision, I then chose to offload the entire Python and Coqui TTS dependency tree onto Fly’s persistent volumes. I knew it wasn’t a great option, as that meant my infrastructure (other than the database) was no longer immutable.

Sometimes it’s necessary to take a step back, re-evaluate, and then press on in a different direction. Which thankfully I did.

The new direction is quite simple really: instead of performing inference locally, use an external service instead.

After doing a quick comparison between the offerings from AWS, Azure and GCP, I ended up using Google’s TTS. Honestly I think I would’ve been happy with any of the options, they all seem to have decent neural based TTS.

In hindsight, these giant corporations have much more resources and expertise to train better models than I ever could on my own.

The end result:

The TTS sounds significantly better than before
It’s just as cheap to run (Google offers a certain amount of free TTS API calls per month)
It no longer needs complex Python calls and FFmpeg calls to make local TTS work
The Fly infrastructure is simple and immutable again

In hindsight, I never should’ve even entertained the idea of running ML locally on CPUs, no matter how simple and efficient a model might be.

That said, with TTS, it wasn’t as simple as just calling the APIs and getting the perfect resulting audio back. Some pre and post processing were needed, but that’s a topic for another time.

More Machine Learning

The cherry on top - now that Google’s APIs were integrated into the app, I ended up also using Google’s PaLM 2 to do text summarisation (it was initially done locally too) as well as for a ChatGPT-like AI prompt service, to power Persumi’s AI writing assistance feature.

The Closing

If you read this far, thank you! I hope you enjoyed reading (or listening) to this post. Please look around and kick tyres, I would love your feedback on how to improve Persumi.

]]>

Some changes to improve the user onboarding experience are coming. There will be a guided tour to explain the concepts of personas and expressions better. Stay tuned!

2023-07-22T00:33:58.092811Z

Some changes to improve the user onboarding experience are coming. There will be a guided tour to explain the concepts of personas and expressions better. Stay tuned!

Hey, welcome! If you're also building your own product I'd love to hear your stories. 😊

2023-07-17T04:56:31.876027Z

Hey, welcome! If you're also building your own product I'd love to hear your stories. 😊

Welcome to Persumi - A Modern Platform for Content Creation

2023-07-17T04:18:07.386547Z

Like many software engineers and indie hackers, I dream of and experiment with different product ideas every now and then.

Five years ago, in 2018, I toyed with an idea where personas and expressions are the enablers for showcasing people’s talents and interests.

I called it, Persumi.

The Origin

The original idea was to break the pattern of modern day social media platforms like Facebook, Instagram and Twitter, where every person is hidden behind a two-dimensional facade. Words and photos are often used to skew the way people see who we are.

After a while, everything becomes bland as information is being overloaded and people’s attention span starts to decrease drastically. The signal to noise ratio is so low that more and more algorithms and tracking are invented to help curate content so advertisers would keep spending their dime on those social media platforms.

It’s a vicious cycle. There is now even a term for it: enshittification.

Enshittification is when an online platform becomes more monetized and less user-oriented the longer it lasts.

The Goals

With Persumi, personas and expressions are the key concepts to modernise content creation, curation and consumption in a simple yet meaningful way.

I set out a few goals for Persumi, in the hope to change the social media landscape for the better:

Find ways to encourage the creation and consumption of long form content
Make it easy for people to create vastly different topics without confusing their audiences
Make it easy for people to follow different topics, rather than relying on following users
Allow people to decorate their pages with their own personality and style via HTML + CSS, bring back MySpace!
An API-driven, open marketplace that allows developers to build and extend advanced functionalities

Some example use cases outlined in my original “pitch deck”:

social personas for friends, family, work…
interest personas as a gamer, traveller, photographer…
functional personas for travel plan, marketing campaign…

It kind of makes sense, but there is one problem.

The Problem

The age-old “chicken-and-egg” problem of starting a platform that thrives on user-generated content - how do I get early-adopter users without a decent amount of users and content already?

These goals might be nice, but they probably aren’t enough to convince users to drop what they are used to for something experimental and without users or traction.

So I parked this idea…

Revisit the Problem

A few years have gone by, and with a whole pandemic thrown into the mix, this Persumi idea has all of a sudden awakened me yet again.

In early 2023, after wrapping up my last freelancing project I decided to tackle Persumi’s chicken-and-egg problem once more.

I reflected on how my own content consumption behaviour has changed over the past few years. Instead of reading physical books, e-books or blogs, I consume more and more podcasts and audio-books. Instead of consuming content fully concentrated, I often consume them while on the go or during my workouts.

Then a light bulb moment occurred - I don’t need to build a massive social network platform to compete with the likes of Twitter. I just need to build a “simple” blogging platform for my own use and my own content. If I find it useful, other people will find it useful too. It doesn’t need to go viral, it just needs to be steadly discovered by like-minded people who share my frustration and vision in content creation and consumption.

Iterate the Idea

With a few more years of experience of leading teams and building products under my belt, I thought about how to iterate on the original idea, and came up with three main “twists” if you will:

Long form text content needs to be consumable in audio form, to suit the modern lifestyle
An API-driven open marketplace is way too ambitious for a new platform, build the core concept first, then iterate
An aura system that helps in both curating content as well as driving away bad actors

The rest of the original goals are still as relevant as before. In hindsight, these are pretty obvious product direction pivots, the result is an MVP that builds on a simple core concept - to turn long form text content into audio. And to make it easy to organically grow the platform and audience.

Build a Sustainable Business

With the chicken-and-egg problem behind me, it’s time for me to think about how to build a sustainable business to support my product ambition.

Interestingly, in Feb/March 2023, I was thinking about what the VC funding model did to the social media platforms we all love and hate, and why it’s a bad idea. The fact that then during the short few months of developing Persumi, both Twitter and Reddit seemed to be actively imploding themselves just proved my initial hypothesis:

To build a truly user-oriented platform, it needs to be built for users and content creators, rather than investors and advertisers.

That means, no obnoxious ads, no annoying pop ups, every visitor can view content without being forced to sign in, etc, etc… We do what VC-backed companies don’t.

If you’ve seen the landing page then you probably would have noticed a few things either available now or later to help create more value for users, such as:

The hero feature - AI audio, will be a paid feature, however, every user will be able to listen to audio posts, as well as to preview their own content with audio
Advanced expressions such as HTML and CSS for those who want to customise their pages to their own style
Ad-revenue share and paid subscribers - coming later, to make Persumi a platform that shares its success with content creators

Keep on Building

The MVP was essentially built in three months in my spare time. It wasn’t easy juggling between having a day job as a CTO, and moonlighting on my side hustle every evening and weekend.

This is just the beginning. I have many things planned for Persumi, some of which are already mentioned on the landing page, but there are much more to come. Early alpha and beta users are also able to help shape what this platform can become.

In the next week or two I will write about how I managed to build Persumi in three months - the tech choices, architecture and feature prioritisation I went through to make this product a reality.

Like what you’ve read or listened to? Why not sign up and try the platform yourself, the Pro plan is free during the alpha and beta testing stages. Also check out Persumi’s founder @fredwu’s profile here to see what personas and expressions look like and can achieve.

Update: Read about How I Built a Mostly Feature-Complete MVP in 3 Months Whilst Working Full-Time, it covers the features, the tech stack, the global infrastructure and the machine learning…

]]>

Tips for Job Interviews as a Junior Software Developer

2020-09-27T05:56:03.000000Z

As coding bootcamps such as Coder Academy and General Assembly churn out more and more software developers, and as more and more people start to realise the importance of software, companies these days are facing an increased amount of candidates applying for junior dev roles.

Recently we had to take down our job ad for a junior full stack React and Elixir role only a few days after posting it due to having received about 300 applications. Suffice to say, the competition is fierce at the entry-level end of the software engineer spectrum.

Software is eating the world. - Marc Andreessen

In the past three months alone I’ve had to sit through over a hundred interviews, many of which were for junior roles. Some candidates did really well but unfortunately the vast majority did quite poorly. In this article I am hoping to offer some tips and insights as a seasoned interviewer and hiring manager, to those who might be struggling landing an entry-level software engineer role.

I’ll break the tips into three sections: before an interview, during an interview and after an interview.

Before an Interview

This is arguably the most important stage of your interview - before it happens! Prepare yourself with enough knowledge and confidence in both software development and conversational skills in order to ace the interview and stand out amongst a sea of candidates.

Study Software Engineering Fundamentals

I get it, bootcamps and many online tutorials and courses focus on teaching the practical things to turn code into products. But just like running or even walking without correct postures, it can be incredibly dangerous over time.

My recommendation is to read on the basics of object-oriented programming, functional programming and some design patterns. For instance, build up some basic understanding of how the JavaScript prototype works or how the Ruby object model works can be extremely beneficial in progressing your technical capability on application design and architecture.

Get More Project Experience

Not having any substantial project experience on your CV is a sure way to significantly lower your chance of getting an interview.

Do junior developers really need substantial project experience, you may ask? Abso-fucking-lutely!

Build your own side projects, participate in open source projects, pick up some freelancing work - the choice is yours. These won’t be your most technically accomplished work, but they will help you accumulate experience, and more importantly to demonstrate your capability to your interviewers with real world experience.

Why Software Development?

I want you to put your hand on heart and answer this question honestly. If your answer is more money or FOMO, that’s okay, but be prepared to be disappointed with the reality - if you are not in this field because of your passion for building products or solving problems, you will find this a long and hard slog.

If you are passionate about building software, then let me tell you this - when you are starting your career you will have to give up your work life balance for a bit. That is, if you want to boost your career and progress quickly beyond being a junior developer.

My interest is in the future, because I am going to spend the rest of my life there. - Charles Kettering

Looking back at when I started my software career many moons ago, I never regretted giving up my social life early on in my career: I worked extremely hard, often did many jobs - a day job as a dev, a night job as a dev, freelancing and doing open source work all at the same time. Some would probably frown and say that I had no life, but I choose to believe that I was making sacrifices so I can become good at what I love doing in the shortest amount of time.

Research the Company You Apply For

If you are one of those who spam your CVs to every job opening, please consider changing your tactic. A well written cover letter will not only increase your chance of getting an interview, but also allow you to ask relevant questions during the interview.

Check out the company’s website and any social media pages on LinkedIn, Facebook and Instagram, etc. Do a bit of “stalking” on the people who work in the company, sometimes this will give you some insights into the type of work environment and culture the company has.

Doing enough research is not only for your own good - knowing whether you will enjoy working there should you end up getting a job offer, but also demonstrates your ability to do research and independent thinking to your interviewers.

Read, Listen and Communicate

This one might not be immediately obvious because it’s very high level and somewhat vague, bear with me.

Some people might be deep thinkers but really struggle to get their points across, and to communicate their ideas and thought process in a clear, concise and useful way. My advice is to read blogs, listen to talks and learn from other people on how they communicate. When you read and listen, pay more attention to how sentences are structured and how titbits of information are given.

The most powerful person in the world is the story teller. - Steve Jobs

And don’t forget to practice! Get comfortable talking to others on various topics in software development.

During an Interview

Hopefully the tips given thus far would help you secure an interview. Now it’s show time!

Turn Up On time, Have Equipment Sorted

Whether you are attending an on-location interview, or more likely during Covid, an online interview, make sure you turn up on time!

If it’s an online interview, also make sure you have whatever software needed (Zoom, etc) installed and tested. I’ve had a few occasions where the first five minutes of the interview was spent on the candidate trying to get their mic or speaker working - it’s not a huge deal, but it does show the lack of preparation and organisational skills.

A Smile Goes a Long Way

Understandably, interviewing is stressful. However, remember to put on a smile when you greet your interviewer and when you do small talks. Everything needs a balance of course, when you’re giving technical answers it’s probably not a good idea to smile.

If you know you get uncomfortable talking to strangers or when under pressure - do more practices! If you can’t find a friend or don’t have a partner to practice with, use a mirror!

Learn to Say “No, I don’t know”

When you don’t know something or can’t recall the details of something, just say it. As an interviewer, when I ask you a technical question I can see through your hmms and ahs. A “sorry, I don’t know” is a much better answer than one that doesn’t make sense.

Bonus point: “Sorry I don’t know, but I will look it up after this interview.”

Read Social Cues

This one can be extremely easy or extremely hard depending on your personality. For those who might struggle with reading social cues, try pay more attention to what type of questions you were asked.

If it was an introduction of yourself to kick off the interview, make it more than just two sentences but also under five minutes - this is usually a warm up for you to find your feet and calm your nerve, and for the interviewer to extract any interesting information as a discussion point later. It doesn’t have to be your memoir and you shouldn’t robotically talk through your entire education and work history.

If it was a technical question, always try to time-box it. Don’t drag on and repeat yourself just because you have already answered it quickly and concisely. When in doubt, ask the interviewers whether they have anything in particular they were looking for.

If it was an open ended question, try to explain your thought process as you explain it. As an interviewer, I’d hate to ask an open ended question and get a short yes/no back. At the same time, give interviewers room to interject and ask follow-up questions by pacing yourself.

Don’t Speak Too Fast

Okay, you might be very nervous, or English might not be your first language, so you tend to speak fast to “get through it”.

If that’s the case, work on your speech pattern, make it concious enough that you will recognise it when you speak too fast. If interviewers can’t get your points clearly, they might not always ask you to clarify - because if you kept doing it repeatedly, you would already be written off as a suitable candidate.

Be Humble

I totally understand that in a competitive job market you want to present yourself in the best light. Just be confident and speak the truths.

As someone who’s been doing interviews for years, I really dislike people who overstate their capability either on their CV or during the interview because these are often very obvious. Claiming to be an Elixir expert but not knowing how the supervision tree works is a sure way to score a black mark, for example.

Show Passion and Drive

To counter the point on over-selling yourself, what works better is to show interviewers your willingness and determination to learn. Remember, you are still a junior therefore we don’t expect you to know everything we know.

What we do expect, is your commitment to learning - not by saying it, but by demonstrating it. Remember the earlier tip on getting more project experience?

Ask Questions

Usually towards the end of the interview, we’d ask you if you have any questions. Don’t say “no”.

Doesn’t matter how much research you’ve done on a company, there are always insights someone who’s working at the company can give you. How do they manage the delivery? How do they support learning? What makes them enjoy working at the company?

Use this opportunity to gain insights into why you should be working at this company, and to demonstrate your interests in working at the company to the interviewers too.

After an Interview

After completing an interview, it’s an opportunity to do a little retrospective and do a follow up when necessary.

Close the Knowledge Gap

Don’t over-analyse on what you could have or should have said. Instead, spend your energy on solidifying the topics you couldn’t or didn’t provide good answers for. If you’ve said “don’t know” or “not sure” during the interview, now is your chance to make it “sure”.

Follow Up

If during the interview there have been things mentioned that you could provide links or more information on, be sure to act on it. You never know, maybe the clarification is just what the interviewers needed to help them make a decision.

To all the junior developers out there - enjoy the journey! You are in for an amazing ride, make every second count.

If you enjoyed this article, checkout my other tips articles:

]]>

Tips for Writing a Good CV / Résumé

2020-05-20T08:59:24.000000Z

Due to COVID-19, not many companies are hiring at the moment. The company I work for therefore is in a very fortunate position to still be thinking about growth and hiring.

As a hiring manager for almost a decade now, I’ve personally reviewed thousands of job applications and CVs, and many hiring managers would probably agree, the vast majority of CVs are terrible. Let’s change that!

During COVID-19 where more and more people are either losing jobs or having their work hours cut, we are experiencing an increased amount of applicants to our job ads. I’d say on average I spend about 30 seconds per applicant due to my busy schedule - most hiring managers are busy people, it is therefore crucial for candidates to realise the importance of having a CV that is clear, easy to read and most importantly sells yourself. And if you have a cover letter, which I highly encourage that you do, congratulations you just bought yourself another 30 seconds. ;)

I’m writing this post mostly from my own perspective - as a hiring manager in a tech company in the western culture (we’re based in Australia). Understandably, different cultural backgrounds and regions may have their own conventions, but certainly in Australia and many similar western cultures, there are things that you do and don’t do on a CV, and there are things that may help your CV stand out. Let’s talk about these things.

At the end of this post I will also share a copy of my own CV to help illustrate my points.

30 Seconds? Surely It’s Unfair to the Candidates

Yes, I agree, to think that you are only given 30 seconds for your perhaps carefully crafted CV and cover letter is definitely soul-crushing. But it is unfortunately the reality. I work for a company where I can still do the first round of vetting myself, many large corporations would use algorithms and/or HR people to reject your applications based on keywords and other things.

Knowing the reality and the constraints, there are a few things I’d like to address in the hope of improving your CV and your chance of scoring an interview, and in turn, helping myself and other hiring managers out there to have a better candidate CV screening experience.

Have a Pronounceable Name or Alias

This one surely would raise some eyebrows - you might think that your name is your identity and you should not change it for anyone. True, however, the reality is that a hard-to-pronounce name discourages your profile to be shared and spoken about. Why not add a pronounceable alias if means there’s an increased chance of getting an interview?

For clarity, I personally would never reject a candidate based on their name (or their cultural background for that matter), but I know some hiring managers might, and for some of them, they are NOT doing it on purpose. However, I have on several occasions had to ask a candidate how to correctly pronounce their name.

A Short Blurb on Who You Are

As a hiring manager, I care about who you are as a person - if you can summarise who you are as a professional in a sentence or two, it will help me determine whether you might be a good fit or not.

As an example, here’s a blurb about me:

A passionate and hands-on software executive with two decades of experience and an entrepreneurial mindset.

A long time open source developer who has created and contributed to a few dozens of projects, including Ruby on Rails.

In two sentences, I explained my industry experience as well as my open source contributions - two things that help define who I am as a working professional. It also invites more questions from hiring managers: what kind of things have I done as an entrepreneur; what other open source projects have I contributed to?

Work Rights

Many companies have restrictions or policies around who they can hire based on their residency and visa status. If you are not a resident or are on a particular visa, make it clear in your job application so you don’t end up wasting time for the employer and for yourself.

List Keywords, But Don’t Overdo It

In the tech space it is important to have keywords visible to highlight your skills. If you are a software developer, your tech stacks should be clearly stated in your CV. As a hiring manager, if I am hiring a PHP developer, I expect to see PHP mentioned in your CV. There are of course exceptions, for example when we were hiring Elixir developers I did not expect to see Elixir as a keyword simply due to the supply constraint.

It is a balancing act however - I’ve seen CVs where candidates put 20-50 keywords on their CVs. I’m sorry but unless you are extremely gifted, you cannot possibly be good at all those things. Do not put keywords on your CV simply because you’ve read an article on the subject.

Oh, and unless you’re going for a data entry role, I honestly don’t care about your Excel skills…

Do Not Overstate Your Capability

Similarly, try to avoid overselling your capability. I once interviewed a candidate who claimed to be an “expert” on Ruby. We were actually hiring for a non-Ruby position, but given the candidate’s CV, I questioned him on some advanced Ruby subjects during our interview and he struggled all the way through and was sweating bullets. Suffice to say that he did not get the job.

Be confident, but also be honest and be humble. Lying on your CV to get an interview is a waste of everyone’s time.

Keep Things Short

As I mentioned in the beginning, I spend on average 30 seconds on each CV. Keep things short and easy to read! I really don’t care about how awesome you were in your last dozens of projects - these will get covered during interviews.

On a CV I expect short and concise blurbs on what you did in each role. Also, take recency into account too - if you’ve been working in the industry for a decade or two, what you did 20 years ago really doesn’t matter as much, so save yourself some time and cut things short.

For example, here’s the blurb for my current role:

Leading a department of 25+ engineers to make great child care and education software. As part of the leadership team and reporting to the CEO, helping building and turning the company into a market leader.

And here are the blurbs for my older roles:

Yes, the blurbs for my older roles are left empty intentionally.

Now, again there are exceptions. If something happened a while ago but is interesting and relevant, do tell! For example, here’s the blurb for my oldest “role”:

Built my first ever website using Microsoft FrontPage Express, on a Pentium 166Mhz computer, uploaded via a 33.6kbps modem.

Explain Over-Qualified Titles

There were a few times where a “CTO” or even a “CEO” applied for a developer role. In most cases it wasn’t about over-qualification, but about what the candidate wanted to achieve professionally. So, either in the CV or in the cover letter, explain what you are looking for in your next role, otherwise you run the risk of being assessed as over-qualified.

Spare the Personal Details That Are Too Personal

This is predominantly a culture thing as I’ve seen it from mostly candidates of certain cultural backgrounds. I really don’t care about your age, gender, marital status or favourite sport. These things do not define who you are as a professional - we might talk about your favourite sport and food during the interview but they are irrelevant on your CV.

Space Things Out

Look up CRAP Principles - make sure your CV has enough white spaces and contrast, and has fonts that are readable! Scrolling through walls of text is no fun and is a sure way to get your CV dismissed.

2-4 Pages

This is not scientific, for me personally I prefer to see CVs of 2-4 pages. Use the length as a constraint to cut things down. There were several occasions where I ran into CVs with 10+ pages. I guarantee you, unless a hiring manager is extremely bored, he or she does not have time to read your War and Peace.

PDF Over Word

When possible, submit your CV in PDF format instead of Word format. Now, sometimes if you use a recruiter you’ll be asked to submit your CV in Word format so they can ~~fuck it up~~ add their branding. A PDF formatted CV ensures the correct formatting and layout always get shown to the hiring managers.

Cover Letter

Always attach a cover letter when possible, but keep it short too. Given the amount of CVs a hiring manager needs to go through, having a crafted cover letter is another way to grab their attention and increase your chance of getting an interview.

Don’t repeat the same information in the cover letter though. Your CV is about the facts of your experiences, your cover letter should be about your thoughts on why the company should hire you. Focus on the value you can bring to the table.

Find A Referral

When possible, find someone who can refer you. A referral gets preferential treatment during the CV screening stage and does not suffer from the same 30-second fate.

Pleasing Design

This one is a “nice-to-have”: if your CV is really well designed, you would earn another 30 seconds of my attention. ;)

These are the main points, hopefully they are helpful. To help illustrate, here is a copy of my own CV, with contact details removed.

If you enjoyed this article, checkout my other tips articles:

]]>

Tips for Becoming a Better Software Developer

2020-05-03T06:34:25.000000Z

Over the past decade or two, both as a software developer and as a manager I have accumulated a few tips for becoming a better software developer.

For context, I’ve been working as a software developer for more than a decade, mostly in small product-based companies; over the last decade or so I’ve worked as a team lead or manager in various capacities; and I’ve moonlighted as a consultant, freelancer and open source contributor in-between. Check out my LinkedIn profile and Github profile if you are interested.

Now, let’s break down the tips into three main areas: mindset, technical and people. The goal of this article is to raise awareness of the topics mentioned, if you want to understand more on a particular topic I encourage you to do your own research.

Mindset

Have a Growth Mindset

This one hopefully should be obvious, if you haven’t heard of it, please read more here.

Have a StarCraft Mindset

This one is “new”, I’ve given a talk that includes more details on what it means, check it out here. In essence, it’s a mindset that requires you to always consider things around you, and forces you to think beyond your narrowly focused task at hand.

Recognise the Dunning–Kruger Effect

I once interviewed a mid-level developer who rated himself as a 9.9 out of 10. The Dunning-Kruger effect is essentially the opposite of the Imposter Syndrome, both are very common in our field.

Agile is Not a Process, It’s a Mindset

Some people are overly obsessed with the processes of “being agile”. In truth, every organisation, every project and every team is different, there is no point to follow a set of processes that might slow you down and not add any value. Discover what being agile means to your team, your project and your organisation is an ongoing effort and should be treated as such.

Different Organisations Require Different Engineering Approaches

Some projects and organisations require tech-driven decisions be made upfront, some on the other hand require product-driven decisions instead. Knowing how your organisation operates and what it values plays a key part in understanding how software development should be approached.

A Sense of Entitlement Only Gets You So Far

Let’s be honest, as a software developer we are extremely lucky to be in a high demand field. As a result, many have started building a sense of entitlement - some ex-colleagues of mine even complained about the lack of tissue boxes in the office. Starting and running a business is extremely hard work, let’s all appreciate what we have before throwing a tantrum.

9-5 is Not Evil, Nor is Crazy Work Hours

Every now and then I see conflicts between people who share different philosophies on what work and career mean to them. We should embrace the different ways people prefer to spend their time and energy on, rather than force our own ideology on them. Discovering and knowing the difference between working at a giant corporation versus working at a small start-up is key to our happiness.

Pain and Gain, There’s Always a Trade-off

An extension of the 9-5 vs crazy work hours: adjust your expectation based on the effort you put in. You can’t have everything, prioritising family life over career building is 100% okay, as long as you know what you are giving up on, vice versa. In my 20s, many of my similarly aged friends and colleagues would go out and have fun after work and on weekends while I was in front of my computer building open source projects, is an example of a conscious trade-off I made.

Speed vs Quality

Internalising the need for speed and the need for quality should always be on our mind. Pursuing only quality or only speed will significantly limit your problem-solving capability as well as your career progression.

Think, Then Do

It’s a balance - don’t overthink it, but also don’t do without think first. When in doubt, ask more experienced people for advice and guidance.

Technical

Avoid Second System Syndrome

I’ve seen Second System Syndrome over and over again throughout my career. Some people always assume a rewrite is a much better approach than alternatives, more often than not though, it is not the case.

Micro-Services is Neither New nor the Holy Grail

There’s the age-long debate of monolith vs micro-services architecture, and SOA (Service-Oriented Architecture) has been around for decades. Each has its own pros and cons, don’t get bought into the hype because a blog post says so, or a particular product and company found it successful. Use the right tool for the right job.

TDD (Test-Driven Development) is not the Holy Grail

Just like micro-services, TDD is not the holy grail and should not be treated as much. Chasing the TDD dream without knowing its pros and cons is just as counter-productive as not doing TDD when necessary.

Code Aesthetics Matter

A lot of people treat coding as more of a scientific endeavour: they focus purely on the algorithms and results. Software development to me is a creative endeavour, and I often use white spaces and the general look and feel of the code base to determine how well-organised a project is - deeply nested code and long functions for instance are obvious ways to determine potential code smell.

You’ve all heard of the SOLID principles, but have you heard of the CRAP principles?

Read As Much Code As Possible

Don’t get me wrong, writing code is important, practice is important, but time and time again I come across developers who clearly have little exposure to established patterns and conventions.

Tech Stack Exposure and Diversity are Always a Good Thing

No matter whether you are in a highly specialised field or being a generalist, having exposure to different tech stacks and paradigms is always a good thing - it widens your field of vision and increases the boundary of your technical understanding. Don’t be the guy or gal who is known as “a \ developer”.

Have a Deeper Technical Understanding Helps

This is strictly speaking a “nice to have”, but I lost count of the number of candidates I interviewed who claimed to be a Ruby expert yet cannot explain the Ruby object model and have never heard of eigenclasses, or senior developers who cannot explain the difference between git merge and git rebase. Sure, you may not need the deep understanding in order to do your job, but it will certainly help!

People

Always Assume Incompetence over Malice

If you haven’t, read more about it here. Knowing this, and paired with having a growth mindset, you should therefore be encouraged to teach, educate and influence, rather than being defensive and participate in “us vs them”.

Truth = Visibility x Tolerance

This is another topic covered in my talk. If you find yourself regularly dissatisfied with the information you receive, ask yourself: what information is and isn’t available to you, and are you sure you have the stomach to tolerate the new information?

Rapport and Productivity are Built Over Time

Some people assume a team’s productivity is determined largely by team member’s individual capability, some also assume a 10x engineer is a reflection on one’s technical capability. It’s not. Productivity evolves over time - building rapport with those around you is often the unsung hero in productivity.

Champion Team Over Champion Individual

This one hopefully is obvious, but I just have to emphasis its importance. Please, if you struggle to work with your team, ask for help! If you have a team mate who is difficult to work with, reach out and help them!

Final Thoughts

Software development is evolving rapidly, some fundamentals however are always useful and relevant. I hope you find this article useful.

I’ve mentioned my talk a few times, but if you haven’t already, check it out below, it covers a few topics mentioned in this article, and some more. Also check out my tiny Youtube channel for more tech talks I’ve done in the past.

video: https://www.youtube.com/watch?v=MBczdO7RgNo

If you enjoyed this article, checkout my other tips articles:

]]>

Launching Focussist Landing Page - An Upcoming Agile Project Management Tool

2020-03-23T11:04:55.000000Z

Over the past few years as I gain more and more experience in not only building software and products but also in leading teams and projects, I realised that my desire of wanting an agile tool that’s super simple to use yet super flexible to adapt to different needs has grown stronger and stronger.

Given the COVID-19 situation that’s going on at the moment, I’ve finally decided to spend a few nights working on a pitch deck for myself to validate the value proposition, and some high level wireframes to visualise the ideas.

Over the weekend I’ve put together a landing page to “market” my ideas. It’s a quick job, but the underlying thinking has been on the back of my mind for years now.

Introducing Focussist, or rather Focussist’s landing page: Focussist.com.

Please check it out and give me your feedback, and don’t forget to sign up. ;)

]]>

Conference Talk: Adaptable Human @ RubyConf China 2019

2019-09-14T08:05:09.000000Z

Seven years ago in 2012 I spoke at RubyConf China 2012. It was a technical talk on how to become a better developer, if you’re interested you can check out the video recording.

Seven years later in August 2019, I headed back to Shanghai again to speak at this year’s RubyConf China. And this time around, it was a non-technical talk on how to develop one’s career. Check out the video recording below:

video: https://www.youtube.com/watch?v=MBczdO7RgNo

And, here are some photos of yours truly. :)

]]>

New Blog with New Design and Gatsby.js - Loving JavaScript Again

2019-01-20T11:27:22.000000Z

In 2010 I set up this blog on Tumblr mostly due to the ease of publishing and not having to worry about the hosting. I also went through two design iterations done in Photoshop:

You can probably tell that these designs were done before the flat and minimalist design trend we are seeing in recent years.

As I decided to revamp my blog once again, I thought I’d use Sketch this time around and aim for a simpler, cleaner and more mature design approach that’s quicker to design and to build. And this is exactly what I did:

Thanks to Sketch, the design process was much better than I would’ve had in Photoshop.

Why the Revamp?

A big part of the reason for the blog revamp was to move away from Tumblr, now that it is part of the dying Yahoo!, and to invest in modern technologies to make the blog function better by adding responsive design and a proper grid system, etc.

As I was reading up on and researching technologies as I often do, I came across Gatsby, and I was immediately convinced that this was going to power my new blog as it appears super fast (by building static content and pre-fetching), is based on React and has a huge collection of useful plugins.

At work I often joke about how I hate JavaScript (rm -rf node_modules anyone?), the truth is I started off my career doing a lot of JavaScript, my most popular open source library by GitHub star count is in JavaScript, and I have always worked as a full stack dev until recent years.

Due to the shift of my responsibilities more into leadership over the past few years, I have opted to focus more on the backend, but deep down I love software in general and given the frontend scene has seen some major improvements in the past few years, not to mention that at work we have several React code bases, it’s time I dig my teeth into it again.

Disclaimer: until recently I had only done two React projects before and my exposure to ES6 has been somewhat limited.

Gatsby

The thing about Gatsby that impressed me the most is how fast it is due to its out-of-box support for pre-fetching. With a wealth of different starter templates it is also very easy to hit the ground running. I used gatsby-starter-netlify-cms as it supports Netlify and is fairly simple.

Export Tumblr Posts

First thing first, I will need to export all my tumblr posts into Markdown files so that Gatsby can consume them. There are heaps of such tools including Tumblr’s own export functionality. I ended up having to use these two:

Jekyll importer for converting all Tumblr posts into Markdown
tumblr-utils for downloading all images in the posts (Tumblr’s own export tool and many others only download images hosted on Tumblr)

Of course, many search-and-replace as well as manual massaging were required to get the new Markdown files to a useable state.

Add More Gatsby Plugins

The starter template was great, but I needed more functionalities so I ended up adding these over time:

gatsby-plugin-styled-components for supporting Styled Components
gatsby-remark-embed-video for embedding videos
gatsby-remark-prismjs for syntax highlighting
gatsby-awesome-pagination for pagination
gatsby-remark-reading-time for adding reading time
gatsby-plugin-feed for RSS feed
gatsby-plugin-google-analytics for Google Analytics
gatsby-redirect-from for page redirects (of old Tumblr pages)
gatsby-plugin-sitemap for sitemaps
gatsby-plugin-robots-txt for robots.txt

Using many of these plugins only requires you to add the plugin itself and some config, then off you go - it could not be any easier!

GraphQL

One other reason why I was interested in Gatsby was because of its out-of-box support for GraphQL. I came across GraphQL a while back but never had the chance to work with it. At work one of our projects started using GraphQL and the team behind it rates it highly, so I really wanted to get some first hand experience using it.

With the inclusion of GraphiQL the in-browser GraphQL IDE, the GraphQL experience in Gatsby is extremely simple and satisfying.

Querying Data using Gatsby’s Page Components and StaticQuery

It’s been a bit of journey.

The starter template I used has GraphQL queries in the page components, but I wanted the GraphQL queries in shareable components so I opted to using Gatsby’s StaticQuery.

Until I decided to add gatsby-awesome-pagination which requires the GraphQL queries to have variables therefore incompatible with StaticQuery.

Despite the back and forth between page components and StaticQuery, I still loved the GraphQL experience in Gatsby.

It’s All Just React (In a Good Way)

The great thing about Gatsby is that despite making things easier by including a bunch of useful functionalities and plugins, down to its core, everything is based on React Components.

What this means is that as long as you have a working knowledge of React and React Components, extending Gatsby’s capability is simple and straightforward.

CSS Powered by Bulma

The starter template also includes Bulma - a modern and popular CSS framework.

Until now I’ve always relied mostly on Bootstrap so it’s nice to get to experience another CSS framework. And it turns out, Bulma is quite nice and easy to use too.

Thanks to Bulma, supporting responsive design was a breeze.

Gotcha: CSS Rules and “gatsby-plugin-purgecss”

gatsby-plugin-purgecss is included as part of the starter template to reduce the size of the CSS files by removing unused CSS rules. It’s wonderful as it saves about 200Kb of CSS (before minifying and gzipping) from unused rules from Bulma.

One gotcha I soon realised though, is that it also purges any CSS rules that are only rendered in the Markdown components. My guess is that the purging happens before the static contents were rendered therefore it cannot detect them.

The fix was quite simple, given that by default it only purges the main css file (in my case, all.scss), and all the specific rules in this file were defined by me, all I had to do was to add the comment lines to tell the plugin not to purge these rules:

@import "~bulma";

/* purgecss start ignore */
.my-own-css-rules-here {
}
/* purgecss end ignore */

Gotcha: CSS Conflicts Between Bulma and Prism.js

One more issue with CSS was the conflicts between Bulma and Prism.js due to the way some of the CSS rules are defined in Bulma.

What I had to do was to “reset” those CSS rules so they can inherit from Prism.js instead:

.content {
  .number,
  .tag {
    align-items: inherit;
    background-color: inherit;
    border-radius: inherit;
    color: inherit;
    display: inherit;
    font-size: inherit;
    height: inherit;
    justify-content: inherit;
    line-height: inherit;
    margin-right: inherit;
    min-width: inherit;
    padding: inherit;
    text-align: inherit;
    vertical-align: inherit;
    white-space: inherit;
  }
}

Netlify: Deployment and Hosting Made Easy (and Free!)

Before Netlify and similar services were available, many people used GitHub Pages to host their static content sites. It worked reasonably well but in the end this isn’t GitHub’s core competency and the features are limited.

Netlify takes deployment of static content sites to a whole new level and I cannot be happier to have finally jumped on the bandwagon too.

Closing

Suffice to say, I am very satisfied with this revamp experience - Sketch, Gatsby, GraphQL, Netlify and many other software have made it a walk in the park.

It only took me around 50 hours to get to 99%, and a few hours more to add in things like Disqus and social sharing.

Oh, and for those who are curious, the source code of this blog is available on GitHub here.

Let me just say:

The last time I fell in love with JavaScript was when JQuery first came to the scene many moons ago. Ever since then I dread working on the JS stack mostly because I’ve been spoiled by the Ruby and Elixir ecosystems. It’s nice to fall in love JavaScript again after so many years of simply “getting by”.

I hope you like my new blog, and if you have a similar experience to share, I’d love to hear from you!

]]>

Coding and Learning Should Never Stop, Open Sourcing is Caring

2017-08-27T11:21:01.000000Z

I’ve had a productive coding weekend, and so I decided to share my experience. Now, many developers choose to treat their career as a series of 9-5 jobs, but if you’re reading this, I assume you’re like the rest of us who love continuous learning and self improvement.

Preface

About a year ago I started learning Elixir. So as part of the learning experience, I wrote two matching learning related libraries: Stemmer and Simple Bayes. It was a great, really enjoyable experience and I learnt a lot about the concepts of word stemming, naive Bayes classification and of course, functional programming.

These topics have been interesting to learn about, but until one gets to use them on a daily basis for a while, key concepts are less likely to convert from short term memory to long term memory. Given my day job is not about writing Elixir (yet), I needed to find other ways to keep my skill-level up and to continue exploring new things.

So about a month ago, I picked up a project I started a year ago but gave up shortly after: Crawler. At the time GenStage was just announced and I was interested in incorporating it into my project as I thought it’d be a great fit. But due to varies reasons - mostly not having a firm grasp of the GenStage concept and implementation, as well as taking on a CTO role at a startup, I couldn’t find enough time and patience to make it work so I had to let it go.

Until now.

Crawler, on Steroids

I’d realised that the way I was trying to incorporate GenStage into my project was never going to work. Not because of GenStage itself, but because of the way I approached learning it. At the time I was so eager to make use of GenStage, and coming off the back of my good streak of releasing the aforementioned machine learning libraries, I thought I could take shortcuts and things would all work out perfectly.

No.

So I licked the wounds, learnt my mistakes and changed tactics. This time, I scoped out and encapsulated my learnings (just as one would in designing a software system), and eventually I came up with another library - OPQ: One Pooled Queue.

OPQ: One Pooled Queue

I knew Crawler could technically work without any queueing system, or GenStage. So I kept on building Crawler, until I felt productive in writing Elixir again, and touched enough areas and concepts that I knew exactly what I needed from GenStage.

And luckily for me too, GenStage over the past year has matured and more importantly, has better documentation with more code examples. Upon closer investigation of the code examples, I found that their GenEvent and RateLimiter examples were almost exactly what I needed. It was an epiphany moment for me after reading and understanding these examples, all of a sudden I “get it”.

If you take a look at the source code of OPQ you’ll notice that the heavy lifting logic was mostly inspired (or even copy-pasted) from those examples.

Open Sourcing is Contributing and Caring

Up until this point, as far as writing open source Elixir code goes, it had mostly been me writing my own code for my own projects. But open sourcing is much more than just writing one’s own code and publishing them on Github.

If you’ve followed my work you’ll know that I’m a big fan of contributing to other projects, some of which are well-known ones like Rails and Slim.

It just so happens, that over the weekend I ran into situations where I needed to contribute to other Elixir projects - and spoiler alert: one of which is Elixir itself.

ElixirRetry

One of the features I set out to add to Crawler, was to allow failed crawls to retry before giving up. Naturally, the first thing I did was to try find an existing package to support the retry functionality.

After some googling and digging into some source code, I’ve found and settled on ElixirRetry - a neat library that was cleverly built.

Soon I found out that since its last release, it offered both retry/2 and retry/3, the latter of which was to support an extra argument that specifies which exceptions to allow as part of the retry flow. It was a great addition, but it doesn’t effect me as Crawler doesn’t need it.

As a developer who cares deeply about code quality and clarity, I immediately thought about how the interfaces for retry/2 and retry/3 could be improved, by simply combining them and making the extra argument in retry/3 an option.

So I did, and issued a pull request here: https://github.com/safwank/ElixirRetry/pull/12 You’ll notice that I’ve also made some improvements to the test suite while I was at it. ;)

Elixir Typespec

One issue I ran into while building Crawler, was the static analysis via Dialyzer - some code that actually ran and passed tests functionally, failed the type checking.

As someone who isn’t as experienced in Elixir, I first opened an issue and phrased it more as a question seeking validity of the issue. I then jumped on Elixir’s Slack group and asked people in the group about this issue.

Fortunately, Ben Wilson immediately came to aid by verifying my issue and validating my suspicion of it being an issue in Elixir’s typespec documentation.

And so, a pull request was created and approved shortly after.

Sharing is Caring

The bulk of Crawler as well as the entirety of OPQ were built in the past month or so. I hope some people will benefit from having these libraries around. And, I hope people also enjoyed me sharing my experience, and perhaps be inspired to start sharing more too.

I will leave you all with a Chinese saying: 滴水石穿. The literal translation is “dripping water penetrates the stone”, and what it means is “constant perseverance yields success“.

]]>

Elixir and Doctest - Help Writing Better Programs, One Function At A Time

2017-08-07T10:00:20.000000Z

Preface

If memory serves right, it’s been several years since I first dabbled in Elixir, but it was about a year ago I really started putting some serious effort into learning Elixir, and as a result I made two libraries in the machine learning space: Simple Bayes, a Naive Bayes text classifier implementation, and Stemmer, an English (Porter2) stemming implementation.

Unfortunately after I’ve released those two libraries, I hadn’t had much opportunities to work with Elixir. My day jobs have been mostly Ruby, JavaScript, PHP and a dash of Golang. And so, after being silent for a year, I’ve decided to pick up something I had started a year ago - a web Crawler. If you are new to Elixir, feel free to follow this project as I am actively developing it.

Learnings

The preface is to give a bit of background of when and how I started learning Elixir, now, let me talk about one of my favourite features of Elixir, and how it helps me write better code not just in Elixir, but in virtually any other language.

Introducing the topic of today, a really simple feature, and in fact it has been part of Python for years - the doctest.

Doctest

In short, a doctest is pieces of code examples that run as part of the test suite, and show up as part of the documentation. For example:

defmodule Greeting do
  @doc """
  ## Examples

      iex> hello("world")
      "hello world"

      iex> hello("dear")
      "hello dear"
  """
  def hello(input) do
    "hello #{input}"
  end
end

And then all you need is in your corresponding test file, to enable doctest:

defmodule GreetingTest do
  use ExUnit.Case

  doctest Greeting
end

Even though I knew about Python’s doctest for years, I’d never realised how impactful it can be given its simplistic nature, probably due to the fact that I’ve never used Python for anything serious.

Until Elixir.

There are three things I find the most impactful as I write more doctests: clarity, scope, and design.

Clarity

As a ruby programmer, I appreciate greatly the beauty of not just the main code base, but also its test suite. Hence I’ve always preferred to use RSpec in larger code bases. However, as the application gets more complex, and as the number of test files grows, the cognitive overhead of reading and processing all the files and lines becomes higher and higher.

Doctest solves this perfectly - no longer do we have to crawl through the right file and the right line for a particular test case, all the test cases are neatly presented right in front of you as you read the function itself.

You might think this as trivial, but just like many organisations spend time and effort to optimise for effective communication by studying proxemics, proxemics between different components of a software code base also plays a role in improving the code clarity, and ultimately the code quality.

Scope

Doctest is purposely simple, and is designed for unit tests. There have been many times when I found myself realising my function was too dependant on external states, or are doing too many things because it was hard to write simple doctests. In a way, the constraints of doctests have forced me to rethink the scope of my function, and that would often lead to an overall better designed system.

Design

As much as I’d like to think about the SOLID principles all the time, it is often too easy to dig deep wholes in the midst of building things.

Every now and then I find myself extracting a piece of logic to a private function and calling it a day. In Elixir, only public functions can have doctests - again, this constraint pushes you to think about the importance and the role of a particular function, perhaps it is better to be moved to another module as a public function therefore can have its own doctests. Here is an example when I did some refactorings on Crawler.

Closing

I wanted to write this article for a while now - as I truly love and appreciate Elixir’s asthetics and features. Many developers might find functional programming as a barrier, but I can assure you that with Elixir’s tooling and ecosystem, and of course doctest (wink), building software feels like a breeze.

Last time when I was this happy building software was when I first discovered ruby. If you haven’t given Elixir a try yet, I encourage you to do so sooner rather than later, it will not only give you a functional programming perspective, but will also help you write better code in other languages.

]]>

I Accidentally Some Machine Learning - My Story of A Month of Learning Elixir

2016-07-23T18:33:29.000000Z

About a month ago I was in-between jobs - I had two weeks to rest up, recharge and get ready for my new job. So I thought, I should use those two weeks to learn something new.

Years ago I briefly looked into Elixir when it was first released to the wild, at the time I wasn’t interested in picking it up due to its syntax similarity to Ruby, despite their vastly different underlying semantics. I love Ruby, and it’s been my weapon of choice for the past 6-7 years, so when it came time for me to learn something new, I naturally wanted to learn something a bit more different than Ruby, syntax-wise.

Fast-forward a few years, I am more mature and open-minded, and are now in a position to welcome Elixir and to embrace the Ruby-like syntax as well as the functional programming mindset with open arms.

Learning Elixir

Given the strong influence of Ruby in Elixir, I can’t help but get nostalgic about learning it by reading another great book by Dave Thomas: Programming Elixir.

Dave Thomas’ original Programming Ruby was instrumental in the success of the Ruby programming language and its communities in the west, and it certainly has helped me to greatly widen my exposure to not only the wonderful world of Ruby but object-oriented programming in general as a PHP developer.

Learning Elixir has been really fun, not only does it share the same developer-friendliness championed by Ruby, it also does so with remarkable strength in concurrency thanks to BEAM (Erlang’s VM).

Many people are drawn to Elixir due to its Ruby influence, its functional and immutability nature, and its Actor-based concurrency model. I would however like to call out one of my favourite features of Elixir that sometimes gets overlooked, and that is how you could write unit tests using ExUnit.DocTest.

Elixir Makes Writing Unit Tests Effortless

Python programmers have been writing doctests for years, but Ruby due to not having an equivalent standard library, has never had writing doctests taken off in its community. I’m glad Elixir has.

Take a look at the code snippet below:

defmodule Stemmer.Step0 do
  @doc """
  ## Examples

      iex> Stemmer.Step0.trim_apostrophes("'ok")
      "ok"

      iex> Stemmer.Step0.trim_apostrophes("o'k")
      "o'k"

      iex> Stemmer.Step0.trim_apostrophes("'o'k'")
      "o'k"
  """
  def trim_apostrophes(word) do
    word
    |> String.replace_prefix("'", "")
    |> String.replace_suffix("'", "")
  end
end

The three test cases given will actually be tested by ExUnit, provided you ask it to do so as below:

defmodule Stemmer.Step0Test do
  use ExUnit.Case, async: true

  doctest Stemmer.Step0
end

You may think having doctests is not a big deal, what I can say is that I am really enjoying the fact that unit tests can be written with minimal friction and high visibility (as they sit with the implementation, rather than within a big test suite with a sea of files). Besides, not all of us practice TDD religiously or 100% of the time, and when I don’t or can’t do TDD, having doctests ensures I don’t forget adding test cases. :)

Toy Robot in Elixir

The best way to learn something new is to practice it as you learn it. Instead of diving straight into a complicated system, or to write a hello world program, I thought I would dig out a code test and re-implement it in Elixir.

The code test is the rather infamous Toy Robot test. As an interviewer, I must have reviewed a few hundreds of these tests, most of which in Ruby. I know what a good OO solution looks like, so naturally I was looking forward to “rethink” the problem and have a crack at it using Elixir.

Here is the result: https://github.com/fredwu/toy-robot-elixir

As a learner, I wanted to practice as many language features as possible, so included in the Toy Robot test code I tried:

Different data structures such as Struct and List
Pattern matching
Data piping (|>)
Agent for managing states
Macros for validation rules

I kept the test code small and nimble on purpose to illustrate the readability of a highly expressive language. I may not have exercised all the language features (such as supervisors and protocols), but at that point I felt I could start the Elixir journey with a big smile on my face already.

Learning Phoenix

I have to admit, a big part of the reason that got me started on learning Elixir is to build a side project. Initially I wanted to simply use Ruby and Rails because I can be very productive using them, but in the end I decided on learning Elixir and Phoenix, because most side projects fail, and when they do, how much learning can you extract out of those experiences? My answer to this question, is to learn a new language, a new framework and most importantly a new programming paradigm to drastically increase the amount of learning I could gain.

But please, before you jump on Phoenix, learn Elixir properly first! Over the years I’ve seen far too many cases of people jumping on Rails without understanding the language features of Ruby…

To learn Phoenix, the Programming Phoenix book is pretty much the bible on this subject aside from the official guide, and it is written by Chris McCord, the author of Phoenix, Bruce Tate, and Jose Valim, the author of Elixir.

As an experienced Rails developer, it only took me a day or two to go through the book (I largely skimmed the chapters on Channels as I don’t intend to build a real-time app).

Now, as a Ruby developer, the biggest pain point when working on Rails or other sizeable Ruby MVC projects for me, is ActiveRecord. I got so frustrated to the status quo I even attempted to fix it.

Fortunately, in Elixir and Phoenix we have Ecto. One of my favourite features of Ecto is the concept of Changeset.

Take a look at the code snippet below:

defmodule MySecretApp.User do
  use MySecretApp.Web, :model

  schema "users" do
    field :username, :string
    field :email, :string
    field :password, :string, virtual: true
    field :encrypted_password, :string

    has_one :profile, MySecretApp.Profile

    timestamps()
  end

  def changeset(struct, params \\ %{}) do
    struct
    |> cast(params, [:username, :email])
    |> validate_required([:username, :email])
    |> validate_length(:username, min: 3, max: 20)
    |> validate_format(:username, ~r/\A[a-zA-Z0-9_]+\z/, message: "alphanumeric and underscores only")
    |> validate_format(:email, ~r/@/)
    |> unique_constraint(:username)
    |> unique_constraint(:email)
  end

  def creation_changeset(struct, params \\ %{}) do
    struct
    |> changeset(params)
    |> password_changeset(params)
  end


  defp password_changeset(struct, params) do
    struct
    |> cast(params, [:password])
    |> validate_required([:password])
    |> validate_length(:password, min: 8, max: 200)
    |> encrypt_password
  end

  defp encrypt_password(changeset) do
    case changeset do
      %Ecto.Changeset{valid?: true, changes: %{password: password}} ->
        put_change(changeset, :encrypted_password, Comeonin.Bcrypt.hashpwsalt(password))
      _ ->
        changeset
    end
  end
end

The changset/2 function can be used when a user needs to be updated, whereas the creation_changeset/2 is to be used only when a user is first created. Sure, you can achieve similar result in Rails by using custom validators, but the fact that this practice is enforced by the library and the framework, is encouraging.

One other thing that I’ve seen in larger Rails apps, is the leaky abstraction of view-level logic, they typically sit in controllers, helpers (which are globally available) or worse, models. Phoenix in this case follows what Hanami, Trailblazer and many other frameworks do: it introduces a “view model” layer.

Something along the lines of:

defmodule MySecretApp.UserView do
  use MySecretApp.Web, :view

  def full_name do
    "#{title} #{first_name} #{last_name}"
  end
end

In a nutshell, Phoenix is just like Rails, but less magical (in a good way) and faster. :) Like the Elixir eco-system, most libraries and concepts feel like their ruby/rails counterparts, but more refined.

There are a few popular Hex packages if you’re familiar with their ruby counterparts:

	Elixir	Ruby
Code analysis	credo	rubocop
Testing	espec	rspec
Browser testing	wallaby	capybara
Test coverage	excoveralls	simplecov
Test factory	ex_machina	factory_girl
Authentication	guardian	devise

And the list goes on and on… Be sure to check out Awesome Elixir for more community curated Elixir libraries.

Learning Machine Learning

As I started building the foundation of my side project, you know, the usual user management and session management, etc, etc, by chance I came across mentions of Bayesian inference.

One thing led to another, very soon I started looking into varies different machine learning algorithms such as Naive Bayes and Random Forest. These are all subjects I had never come across or even heard of before, as I have no strong mathematics, statistics or computer science background. I was however, intrigued and inspired nonetheless, and wanted to employ some machine learning in my side project.

And the best way to understand and learn about machine learning algorithms? You guessed, is to write one! After some research, I came to the conclusion that Naive Bayes is one of the simplest to implement, is great for text classification which is useful for my side project, and has good accuracy given its simplicity and fast speed.

Introducing Simple Bayes

So, after a few train rides to and from work, on my newly purchased Macbook (yes, the one with only a single USB-C port), I built a Naive Bayes library: Simple Bayes.

As of writing, this is the only Naive Bayes library in Elixir that supports the following features:

Naive Bayes algorithm with different models
- Multinomial
- Binarized (boolean) multinomial
- Bernoulli
No external dependencies
Ignores stop words
Additive smoothing
TF-IDF
Optional keywords weighting
Optional word stemming via Stemmer

Let’s see it in action shall we? :)

SimpleBayes.init
|> SimpleBayes.train(:ruby, "I enjoyed using Rails and ActiveRecord for the most part.")
|> SimpleBayes.train(:ruby, "The Ruby community is awesome.")
|> SimpleBayes.train(:ruby, "There is a new framework called Hanami that's promising.")
|> SimpleBayes.train(:ruby, "Please learn Ruby before you learn Rails.")
|> SimpleBayes.train(:ruby, "We use Rails at work.")
|> SimpleBayes.train(:elixir, "It has Phoenix which is a Rails-like framework.")
|> SimpleBayes.train(:elixir, "Its author is a Rails core member, Jose Valim.")
|> SimpleBayes.train(:elixir, "Phoenix and Rails are on many levels, comparable.")
|> SimpleBayes.train(:elixir, "Phoenix has great performance.")
|> SimpleBayes.train(:elixir, "I love Elixir.")
|> SimpleBayes.train(:php, "I haven't written any PHP in years.")
|> SimpleBayes.train(:php, "The PHP framework Laravel is inspired by Rails.")
|> SimpleBayes.classify("I wrote some Rails code at work today.")
# => [
# ruby: 0.20761437345986136,
# elixir: 0.08101868169313056,
# php: 0.019047884912605735
# ]

As you can see, provided with reasonable training data, Naive Bayes can work extremely well.

Introducing Stemmer

Something I discovered as I was building Simple Bayes, is something called stemming. Let’s see another example:

SimpleBayes.init(stem: false)
|> SimpleBayes.train(:apple, "buying apple")
|> SimpleBayes.train(:banana, "buy banana")
|> SimpleBayes.classify("buy apple")
# => [
# banana: 0.057143654737817205,
# apple: 0.057143654737817205
# ]

Oops, the probabilities of the sentence “buy apple” of being apple and banana are the same, that’s not good, as we know “buying” and “buy” should mean the same thing. This is where stemming comes in handy. Let’s enable stemming and run the example again:

SimpleBayes.init(stem: true)
|> SimpleBayes.train(:apple, "buying apple")
|> SimpleBayes.train(:banana, "buy banana")
|> SimpleBayes.classify("buy apple")
# => [
# apple: 0.18096114003107086,
# banana: 0.1505149978319906
# ]

Stemming in this case correctly identified “buy” and “buying” as the same thing (they both have the stemmed root of “buy”).

How did I include stemming in Simple Bayes you wonder? Let me introduce you to Stemmer - as of writing this is the only Porter2 stemmer available in Elixir. :)

Elixir and BEAM may not be known for their raw performance, but compared to a similar Porter2 stemmer implemented in pure ruby, my Elixir version runs about five times faster (under 5s to stem 29000+ words, compared to 25s with the ruby version).

The Learning Continues…

I’ve only been writing Elixir for a month, and there are still heaps to learn and to practice. So far though, I have to say my experience with both Elixir and Phoenix has been fantastic - I get the same satisfying and pumped feeling I got when I was learning Ruby and Rails years ago.

If you haven’t checked out Elixir, I strongly encourage you to do so, after all, the world seems to be moving in the concurrency direction at an increasingly rapid speed, including Ruby 3x3. :)

]]>

Developers, Being Treated Poorly? You Are Not Alone!

2016-04-05T11:40:45.000000Z

UPDATE: Reading Hacker News comments it is apparent to me that I have not explained my position and intention very well. Please allow me to clarify - I do not resent the people involved in these moments, if anything, I thought those encounters were good learning experience for me, as I know there are things I would do and say differently so others around me won’t feel I’ve been too harsh on them. I did not share these moments to complain, but to raise the awareness that kinder communication styles might be better received.

As a software professional who’s been in the industry long enough to have accumulated quite a few battle scars, I tend not to think back about the unpleasant, unfair and sometimes bizarre moments that make me feel depressed or angry. If there is one thing I learnt over the years, it’s to let go and move on.

Despite the effort in staying positive, I still believe it is valuable to share these moments. If you work in software and feel like you are being treated poorly, you are definitely not alone. By openly sharing these moments, I am hoping more people will start taking notice, and perhaps will lend a hand or a shoulder when a colleague is in need of support.

So, I’d like to share a few moments that have happened in my career, in no particular order. There are a few of these moments that I still find hard to swallow, but most of them I simply chuckle whenever I think about them. ;)

One morning I came to work, noticed a few server alerts that required attention. This was pre AWS and the whole Devops automation movement days so I needed to log onto the server. For some reason my access was revoked so I had to log into the hosting dashboard to reset the root password. The senior dev on the team turned up, and when asked about my access, casually told me: “oh you did something on the servers the other day, so I removed your access.”

CTO asked me to help building a new mission critical PHP and MySQL service, upon discovering an existing service already built on Ruby and MongoDB by three very senior developers and consultants, CTO said the reason for the rebuild was because “ActiveRecord is too slow.” Let me remind you that the initial Ruby solution was built using MongoDB. At the time the company had about 30-40 experienced Ruby developers, and only 2 junior to mid level PHP developers who were specifically hired to build the PHP/MySQL service which turned out to be a disaster.

As the team lead who was responsible for evaluating performance and setting salary, I was told by one of my team members that they had gotten a pay rise. Surprised, I asked one of the founders of the company who is non-technical what was going on, during the conversation I discovered another pay rise was given to one other member of my team - all without consulting or even informing me.

Excited to have had my overseas conference talk proposal accepted, I asked the new head of our department who had joined our company for only a month or two for travel approval. Fully expected a pad on the back, I was surprised to have been told that I cannot go, with the reason being “it is just not a good time”.

One of our servers needed to be rebuilt, so trying to be a team player and help, I started installing some basic packages. The senior dev on the team turned to me, straight faced and enunciated in a deep and cold voice: “don’t touch anything on it, this is my server!”

The new development manager walked into our meeting, pulled me out, guided me to another meeting room where the new CTO was waiting. The new CTO opened with “so it’s time for us to go separate ways”, and implied the reason being something that I could definitely have pushed for unfair dismissal. Trying to be professional, I walked back into my original meeting, but the new development manager quickly pulled me out and said “you need to go right now”. He then stood behind me, saw me formatted my work laptop, then literally escorted me out of the office. The new development manager and CTO soon drove the company to the ground and left the country.

After submitting the salary increases for the all my direct reports, I was delighted that the CEO was happy with them all. Given I have had no pay increases in over one and half years I asked for a modest pay increase for myself. I was told “no, we have to discuss this after your project delivery.” My direct reports and I were in the same delivery team.

It was near the end of the working day, around 5pm, as the person coordinating the developer recruitment in our area I ping’ed our Slack channel with a lighthearted message to encourage our developers to start reviewing some code tests from job candidates if they’re free. One person replied: “My end of day activity is doing the stuff I should have been doing all day instead of the other things that came up.”

One day I got really really sick - difficult to walk or even stand still, dizzy and having fever. Upon informing one of the founders, I was told to come into the office, even though there was no urgent tasks need to be done. Reluctantly obliged, I came into the office, went out for lunch with colleagues but I was so sick I could not eat anything and can barely sit straight. I then went home, and was sick for the rest of the week…

My manager, who was the general manager, wanted to fire two of my developers. His tactic for firing the senior developer was to make the senior developer role redundant and offering the dev a junior developer role instead.

I was tasked to investigate setting up a Cassandra database cluster. Upon discovering certain network limitations, I approached the then head of network operations, when asked about the inability to access the Cassandra cluster, he replied confidently that “it works”. Later on I discovered that he successfully telnet’ed to the ports and declared it “working”.

The head of HR who is female and a big advocate for workplace diversity walked into my exit interview. The opening line was “Ordinarily I only do exit interviews with female employees…”

New to the big corporation environment, after asking around a few people and could not figure out how to provision an iPad for my team (but knowing some other teams do have company iPads), in an email reply by the head of enterprise IT: “Hey Gents, Fred has escalated this through multiple channels today. Awesome effort on his behalf, to seems like he didn’t like our response.” To which a colleague had to gently remind him that it wasn’t my fault there was no documented process for provisioning hardware.

After having some discussions on our MySQL database, the dev who had recently joined my team all of a sudden raised his voice in an extremely arrogant and dismissive tone: “you clearly don’t understand this feature, do you?”

A senior software architect who I used to respect walked close by in a meetup. We worked for the same company a while back so I smiled, said hi and was about to start a conversation, he quickly cut me off with “I need to get a drink” without looking at me and wondered off.

The general manager who is non-technical, asked me to investigate options to uplift our ageing bespoke ecommerce solution. Upon delivering my findings, I was told that “your findings are biased.” Later it was clear to me that he wanted confirmation of his agenda from the technical perspective and his mind had already been made on the approach we should take.

A new big shot executive joined the company as our new CTO. He had no agile background and our company was transforming and pushing for a lot of agile principles at the time. Two weeks into his new appointment, the new CTO published an internal document titled “Controlled Chaos”. After reading the document everyone immediately realised that he was describing waterfall. The document was shared as a Google Doc and was open for comments, so people started asking hard questions. Weeks later, many of us who were vocal about his document were let go.

There you are, these were my moments. What were yours? Care to share? :)

]]>

History Text Analysis Over Spreadsheets - A Poker Player and Developer's Road to Agile Project Management

2015-03-09T21:50:23.000000Z

Ever since I started transitioning into a team leadership role over three years ago, I had been trying to find ways to eliminate waste caused by repetitive work and to keep myself on the fringe of pushing the technical boundaries.

Four months ago I started my current role where my official job title is Delivery Lead. People don’t often know what a delivery lead is, but in my mind it is a role to ensure the success of the project delivery by identifying and closing the gaps in the team and in the organisation. And in order to do that, one of our responsibilities is to measure, understand and improve our team’s agile process.

It is very tempting to rely on the wonderful and powerful Excel formulas to help record and analyse data points and generate metrics such as cycle time. However, punching things into a spreadsheet is tedious, error-prone, time consuming and violates the DRY principle.

The spreadsheet I used to use for tracking cards.

Introducing Amaze Hands

As someone who strives to keep writing code even in a non-technical role, I started building a tool called Amaze Hands to help reduce the amount of waste I accumulate as a delivery lead.

Amaze Hands’ simple Web UI.

Analyse Cards Like Poker Hands

I used to play a bit of online poker and one thing you do in online poker is to look at your hand history to understand the game and your opponents.

If you think about it, agile boards are just like poker games - there is history to what has happened in the past, and in order to optimise for future gains, we need to understand what went wrong and what to improve on a case-by-case basis.

One of the teams I am apart of uses LeanKit, whilst it is a good tool, its reporting functionalities are very limited and its XML export function is completely broken. As a result I started building Amaze Hands to parse the copy-pasted card history from LeanKit, and to eventually generate the metrics I care about.

The LeanKit strategy which consists of a parser and a transformer is able to parse the copy-pasted text from a card as shown below:

And below is the high level architecture of Amaze Hands:

    +---------------------+
    |        Text         | <- Raw text input.
    +----------+----------+
               |
+--------------v--------------+
|         Strategies          |
+-----------------------------+
|   +---------------------+   |
|   |       Parser        |   | <- Parses text into an AST.
|   +----------+----------+   |
|              |              |
|   +----------v----------+   |
|   |     Transformer     |   | <- Transforms the AST into a common AST.
|   +---------------------+   |
+--------------+--------------+
               |
    +----------v----------+
    |       Builder       | <- Builds the dataset from the common AST.
    +----------+----------+
               |
    +----------v----------+
    |       Reducer       | <- Filters the dataset.
    +----------+----------+
               |
    +----------v----------+
    |      Analyser       | <- Analyses the dataset for metrics.
    +----------+----------+
               |
    +----------v----------+
    |      Producer       | <- Produces metrics.
    +----------+----------+
               |
    +----------v----------+
    |      Presenter      | <- Presents metrics.
    +---------------------+

Incremental Analysis - Zero In On Metrics That Matter

No projects are equal, and no project teams are equal - the goal of Amaze Hands is to incrementally add intelligence to our agile process that matters to a particular project and its delivery team.

By incrementally adding and/or filtering data points for analysis, we will be able to zero in on the problematic areas of our agile process. The following is a list of potential areas we could perform analysis on:

cycle time
wait time
blocked time
knocked-back time
context switch (between different streams of work)
other factors such as meetings, attrition, etc

As of the time of writing, Amaze Hands supports the following common metrics:

cycle time (mean and median)
cycle time rolling average (mean and median)
wait time (mean and median)
wait time rolling average (mean and median)
standard deviation rolling average
cycle time scatter

It’s Just the Beginning

Amaze Hands started off as an REA Hackday project - on the technical level (hey I still see myself as a developer!), the tool was built in a way that it isn’t over-engineered (a.k.a. slow to get it out the door and validate its usefulness) but at the same time has multiple layers as shown in the architecture diagram above so I could refactor and optimise each layer independently when necessary.

It is still early stage, but I thought I’d share what I have right now to gather some feedback and perhaps inspire fellow project leaders to look into optimising your own workflow.

When I started Amaze Hands I was only leading one project team that uses LeanKit, but since last week I started leading another team that uses a physical wall - I can’t wait to adapt Amaze Hands to support the new input stream.

So, do you have any interesting tools or techniques to help you lead projects? If so, I would love to hear about them!

]]>

Leading Snowflakes by Oren Ellenbogen - A Pragmatic Book on Software Team Leadership I Wish I Read Sooner

2014-07-26T15:49:00.000000Z

Two weeks ago as I was reading Software Lead Weekly which I had subscribed to for a while, I discovered its curator, Oren Ellenbogen’s book - Leading Snowflakes.

It was a moment of discovery that lead to a stream of delightfulness.

It Started with a Long Day at Work…

After a long day at work, I was so beat I couldn’t even listen to audio books like I always do on my way to and from work. So I drove home that night in total silence. One thing that was on my mind at the time was - who should I reach out to for some advice and mentorship?

“Probably no one.” I shrugged, and decided to simply suck it up and sleep it off, hoping solutions to the problems I had would come to the light of the day.

I got home, opened up my laptop and started reading unread emails. “Software Lead Weekly, ah I haven’t read the latest issue yet,” I thought to myself, “I wonder who the curator is.”

Discovered “Leading Snowflakes”

And that had lead me to discover Oren Ellenbogen’s book Leading Snowflakes.

On the book’s landing page it says:

This isn’t another 350-page book you’ll agonize yourself over not reading. It’s short, concise and pragmatic. You’re busy, I get that.

“This is great!” I thought, with a sense of uncontrollable excitement. After all, these days I found myself hard to justify spending hours and hours on books that talk about subjects in a hand-wavy manner. A short, concise and pragmatic book on leadership was exactly what I needed.

There was a free sample chapter for download, so I clicked on it without hesitation. To my surprise, after downloading the sample chapter, Oren reached out to me with a personal email, and that had quickly lead to a chain of emails of me writing about the things at work that I found challenging, and Oren providing sound advice and suggestions.

I felt not only grateful, but lucky. Over the years I had worked at varies places where I was fortunate enough to have met quite a few people who I consider my mentor. But to run into one who I had never met before, on the same night my spirit was crushed, and who were willing to not only listen to my rant but also to walk me through my problems and offer advice - it was like finding a needle in a haystack.

My decision to purchase the book was made half way through our email exchanges, as I was browsing through the links Oren provided on his book landing page (under the section of “Sharing my lessons learned”), I found them to be very insightful.

Convinced that Oren’s book would offer even more value, I quickly bought the complete package with the book and all 10 interviews.

Leading Snowflakes - A Book Every Software Team Manager Should Read

Fast forward to today, I have finally finished reading the book, and I could confidently say that this is a book every software team manager should read.

There is no hand-wavy theories or lectures, what the book offers is exactly what is said on the landing page - short, concise and pragmatic.

The book contains a lot of sound advice, practical tools and techniques as well as a ton of useful links. Reading through it, I can recall many moments where I thought to myself, “hey, that indeed is a very cool technique, I can totally see how I could use it and be better at leading.”

I am now ready to implement those techniques and I am hopeful that I would become a better team leader and my team would become a better team as a result. :)

If you are a software team manager, or aspire to be one, I highly recommend this book!

To purchase only the ebook itself, Oren has kindly offered a 20% off discount, click to buy now.

]]>

Photos from JSConf AU 2014

2014-04-10T12:09:00.000000Z

JSConf AU has been fun - great talks and great venue! It was also a wonderful opportunity for me to test out my new camera kit: SONY A7r + Sony Sonnar T* FE 35mm f/2.8 ZA + Voigtlander Color Skopar 20mm f/3.5 SL-II with Novoflex SONY E-Mount to Nikon adapter.

Here’s the kit (taken by Nikon D800 + Tamron SP 24-70mm F/2.8 Di VC USD):

Photos from JSConf AU 2014:

Breakfast in North Melbourne:

Conference:

Check out my Google+ Photo Album for larger versions.

]]>

On Hiring: Trial Week - Yay or Nay?

2013-10-29T12:09:00.000000Z

Today a blog post titled “Trial Week: Our Hiring Secret“ has made to the Hacker News homepage. I naively tweeted my dislike and now I feel obligated to share my thoughts in a more meaningful and constructive way.

First of all, congratulations to the Weebly team, as this trial week strategy is clearly working very well for them.

I, on the other hand, am against using a trial week for vetting candidates, and I am going to share my thoughts.

Let this serve as a reminder to the rest of us: every organisation and team is different, so think carefully before committing to a given strategy.

One Week is a Major Commitment for the Candidate

In Australia, a full time employee typically gets four weeks of annual leave, with one or two weeks of which used up for the Christmas / New year down time. We are looking at asking candidates to spend 33-50% of their vacation time to commit to a trial week for one company - a terrible ROI (Return On Investment) from the candidate’s perspective if you ask me.

Candidates who are currently employed, with multiple offers from other organisations are more likely to skip the trial week - from experience, this is often the higher quality candidate pool.

Side Effects

Increases the likelihood of burnout due to the reduced vacation time
Shrinks the candidate pool
Misses top talents who are unable to make the one-week commitment
As a result, the overall quality of the candidate pool drops
Paints an image of “not-caring (enough) about the employee’s well being”

Of course, since the trial week is paid for, the employee could always take unpaid leave from their current employer.

Side Effect

Raises alarm bells at current workplace since one week of unpaid leave is significant

One Week is a Major Commitment for the Team

Given the trial only lasts a week - we better make it count! That means one or more current developers need to be assigned to take care of the trial developer - pairing and walking through existing systems, etc. This is assuming we are going to act responsibly, and not simply just direct the trial developers to their desks and ask them to “go for it”.

Side Effects

Higher pressure for the team
More difficult to act on other priority tasks

Developer Productivity Curve (One Week is Not Enough)

From my experience of on-boarding new developers, it typically takes 4-8 weeks for a developer to become productive and effective in a new work environment.

According to Weebly, candidates are assigned with a project that is small enough to do in a week, but still resembles what the candidate would be doing if hired. It sounds great if it works, but for many organisations this is unfeasible, for instance:

There is no small projects to assign, unless invented
Navigating documentation and source code would take days, if not weeks

Either way, with one week of trial, the candidate is unlikely to have enough time to contribute as well as to be integrated into the team culture.

Side Effects

Higher chance of misjudging the candidate’s ability and productivity
Significantly higher chance of creating solutions misaligned with the team and/or the organisation
Higher maintenance cost should the team decides to keep the solutions created

66% Hire Rate Suggests Deeper Hiring Issue

Weebly at the end of their blog post writes:

Our hire rate out of trial week is around 66%, which feels like the right level.

I respectfully disagree. A 66% hire rate from the trial week is a 34% failure rate on the pre-trial week recruitment process, and this is significant.

Which brings us to…

More Effective Ways to Vet a Candidate

Where I work, we have a simple, three-step recruitment process:

Complete a small and fun code challenge, in your own time and with your own pace. The code challenge usually takes 2-4 hours to complete.
Invited to our office to chat with our developers and founders, optionally done via video chat. This usually takes an hour or so.
Pairing session, usually takes 30-60min.

Step 2 and 3 are sometimes swapped. And we also check out the candidate’s Github account if available, and their past projects if public.

In the code challenge we vet the candidate’s problem-solving ability, software design sense, code quality, code style and ethics (it’s easy to tell whether they cheated).

During the chat we vet the candidate’s project experience, depth of knowledge, breadth of knowledge, communication skill and culture fit.

In the pairing session we vet the candidate’s development practice, thought process and the ability to articulate.

By the end of the three steps we are usually pretty confident on +1 or -1 to hire the candidate. If we aren’t, it’s a -1.

But hold on, didn’t I mention one week is not enough for a candidate to be productive and effective? Yes! And that’s why most places have a three-month probation.

The difference between the long probation period and the short trial period, is not only in duration, but more importantly in commitment. In my opinion, only when both parties are committed can you achieve great result.

So, let’s hear your say, what do you think? :)

Poll: Trial Week, Yay or Nay?

]]>

Protip: Unsync the "Index" Folder of Sublime Text 3 from Dropbox

2013-10-12T04:26:30.000000Z

If you’re like me who uses both Sublime Text 3 and Dropbox, chances are you have your Sublime Text 3 folder synced in Dropbox.

I use my laptop as my primary workstation so most of the time it’s docked and charged. Occasionally when I do use it on battery power however I notice the extremely poor battery life - typically only 2-3 hours.

Eventually I realised the power consumption was caused by Sublime Text 3 generating a temp file in its “Index” folder every second or so, and that triggers Dropbox to do the syncing - causing it to take 40-50% of CPU constantly.

So, here’s a simple fix. Click on the Dropbox icon, then click on the gear icon and select “Preferences…”, select the “Advanced” tab, and select “Change Settings…” for Selective Sync to bring up the column-based file dialog, now simply find and untick the “Index” folder and you’re all set. :)

]]>

Protip: Faster Ruby Tests with DatabaseCleaner and DatabaseRewinder

2013-09-18T06:50:00.000000Z

Please also see this blog post on tweaking your ruby GC settings.

I use and love DatabaseCleaner, although historically I had never paid too much attention on the performance of its varies cleaning strategies - I’d always used truncation.

We use Postgres, and after digging around and finding out the difference between DELETE and TRUNCATE, I ended up improving our test suite speed by about 30-40% simply by tweaking the cleaning strategies.

RSpec.configure do |config|
  config.before :suite do
    DatabaseCleaner.clean_with :truncation
    DatabaseCleaner.strategy = :transaction
  end

  config.before do
    if example.metadata[:js] || example.metadata[:type] == :feature
      DatabaseCleaner.strategy = :deletion
    else
      DatabaseCleaner.strategy = :transaction
      DatabaseCleaner.start
    end
  end

  config.after do
    DatabaseCleaner.clean
  end
end

Essentially, we want to truncate the DB only once before the whole suite runs to ensure a clean slate DB, then we only want to use deletion on Capybara tests, everything else should just use transaction which is the fastest strategy.

Now, as a bonus, I have just discovered @amatsuda’s DatabaseRewinder which is a lightweight alternative that supports only ActiveRecord. It offers comparable performance with a much similar API.

RSpec.configure do |config|
  config.before :suite do
    DatabaseRewinder.clean_all
  end

  config.after do
    DatabaseRewinder.clean
  end
end

By the way, we also use parallel_tests to scale our test suite to multiple processes, even on Travis CI and Wercker.

Hooray to faster tests! :)

]]>

Protip: Ruby Devs, Please Tweak Your GC Settings for Tests!

2013-09-06T11:13:37.000000Z

It was made apparent to me that many ruby devs either aren’t aware or couldn’t be bothered to tweak their ruby garbage collector settings.

Well, if you are using MRI, please start tweaking your GC settings. Here’s what I use (on my 15” Macbook Pro Retina):

export RUBY_GC_MALLOC_LIMIT=90000000
export RUBY_FREE_MIN=200000

Not only can you tweak it for your local dev machine, you can also tweak Jenkins, Travis CI, Wercker and other CI solutions, making instant speed gain for your test suite!

Here’s what we get:

            Before After
Local:      8m22s 5m52s
Travis CI:  10m10s 6m0s
Wercker:    8m45s 6m24s

YMMV depending on your system and available RAM.

]]>

Writing Sensible Tests for Happiness

2013-08-26T14:11:00.000000Z

Writing good, sensible tests is hard. As a Rubyist, I feel lucky to be part of a community that embraces tests. Though at the same time, I have come across too many projects that suffered from not having sensible tests.

What are Sensible Tests?

There often isn’t a silver bullet when it comes to software development. Technical stuff aside, many things contribute to the solution to a given problem - the team, the project and the business to name a few. This article does not attempt to present any insights into the best practices for testing, rather it collects a few tips I believe would benefit those who are not yet comfortable with writing tests.

To me, sensible tests often have the following characteristics:

it does not replicate implementation details;
it does not provide false sense of security;
it runs reasonably quickly;
it does not slow down the development significantly;
it guides the programmer towards a better architecture;
and, it does not make you sigh every time you want to modify your tests.

Art and Science, TDD or Not TDD

Just like writing production code, writing tests is also a combined form of art and science. It takes not only experience, but also intuition to write sensible tests. You have to remember that not all projects and programmers are equal - take what you get, practise, and reflect on your findings.

Many times I had come across seasoned programmers practising TDD, only to find themselves cornered into a bad design that ultimately had to be thrown away. TDD does not save you from writing bad code, this article is not about TDD, it’s about testing in general.

I am most comfortable with using RSpec, FactoryGirl, Capybara and Turnip, so I’m going to use these tools in the code. The principles however apply to any testing framework.

Test as Little as Possible to Reach a Given Level of Confidence

Kent Beck, the inventor (or more correctly, ‘rediscoverer’) of TDD once said:

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence.

I used to prefer testing almost everything, but over the recent years I find myself increasingly look for key areas of the system that need the test coverage the most. Typically, our systems would have:

unit and functional tests for model behaviours
unit and functional tests for services
integration tests for controller actions
request tests for API endpoints
isolated JavaScript tests
high level integration/acceptance tests in Gherkin

Model and service level tests are arguably the most important ones so we make sure we have really good test coverage for those. For controller tests we rely heavily on reusable production and test code for maintainability and sanity. For API endpoints we mostly test presented data structure - as business logic and data integrity should have been covered in model, service and controller layers. Isolated JavaScript tests take care of both presentational business logic and tricky UI tasks. And finally, acceptance tests handle happy-path user interactions.

Do Not Test Framework and Library Code

Writing application-specific business logic is difficult enough, you really should not test functionalities provided by the framework or libraries. Below is an example of such bad tests:

describe ApprovalStakeholder do
  it { should belong_to(:approval) }
  it { should_not validate_presence_of(:approval) }
end

Similarly to how you would add useful comments, i.e. describe why instead of what, these tests should be replaced by tests that cover actual functionalities, for instance the reason why an ApprovalStakeholder doesn’t need an Approval to be presence should be demonstrated in the tests:

shared_examples_for "non-approval specific stakeholder" do
  its(:action_that_does_not_care_about_approval) { should be_true }
end

describe ApprovalStakeholder do
  let(:approval) { create(:approval) }
  let(:user) { create(:user) }
  let(:role) { create(:role) }

  subject do
    build(:approval_stakeholder,
      :user_id => user.id,
      :role_id => role.id
    )
  end

  context "with an approval" do
    before { subject.approval = approval }

    it_behaves_like "non-approval specific stakeholder"

    its(:action_that_does_care_about_approval) { should be_true }
  end

  context "without an approval" do
    it_behaves_like "non-approval specific stakeholder"

    its(:action_that_does_care_about_approval) { should be_false }
  end
end

Ensure What You are Testing Makes Sense

The test case below showcases the original developer’s lack of attention and awareness on designing a functional and secure system. It actually tests the reference keys for the ApprovalStakeholder object are allowed to be mass assignable, which is a recipe for disaster.

describe ApprovalStakeholder do
  it { should allow_mass_assignment_of(:user_id) }
  it { should allow_mass_assignment_of(:role_id) }
end

De-Duplicate Test Cases

Looking at the example below, the first thing you’d notice is the amount of duplication.

describe ApprovalStakeholder do
  it "#traveller" do
    stakeholder = create(:approval_stakeholder,
      :approval => approval,
      :user_id => traveller.id
    )
    stakeholder.stub(:user).and_return(traveller)
    approval.stub(:stakeholders_as).and_return([stakeholder])

    approval.traveller.should == traveller
  end

  it "#authoriser" do
    stakeholder = create(:approval_stakeholder,
      :approval => approval,
      :user_id => authoriser.id
    )
    stakeholder.stub(:user).and_return(authoriser)
    approval.stub(:stakeholders_as).and_return([stakeholder])

    approval.authoriser.should == authoriser
  end
end

It’s true that tests act as a form of specification therefore should be optimised for clarity, in this case however, we could still maintain the clarity with significantly reduced duplication:

describe ApprovalStakeholder do
  let(:stakeholder) do
    create(:approval_stakeholder,
      :approval => approval,
      :user_id => user.id
    )
  end

  subject { approval }

  before do
    stakeholder.stub(:user).and_return(user)
    approval.stub(:stakeholders_as).and_return([stakeholder])
  end

  describe "#traveller" do
    let(:user) { traveller }

    its(:traveller) { should == traveller }
  end

  describe "#authoriser" do
    let(:user) { authoriser }

    its(:authoriser) { should == authoriser }
  end
end

Do Not Replicate Implementation Details

I am often surprised to see many seasoned developers “enjoy” writing tests that essentially replicate the production code logic without much benefit. See below:

describe ApprovalStakeholder do
  it "references a user" do
    approval_stakeholder = build :approval_stakeholder, :user_id => 1
    User.should_receive(:find).with(1)
    approval_stakeholder.user
  end

  it "references a role" do
    approval_stakeholder = build :approval_stakeholder, :role_id => 1
    Role.should_receive(:find).with(1)
    approval_stakeholder.role
  end
end

Rather than creating noisy tests, tests with actual assertions seem much more meaningful and readable:

describe ApprovalStakeholder do
  subject do
    build(:approval_stakeholder,
      :approval => approval,
      :user_id => user.id,
      :role_id => role.id,
    )
  end

  its(:name) { should == "#{user.first_name} #{user.last_name}" }
  its(:role_name) { should == role.name }
end

Reduce the Reliance on Mocks and Stubs

This is a difficult and often-debated subject. In my experience, having too many mocks and stubs even though speeds up the test suite, usually leaves too many holes in your tests and makes the test suite less accurate and effective. Fortunately, by using more service objects (described below), mocking and stubbing become more manageable as you use them mostly on external objects and interfaces.

Take Apart the System, One Service at a Time

If you’re a Rails developer, you are already familiar with MVC. But just relying on MVC to hold your application architecture is probably not going to be sufficient for an average modern day web application. Many people like Service-oriented architecture, so do I.

Services are unassociated, loosely coupled units of functionality that are self-contained.

In my experience, as long as you are disciplined in having services do one and only one thing really well, testing becomes much easier.

For instance, we have a Bouncer service that is responsible for safeguarding resources - ensuring read-only attributes don’t get overridden.

module Services
  class Bouncer
    def self.guard(resource, options = {})
      if options[:existing_resource]
        resource.readonly_attributes.each do |attr_name|
          resource.send("#{attr_name}=", options[:existing_resource].send(attr_name))
        end
      end

      resource
    end
  end
end

The corresponding tests for this service are both fast and self-contained:

describe Services::Bouncer do
  class BouncerDude
    include Mos::Entity

    set_readonly_attributes :age, :gender

    attribute :name
    attribute :age
    attribute :gender
  end

  let(:resource) { BouncerDude.new(name: 'Penny', age: 28, gender: 'female') }
  let(:existing_resource) { BouncerDude.new(name: 'Sheldon Cooper', age: 34, gender: 'male') }
  subject { Services::Bouncer.guard(resource, existing_resource: existing_resource) }

  describe "#guard" do
    its(:name) { should == 'Penny' }
    its(:age) { should == 34 }
    its(:gender) { should == 'male' }
  end
end

Recognise Common Patterns and Refactor Them into Services

One of the reasons why service-oriented architecture is so popular is because things are broken down into smaller, more manageable and more testable pieces. It is especially helpful for TDD practitioners as it significantly reduces the amount of coupling between your production code and your tests due to having simpler internals per test subject.

Take a look at the below example, which is hard to read, hard to test and error-prone:

module ApplicationHelper
  def branch_logo_options(branch)
    BranchLogo.where(branch_id: branch.id).map { |logo| [logo.file, logo.id] }
  end

  def branch_options(agency)
    BranchRepository.find(agency_id: agency.id, archived: false).map do |b|
      [b.name, b.id]
    end
  end

  def agency_user_options(agency, filtered_users)
    filtered_user_ids = filtered_users.compact.map(&:id) || []
    AgencyUserRepository.find(agency_id: agency.id, archived: false).select do |u|
      !filtered_user_ids.include?(u.id)
    end.map { |u| [u.full_name, u.id] }
  end

  def current_agency_user_options(filtered_users = [])
    agency_user_options(current_agency, filtered_users)
  end

  def current_agency_trust_bank_account_options
    BankAccountRepository.find(
      agency_id: current_agency.id,
      archived: false,
      account_type: BankAccount::TRUST_ACCOUNT).map do |b|
      [b.account_name, b.id]
    end
  end

  def code_options_for(klass)
    klass.all.map { |cc| ["#{cc.code} - #{cc.name}", cc.id] }.sort
  end
end

Let’s refactor it into something more manageable, by introducing a service ShowGirl for fetching and presenting data collections:

module CollectionOptionsHelper
  def branch_logo_options(branch)
    Services::ShowGirl.present(branch, from: BranchLogo, show: :file)
  end

  def branch_options
    Services::ShowGirl.present(current_agency, from: BranchRepository)
  end

  def consultant_options(excluded_users = [])
    Services::ShowGirl.present(
      current_agency,
      from: AgencyUserRepository,
      show: :full_name
    ) do |collection|
      collection.reject { |user| user.id.in?(Array.wrap(excluded_users).map(&:id)) }
    end
  end

  def trust_bank_account_options
    Services::ShowGirl.present(
      current_agency,
      from: BankAccountRepository,
      show: :account_name,
      filters: { account_type: BankAccount::TRUST_ACCOUNT },
    )
  end

  def code_options_for(name)
    Services::ShowGirl.present(
      current_agency,
      from: Admin::Configurations::Essential.descendants.find { |d| d.name =~ /::#{name.to_s.classify}/ },
      show: -> (item) { "#{item.code} - #{item.name}" }
    )
  end
end

Better yet, we can clean it up even further by introducing another service, BusBoy for just serving the data, and leaving ShowGirl for only presenting the data:

module CollectionOptionsHelper
  def branch_logo_options(branch)
    Services::ShowGirl.present(
      Services::BusBoy.serve(:branch_logos, branch: branch)
    )
  end

  def branch_options
    Services::ShowGirl.present(
      Services::BusBoy.serve(:branches, agency: current_agency)
    )
  end

  def consultant_options(excluded_users = [])
    Services::ShowGirl.present(
      Services::BusBoy.serve(:consultants, agency: current_agency),
      show: :full_name
    ) do |collection|
      collection.reject { |user| user.id.in?(Array.wrap(excluded_users).map(&:id)) }
    end
  end

  def trust_bank_account_options(account_type)
    Services::ShowGirl.present(
      Services::BusBoy.serve(:bank_accounts,
        { agency: current_agency, BankAccount::TRUST_ACCOUNT }
      ),
      show: :account_name
    )
  end

  def code_options_for(name, options = {})
    Services::ShowGirl.present(
      Services::BusBoy.serve(name, agency: current_agency), options
    )
  end
end

Basic Controller CRUD Actions

In one of our projects we have lots and lots of forms. Consequently we have lots and lots of CRUD actions. In order to keep our sanity as well as to make basic CRUD controllers maintainable, we have a custom DSL to make CRUD actions portable and testable:

module Profiles
  class TravellersController < BaseController
    authorize_resource class: Traveller

    datamappify_resources entity: Traveller,
                          repository: TravellerRepository,
                          filter_by: :agency_id,
                          filter_value: -> { current_user.agency_id }
  end
end

Most of our controller tests look like this:

require 'spec_helper'

describe Profiles::AccountsController do
  let(:existing_resources) { [] }
  let(:create_resource) { Mos::Data.create_account }
  let(:create_resources) { Mos::Data.create_accounts(2) }
  let(:a_resource) { assigns(:resource) }
  let(:invalid_param) { { name: '' } }
  let(:params_key) { :account }
  let(:redirect_path) { profiles_accounts_path }

  it_behaves_like 'datamappify resources controller'
  it_behaves_like 'searchable resources controller', :name,
                                                      :profile_id,
                                                      :branch_id,
                                                      :activated

  describe "permission" do
    context 'as a manager' do
      before do
        sign_in_as :manager
      end

      it_behaves_like 'with write access'
      it_behaves_like 'with read access'
      it_behaves_like 'with index access'
    end

    context 'as a consultant' do
      before do
        sign_in_as :consultant
      end

      it_behaves_like 'without write access'
      it_behaves_like 'with read access'
      it_behaves_like 'with index access'
    end
  end
end

API Endpoint Tests

One of our projects at work is an API service that is essential to our platform. Naturally, we not only need to test the models, services and controllers, we also need to ensure the API endpoints do what they are supposed to do - mostly exposing the correct data structure.

During the early stage of the development, I had come up with ApiTaster - a super useful gem for visually testing our Rails application’s APIs. Later on, as we continued to grow our API endpoints, we started utilising ApiTaster for our automated test suite too.

In essence, we have one API spec file responsible for describing which endpoints are tested and missed according to the information given by ApiTaster:

describe "API" do
  load 'db/seeds.rb'
  load 'spec/api_endpoints.rb'

  ApiTaster::Route.map_routes

  ApiTaster::Route.defined_definitions.each do |route|
    it "api endpoint #{route[:verb]} #{route[:path]}" do
      params = ApiTaster::Route.params_for(route).first
      expectation = ApiTaster::Route.metadata_for(route)[:expectation]
      setup = ApiTaster::Route.metadata_for(route)[:setup]
      verb = route[:verb].downcase
      path = parse_path_with_url_params(route[:path], params[:url_params])

      setup.call if setup

      send verb, path, params[:post_params]

      response.body.should match_json_expression(expectation)
    end
  end

  # warn about undefined definitions
  ApiTaster::Route.missing_definitions.each do |route|
    pending "api endpoint #{route[:verb]} #{route[:path]}"
  end
end

Then, we have a bunch of endpoint test files to do the actual testing, like this:

resource_response = ResponseHash[
  :response => {
    :id => Integer,
    :name => String,
    :token => String
  }
]

get '/:version/company', {}, {
  :expectation => resource_response
}

post '/:version/companies', {
  :model => FactoryGirl.attributes_for(:company)
}, {
  :expectation => resource_response
}

put '/:version/companies/:id', {
  :id => 1,
  :model => { :name => 'New Company' }
}, {
  :expectation => resource_response.with(:name => 'New Company')
}

delete '/:version/companies/:id', {
  :id => 1
}, {
  :expectation => resource_response
}

Notice that for API endpoint tests we don’t test the business logic or data integrity - these should be tested in models, services and controllers. What we do test are correct endpoints are exposed, correct parameters are accepted and correct data structures are returned.

Isolated JavaScript Tests

Many developers prefer to rely on their integration test suite to do JavaScript / UI testing. This approach is fine until you start making lots of front-end changes and constantly need to pinpoint the relevant feature spec.

Having an isolated JavaScript test suite (which should be run as part of your continuous integration process) is extremely beneficial and often saves debugging time.

I like Mocha so we use Konacha in our Rails app. Though Mocha with Chai is really not that different to Jasmine.

Custom JavaScript behaviour is obviously a good candidate for isolated testing:

#= require spec_helper

describe "form toggle", ->
  beforeEach ->
    $("body").append(JST["templates/form/toggle"])

  it "hides the collapsible field by default", ->
    $(".control-group.branch_deactivation_date").hasClass('in').should.be.false

  it "does not override if there is already a value", ->
    value = $("input#agency_deactivation_date").val()
    $("input#agency_activated").click()
    $("input#agency_deactivation_date").val().should.equal(value)

Sometimes it’s also useful to ensure library code is initiated and triggered correctly, if you have other custom JS interact with it:

#= require spec_helper
#= require bootstrap-datepicker

describe "form dates", ->
  beforeEach ->
    @dateFormat = 'DD/MM/YYYY'
    $("body").append(JST["templates/form/dates"](dateFormat: @dateFormat))

  it "has a placeholder", ->
    $("input").attr("placeholder").should.equal(@dateFormat)

  it "defaults to today's date", ->
    $("input#empty").focus()
    $("input#empty").focus()
    $("input#empty").val().should.equal(moment().format(@dateFormat))

  it "does not override if there is already a value", ->
    value = $("input#filled").val()
    $("input#filled").focus()
    $("input#filled").val().should.equal(value)

“Real” UI Tests

Isolated JavaScript tests are super fast and useful. However, there are times when having pure JavaScript tests simply isn’t enough, due to the complicated nature of DOM interaction and template rendering.

A while ago our calendar widget was broken due to a production and UAT environment issue that was not picked up by our JavaScript test suite. Since then we started adding dedicated UI tests in our acceptance test suite (we use Turnip):

@ui
Feature: UI
  Background:
    Given I am signed in
      And I go to agency consultants page
      And I click on "Add New Consultant"

  Scenario: Calendar
      When I click "#agency_user_start_date"
      And I click ".day.active" within ".datepicker"
      Then I should see today as part of the date field

Effective Acceptance Tests

Writing acceptance tests - also known to many Rubyists as “Cucumber tests”, is a double-edged sword - it’s extremely useful, but very few developers can write good, maintainable Gherkin-style acceptance tests.

Here’s an example of a badly written feature spec with too much implementation details and noise:

Feature: Session
  Background:
    Given I visit "/"
      And there is a user "admin" "password"

  Scenario: Sign in with valid credentials
      When I fill in "Username" with "admin"
      And I fill in "Password" with "password"
      And I click "Sign In"
      Then I should be on "/dashboard"

  Scenario: Sign in with invalid credentials
      When I fill in "Username" with "admin"
      And I fill in "Password" with "invalid_password"
      And I click "Sign In"
      Then I should not be on "/dashboard"

  Scenario: Sign out
      When I fill in "Username" with "admin"
      And I fill in "Password" with "password"
      And I click "Sign In"
      And I click "Sign Out"
      Then I should be on "/sign_in"

A much cleaner version with only high level, descriptive steps:

Feature: Session
  Background:
    Given I am on the homepage
      And there is a user "admin" with password "password"

  Scenario: Sign in with valid credentials
      When I sign in as "admin" with password "password"
      Then I should be signed in

  Scenario: Sign in with invalid credentials
      When I sign in as "admin" with password "invalid_password"
      Then I should not be signed in

  Scenario: Sign out
    Given I am signed in as "admin" with password "password"
      When I sign out
      Then I should be signed out

Final Thoughts

Writing good, sensible tests is hard. These examples and tips are by no means the silver bullet, and you might actually find some of them counter-intuitive in your particular situation. So again, take what you get, practise, and reflect on your findings. For Happiness! :)

Do you have any tips to share? If so please feel free to add a few comments!

]]>

Gotchas in the Ruby Sequel Gem

2013-08-21T12:39:00.000000Z

I haven’t really used Sequel much therefore I am definitely a newbie. However, after days and nights of frustration, endless debugging and some search-fu during the development of Datamappify, I have finally arrived at the conclusion that Sequel is a capable library, as long as you are aware of the gotchas.

Gotcha 1: Always use “`select`“/“`select_all`“, or your data records will mysteriously have wrong IDs!

In ActiveRecord, joining an associated model couldn’t be simpler:

Post.joins(:author)

In Sequel, despite having a similar API for models to declare associations and their corresponding primary and foreign keys, you cannot do a join without specifying the keys:

Not good:

Post.join(:authors)
# or
Post.join(Author)

Better:

Post.join(:authors, :id => :author_id)

You would think the version above works - it doesn’t. Even worse, the above example will give you incorrect data - the IDs of the Post records will now contain the IDs from their corresponding Author records! This is because upon a join, Sequel merges attributes from both models into a single hash.

The correct version:

Post.join(:authors, :id => :author_id).select(:posts __id, :posts__ title, :posts__body)
# or
Post.join(:authors, :id => :author_id).select_all(:posts)

Gotcha 2: Always call “`all`“ at the end of the chain, or the chain will present data in a different format.

In ActiveRecord, all of the below examples return an ActiveRecord::Relation collection:

Post.where(:title => 'Hello world')
Post.joins(:author)
Post.includes(:author)

And indeed, calling first on any of them returns an object of class Post (assuming the result collection is not empty).

Post.where(:title => 'Hello world').first.class #=> Post
Post.joins(:author).first.class #=> Post
Post.includes(:author).first.class #=> Post

In Sequel, the below examples all return a Sequel::DataSet collection:

Post.where(:title => 'Hello world')
Post.eager(:author)
Post.eager_graph(:author)

But let’s see what we get from calling first.class on them:

Post.where(:title => 'Hello world').first.class #=> Post
Post.eager(:author).first.class #=> Post
Post.eager_graph(:author).first.class #=> Hash

Huh? Last one is a Hash? It turns out, if you call all at the end of chains to convert them to Arrays, then the returned collections are consistent:

Post.where(:title => 'Hello world').all.first.class #=> Post
Post.eager(:author).all.first.class #=> Post
Post.eager_graph(:author).all.first.class #=> Post

]]>

The Future of Computing, The Future of Computer Programmers - An Interview with Yukihiro "Matz" Matsumoto

2013-06-29T12:04:00.000000Z

A while ago I translated an interview with Matz done by a Chinese book publisher. The interview and the translation were well received, so this time I am translating another interview with Matz, done by Ito, the editor-in-chief from Japanese website Engineer Type. Since I don’t read Japanese, the translation is based on Turing Book’s Chinese translation.

The Chinese translator has done a great job translating the interview, but there are still many words and sentences lack sufficient context and therefore are difficult to grasp. I have put in many hours translating the text as well as doing researches to ensure the final article is readable. I hope you will enjoy it! :)

Ito: Thank you for doing an interview with us, Matz. I have just finished reading your latest book The Future of Computing, could you perhaps talk about the future of programming and software programmers in general?

Matz: Hmmm, this is difficult to answer… but thanks for reading my book!

Ito: In the book you’ve shared your thoughts on the past, present and future of different programming languages and software design patterns. Would you like to talk about the current state of the software industry? And is there going to be another paradigm shift in software development?

Matz: As discussed in my book, to predict the future of a high tech industry such as computing is not particularly difficult. I believe in the foreseeable future the computing industry is still going to advance based on Moore’s law. Although, it is possible that in the next year or two quantum computers become a practical reality, in that case it will change everything! *chuckles* On a serious note, according to Moore’s law, the cost of computing will decrease and the performance and capacity of computing will increase - this basic principle is unlikely to change. One thing I did notice in recent years is that due to the advancement in computer hardware, the software industry is subtly changing too.

Software Development in the Era of Multi-core and Cloud Computing

Matz: It was about twenty years ago (in 1993) I invented the Ruby programming language, yet it still runs surprisingly well on modern computers.

What this means is that in the past twenty years the computing environment which the software runs on did not see any fundamental changes. In recent years, we started seeing computing power being shifted from having higher CPU frequencies to being distributed over more CPU cores. And that means software needs to move in that direction too.

Matz: Software has not seen major changes for years.

Ito: And this is covered in the last chapter from the book, right?

Matz: Yes. Similarly to multi-core, cloud computing is advancing in the same direction. The future of computing is all about utilise multiple CPUs or computers effectively.

Ito: So, how does that change software development?

Matz: In the past ten years or so we have been seeing more and more things happen on the Internet, and the Internet is an amazing application platform for extension and distribution. Compared to software engineers working on mainframe computers, web developers are naturally more familiar with the concepts of multi-core and cloud computing.

Ito: After interviewing many web and mobile startups, we realised that the number of software engineers working in PaaS and cloud computing have been increasing rapidly.

Matz: Absolutely. And I do believe that “not needing to purchase and own dedicated hardware” is going to be the mainstream. The idea and thought process of “not owning” is not only important for software development, but also important for business development.

“Owning” Becoming a Liability, Not Asset

Matz: In the past, “owning” was seen as the source of vitality of a corporation - those who own high performance mainframe computers were able to do business transactions in high volume whereas those who do not were not able to compete.

These days the landscape is changing - those corporations who do not “own” expensive hardware have more competitive edge. Let’s say it takes five years to break even from the expensive investment of servers, during that time those machines are being put in use to realise their full potential and to justify their cost. It may appear to have saved the business cost but it is not, simply because the value of the hardware decreases as each day passes by.

To put simply, we are now entering the era of “owning” being a liability rather than asset. If you had the most advanced hardware, software engineers were able to develop efficiently. On the contrary, if you didn’t, then you might want to get used to the hours-long waiting for the code to compile. *chuckles* The rise of cloud computing platforms like Heroku is making “owning” a thing of the past.

Also, “not owning” has several advantages on the development as well as the commercial front. For instance, it allows many startups to rise. In the past, in order to start a new business you would need capital for purchasing servers and/or renting servers in a data centre. These days, to get started on a platform like Heroku couldn’t be easier, for example on Heroku you could start with just one dyno for free. This new way of developing software significantly reduces costs and risks.

Years ago I read an essay called Ramen Profitable by Y Combinator’s founder Paul Graham. “Not owning”’s flexibility and agility contribute a great deal to it. And this trend has now grown beyond just relevant to startups, in fact in the recent years many large corporations have begun adapting this approach too.

In the United States, corporations like Disney and Best Buy are indeed utilising Ruby, Rails and Heroku to rapidly grow their internal infrastructure in a cost-effective fashion. What was once considered competitive edges to venture capitalists, like “rapid development” and “development flexibility” are now also possible for these giant corporations.

Ito: What about the giant corporations in Japan?

Matz: I have never worked in a big corporation so I can’t tell where they are heading. People have been optimistic, though as an observer I am concerned.

Real Benefits of Innovation in Cloud Computing Not to Be Overlooked

Ito: What makes you concerned about software development?

Matz: The traditional approach of developing software is still the norm. For example, some corporations, even though use Amazon Web Services, still rely on system administrators to handle their infrastructure. It is too common to see a software development team consists of over a dozen people.

This in my opinion defeats the purpose and forfeits the benefits of “not owning” servers. There are simply too many of these case studies whereby only on the surface of cloud computing is explored and understood.

I have to say I am disappointed by some of the so-called “private clouds” owned by large corporations. The advantage of cloud computing is to utilise multiple computers in the cloud, but those private clouds are essentially their internal data centres. Isn’t that the same as owning a bunch of servers?

Matz: Many companies barely scratch the surface of emerging technologies.

Ito: Indeed, it is too often to see the real benefits of emerging technologies overlooked or misunderstood. Anything else that makes you concerned about the future?

Matz: Nowadays the speed of development has always been a priority from big development projects in B2B to small development projects in many startups. Yahoo! Japan even coined a term “爆速化” (explosively high speed) to indicate the importance of development speed in the ever more competitive and engaging markets.

Looking at things this way, those so-called “system integrators” are becoming obsolete. Should they just give up what they do or continue? I don’t know, but I do know that the gap between them and engineers who have the capability and skills to create real value is increasing.

Career Longevity of Software Engineering

Ito: Who are those engineers who have the capability and skills to create real value?

Matz: The ones who would put in effort to create software or systems from a prototype to a final product. And this has nothing to do with whether they work in web or system integration, or whether it’s consumer oriented or corporate oriented.

Matz: Unbalanced skill combination leads to a gloomy future.

Ito: Do you mean the engineers who are capable from design to implementation?

Matz: Yes. Speaking of which, software developers have to know more than just system design - they cannot survive without knowing how to code. Just like in life, you cannot survive without being down to earth. *chuckles*

Despite the fact that it is pointless to have someone doing only the system design, and not the development as well, the System Integration industry is still going strong in Japan - and it is in fact an industry with high profit margin.

Even if the system designers came up with questionable specifications, or if the programmers were sloppy so the software was terrible to use, users would still use it despite whining. Flaws are easily glossed over under high profit margins.

But just as discussed before, as development speed increases, profit margins would undoubtedly become smaller. Flaws are therefore harder to gloss over.

In my opinion, if things don’t change, those run-of-the-mill software engineers might not survive in five years. Worse, the junior to mid-level to senior programmer corporate ladder is going to collapse.

Say, you wouldn’t want to start a VHS rental shop when DVDs were on the rise, would you?

Difference Between Those Who Control Their Destiny to Those Who Don’t

Ito: Do you have any advice for those who do not wish to be in a “gloomy future”? What can they do?

Matz: To innovate and to create new things, I suppose.

It’s not all doom and gloom. Even though many ageing technologies have been or are being replaced by the web, jobs will not disappear overnight. I think many software developers will still be employed in those jobs.

Having said that, it is always good to create new things or even invent new programming languages.

Ito: What are these “new things”?

Matz: I see three types of new things.

First of all, new services. If you can create a new service, or a service that offers superior user experience - it would be an innovation.

Secondly, new technologies. To come up with technologies better than the existing ones - and this is what I have been doing.

mruby was released earlier this year on Github.

Or, another way is to invent new algorithms.

The three ways I mentioned have different difficulties, but they share the same goal - to create something that hasn’t existed yet. Those who keep working on these kinds of challenges are the true outstanding software engineers.

The ones who do not challenge themselves to create new things are often falling behind - they learn a hip new language today and try a new web framework tomorrow, but still lack the foresight to invent and to improve.

Of course, it is important to learn and try new things, but if you see them as your ultimate goal then you will lose control of your destiny. I believe that the ones who do not get boggled down in every new trendy thing will ultimately be happier.

Software Development is a Punch to Deficiency

Ito: Here is a sharp question: be a follower rather than an inventor is always easier and perhaps makes more money too. What makes you keep inventing?

Matz: My standard answer would be “because writing and running new programs make me happy”. But the real reason is because I don’t like deficiency.

There are people who have different opinions and thought processes, I would often come up with questions like “why was it done this way” or “this will be too hard to use”.

Matz dislikes deficiency, so he invented the ruby programming language.

Ito: True, but all products more or less reflect their producers’ preferences, right?

Matz: Absolutely, and I am not saying that this is bad or anything. I just hate to point fingers at other people’s preferences - if you don’t like something, make your own! This is a basic trait of a good software engineer, and is what makes open source sustainable.

In open source projects, all the source code is publicly available therefore it is very easy to see how a program is designed. As long as you have ideas on how to improve and optimise the design, you are welcome to do so.

Now it is an entirely different story for certain things in the society. *chuckles* At least in software development, we can rely on our skills and knowledge to improve and to change. If it’s your own creation, it can be adjusted and adapted to suit the ever changing needs.

This is same for Ruby - I like programming languages and more importantly I like improving programming languages myself, and that’s why I still work on Ruby till this day.

Software Development, One of the Rare Careers that Could Make a Change on Your Own

Matz talking about developer happiness, wearing his “Ruby City MATSUE“ polo shirt.

Matz: I think I have the right personality for developing software. Only the software industry can tolerate my carefreeness - am I too arrogant for saying that? *chuckles*

In all honesty, software development is one of the rare careers that could bring positive changes to the society on your own. It’s a wonderful occupation that brings happiness and fulfillment!

Ito: Many people would predict their software future based on theories, but Matz you always use “happiness”.

Matz: That’s right. Because only you can control your own destiny. It doesn’t matter if you were told to do things in a way just becasue “Matz said so” - ultimately, I cannot be responsible for your destiny. You should make your own decisions.

I would still say things like “the future might look like this”, but these are just my personal opinions.

And this is the same even for today’s discussion - if someone thinks he does not agree with what Matz has said, he should follow his own decision and the path he chooses.

Exploring the Future: “You” Are the Only Constant

Ito: Having read The Future of Computing, I remember you talked about the inception and development of varies programming languages. But we all know that the IT industry is moving in a rapid pace, it is difficult to rely on history to guide us through to the future. If multi-core and cloud computing are only just the beginning of a paradigm shift, why did you write about the things happened in the past?

Matz: Technologies progress just like a pendulum clock.

Matz: People see things differently - and I believe the IT industry is progressing in a manner similar to the swing motion of a pendulum clock.

As more and more new programming languages, techniques and frameworks pop up, software development related technologies are progressing whilst seeking for balance.

So, how does “the most balanced from the past” become “the most balanced right now”? Think about how pendulum clock swings and in the past how technologies have emerged - you could then predict roughly what would constitute “the most balanced in the future”.

Use “centralised computing vs distributed computing” as an example, in the past there was usually only one centralised mainframe computer, later on to increase the processing capability commodity server farms were utilised, and now we are moving towards cloud computing.

There is no point to look at a particular past event. If you wanted to predict a technology in the future, knowing what has contributed to the balance of a past technology’s rise and fall is going to help.

Human’s ability is one of the factors too, because we have limited capability as a language designer it is useful to look at what others have done to cater for our ability, and therefore improve and evolve the technology.

In the book I briefly talked about Dart and Go. As a programming language inventor I find it really fascinating to explore the thought processes behind those language designers. And it has helped me to gain a deeper understanding of human behaviour.

Ito: I was going to ask why it is so important to study the past, now I know.

Matz: I mentioned this in the beginning - computing has not seen major changes for years.

Programming languages invented over fifty years ago are still in use today, and Ruby has been around for twenty years now. This proves that computing is progressing slower than what a lot of people believe.

On that note, there are many past cases whereby focuses were put on what was cool and new without understanding why. Compared to those “follower” software developers, the ones who command and understand the principles and theories behind changes and progresses have a much longer career longevity.

If you are a software developer who wants a longer career longevity, please read The Future of Computing! *chuckles*

Ito: Thank you Matz for talking to us today!

]]>

Datamappify - A New Take on Decoupling Domain, Form and Persistence in Rails

2013-06-27T12:09:00.000000Z

This post is about the ruby library we are building - Datamappify, please go check it out.

At Locomote we are building a relatively large web application using Rails. Before we began to lay the foundation, we knew very well that if we wanted the project to be maintainable we had to architect the system with extra care and attention. More specifically, we can’t rely on simply using ActiveRecord which combines behaviour and persistence as our domain models.

We began our search for something that would help us decouple our application from the domain layer down to the form handling. We’ve found a couple of gems that are close to what we were after - Curator, Minimapper, Edr and later on Reform. They are all wonderful gems but unfortunately none of them has everything we need.

Here are the things we need:

Decouple domain logic and data persistence.
Decouple models and forms.
Support ActiveRecord (or at the very least, ActiveModel) so we can still use many of the awesome gems like Devise, SimpleForm and CarrierWave.
Support attributes mapped from different data sources (e.g. remote web services from third-party vendors).
Support lazy loading so that attributes stored on remote data sources will not get triggered upon loading the entities.

All things considered, we bit the bullet and started working on Datamappify.

Before I go into details and examples, here is an extremely simplified overview of Datamappify’s architecture:

As you can see, Datamappify is loosely based on Repository pattern and Entity Aggregation pattern.

Datamappify has three main design goals:

To utilise the powerfulness of existing ORMs so that using Datamappify doesn’t interrupt too much of your current workflow. For example, Devise would still work if you use it with a UserAccount ActiveRecord model that is attached to a User entity managed by Datamappify.
To have a flexible entity model that works great with dealing with form data. For example, SimpleForm would still work with nested attributes from different ORM models if you map entity attributes smartly in your repositories managed by Datamappify.
To have a set of data providers to encapsulate the handling of how the data is persisted. This is especially useful for dealing with external data sources such as a web service. For example, by calling UserRepository.save(user), certain attributes of the user entity are now persisted on a remote web service. Better yet, dirty tracking and lazy loading are supported out of the box!

You can read more about Datamappify in the project’s README. For this blog post I will focus on how we use Datamappify in our Rails application. We are still early days in our application development and Datamappify still has quirks and issues, but I am hoping this post will illustrate some of the key benefits of Datamappify.

Getting Started

Getting started with using Datamappify is really easy. We simply include it in the Gemfile:

gem 'datamappify'

To keep things organised, we put entities and repositories in their respective directories under app:

Entities

The reason we wanted Datamappify to utilise existing ORMs like ActiveRecord, is so that we could still use gems like Devise.

So, we have a UserAccount model that handles authentication:

class UserAccount < ActiveRecord::Base
  devise :database_authenticatable,
          :recoverable, :rememberable, :trackable, :validatable
end

UserAccount has one and only one purpose - user account authentication. Other user behaviours would be contained in either the User entity itself or service objects. Speaking of the User entity, it looks like this:

class User
  include Datamappify::Entity

  attr_accessor :user_account

  attribute :account_username, String
  attribute :account_email, String
  attribute :first_name, String
  attribute :last_name, String
  attribute :activated, Boolean, :default => true

  attributes_from Contact, prefix_with: :work

  validates :first_name, presence: true
  validates :last_name, presence: true

  references :agency

  def full_name
    "#{first_name} #{last_name}"
  end
end

We include Datamappify::Entity in the User class to make it an entity. We set the :user_account accessor is so that we could attach the UserAccount object onto the entity.

The attribute DSL is from Virtus - we get attribute type coercion for free, awesome!

attributes_from is a DSL provided by Datamappify - it essentially “mounts” all the attributes from another entity, in this case the Contact entity, which looks like this:

class Contact
  include Datamappify::Entity

  attribute :email, String
  attribute :phone_number, String
  attribute :fax_number, String

  validates :phone_number, presence: true
  validates :fax_number, presence: true
end

All the attributes and validation rules from Contact are now “mounted” on User. attributes_from Contact, prefix_with: :work is equivalent to:

attribute :work_email, String
attribute :work_phone_number, String
attribute :work_fax_number, String

validates :work_phone_number, presence: true
validates :work_fax_number, presence: true

Validation DSL is provided by ActiveModel::Validations.

references :agency is a convenient DSL provided by Datamappify. It is equivalent to:

attr_accessor :agency

attribute :agency_id, Integer

Now that we have entities, we need a way to retrieve and store them. For that we need repositories.

Repositories

Here is the user repository:

class UserRepository
  include Datamappify::Repository

  for_entity User

  default_provider :ActiveRecord

  map_attribute :account_username, 'ActiveRecord::UserAccount#username'
  map_attribute :account_email, 'ActiveRecord::UserAccount#email'

  map_attribute :work_email, 'ActiveRecord::Contact#email'
  map_attribute :work_phone_number, 'ActiveRecord::Contact#phone_number'
  map_attribute :work_fax_number, 'ActiveRecord::Contact#fax_number'
end

Similarly to an entity, we include Datamappify::Repository to make the class a repository. We specify for_entity to link the repository to an entity, and default_provider to use a specific data provider for unmapped attributes.

Unmapped attributes are the ones not specified in map_attribute, in this case they are first_name, last_name and activated. Unmapped attributes are actually automatically mapped by Datamappify, so the user repository essentially does this:

map_attribute :first_name, 'ActiveRecord::User#first_name'
map_attribute :last_name, 'ActiveRecord::User#last_name'
map_attribute :activated, 'ActiveRecord::User#activated'

The first argument of map_attribute is the name of the attribute from the User entity (e.g. :first_name).

The second argument is a string containing three things:

ActiveRecord is the data provider.
::User is the ActiveRecord model class.
#first_name is the ActiveRecord attribute from the model class.

Even though the User entity is a representation of a user on the domain level, the underlying data structure does not necessary have to be. As you can see from the user repository example, we are mapping :account_username and :account_email to the UserAccount ActiveRecord model we’ve seen before. And we have a bunch of contact details attributes mapped to the Contact ActiveRecord model.

The database schema therefore looks like this:

create_table "contacts", force: true do |t|
  t.string "email"
  t.string "phone_number"
  t.string "fax_number"
  t.integer "user_id"
end

create_table "user_accounts", force: true do |t|
  t.string "username", default: "", null: false
  t.string "email", default: "", null: false
  t.string "encrypted_password", default: "", null: false
  t.string "reset_password_token"
  t.datetime "reset_password_sent_at"
  t.datetime "remember_created_at"
  t.integer "sign_in_count", default: 0
  t.datetime "current_sign_in_at"
  t.datetime "last_sign_in_at"
  t.string "current_sign_in_ip"
  t.string "last_sign_in_ip"
  t.integer "user_id"
  t.datetime "created_at"
  t.datetime "updated_at"
end

create_table "users", force: true do |t|
  t.string "first_name"
  t.string "last_name"
  t.boolean "activated"
  t.integer "agency_id"
end

Note that because the Contact entity is mounted on the User entity, we need a foreign key user_id in the contacts table to link them.

Data providers

Because we are allowed to specify a data provider (i.e. ActiveRecord) for each attribute, we can map attributes to entirely different data providers! For instance, we could have:

map_attribute :first_name, 'ActiveRecord::User#first_name'
map_attribute :last_name, 'ActiveRecord::User#last_name'
map_attribute :activated, 'Sequel::UserActivation#activated'

Now the activated attribute is mapped to the UserActivation Sequel model. This powerful feature would allow us to develop data providers that communicate with remote web services. :)

Repository APIs

Datamappify provides a bunch of APIs for retrieving and storing data. These APIs are being developed as needed during our application development.

For instance, to find a particular user entity by ID:

user = UserRepository.find(1)

To get all the users:

users = UserRepository.all

To search for users:

users = UserRepository.find(:first_name => 'Fred', :activated => true)

To save a user:

UserRepository.save(user)

To delete a user:

UserRepository.destroy(user)

There are some limitations for certain APIs, you can read more about them here.

When saving an entity, because Datamappify has an internal pool for tracking dirty attributes, only those dirty attributes will get saved.

Also, Datamappify supports attribute lazy loading, all we have to do is to tell our entity to become lazy aware:

class User
  include Datamappify::Entity
  include Datamappify::Lazy
end

Once an entity is lazy aware, a repository will inject the lazy loading mechanism onto the entity when it retrieves such entity (via find).

Both repositories and entities support inheritance, again, you may read more about them in the project’s README.

Putting Things Together With A Controller

Remember we have the UserAccount for authentication? So how does that work? Well, here’s the piece of code in our application controller:

class ApplicationController < ActionController::Base
  before_filter :authenticate_user_account!

  private

  def current_user
    unless current_user_account.blank?
      @current_user ||= UserRepository.find(current_user_account.user_id)
    end
  end

  helper_method :current_user
end

Not too different from the normal Devise workflow hey? ;)

Forms

Web applications typically have lots of forms - ours is no different. It turns out, Datamappify can help build form objects too!

Here’s the view portion of one of our forms (we use Slim templates):

# our standard form layout

= simple_form_for submission_target, url: submission_path, signed: true do |f|

  = yield(f)

  .form-actions
    .pull-left
      - if archivable? && resource.persisted?
        = link_to 'Archive', '#', method: :delete, class: 'btn-archive'

    = f.submit 'Save'
    = link_to 'Cancel', cancel_path

# actual form content

= render layout: 'forms/resource' do |f|
  h3 Account Details
  .row-fluid
    .span4 = f.input :account_type, collection: BankAccount::ACCOUNT_TYPES.invert
    .span4 = f.input :account_code
    .span4 = f.input :account_name
  .row-fluid
    .span4 = f.input :account_number
    .span4 = f.input :bank_name
    .span4 = f.input :bank_branch
  .row-fluid
    .span4 = f.input :bank_bsb, label: "Bank BSB"
    .span4 = f.input :activated, as: :boolean

  h3 Branch Details
  .row-fluid
    .span4 = f.input :branch_address_line_1, label: "Address 1"
    .span4 = f.input :branch_address_line_2, label: "Address 2"
    .span4 = f.input :branch_address_line_3, label: "Address 3"
  .row-fluid
    .span4 = f.input :branch_state, label: "State"
    .span4 = f.input :branch_postcode, label: "Postcode"
    .span4
      = f.input :branch_country_code, label: "Country" do
        = f.country_select :branch_country_code, ['au', 'us', 'gb'], value: 'au'
      - f.add_signed_fields :branch_country_code
  .row-fluid
    .span4 = f.input :branch_phone_number, label: "Phone number"
    .span4 = f.input :branch_fax_number, label: "Fax number"

  h3 Merchant Details
  .row-fluid
    .span4 = f.input :merchant_number
    .span4 = f.input :merchant_name
    .span4 = f.input :merchant_address
  .row-fluid
    .span4 = f.input :merchant_biller_code, label: "Biller code"
    .span4 = f.input :merchant_bpay_method, collection: Merchant::BPAY_METHODS.invert, label: "BPAY method"

  h3 Payment Details
  .row-fluid
    .span4 = f.input :payment_number
    .span4 = f.input :payment_merchant, label: "Merchant description"
    .span4 = f.input :payment_reference_type, collection: PaymentAccount::REFERENCE_TYPES.invert, label: 'Reference type'

As you can see, it’s a form containing details from bank account, branch address, branch contact information, merchant and payment. And indeed we do have bank_accounts, addresses, contacts, merchants and payment_accounts tables in our database. Yet, the form still remains flat-structured and submits via simple_form_for.

This is thanks to our BankAccount entity and BankAccountRepository repository:

class BankAccount
  include Datamappify::Entity
  include Concerns::Archivable

  ACCOUNT_TYPES = {
    'general' => 'General',
    'trust' => 'Trust'
  }

  attribute :account_type, String
  attribute :account_code, String
  attribute :account_name, String
  attribute :account_number, String
  attribute :bank_name, String
  attribute :bank_branch, String
  attribute :bank_bsb, String
  attribute :activated, Boolean, default: true

  attributes_from Address, prefix_with: :branch
  attributes_from Contact, prefix_with: :branch
  attributes_from Merchant, prefix_with: :merchant
  attributes_from PaymentAccount, prefix_with: :payment

  references :agency

  validates :account_type, Mos::Validation.in(ACCOUNT_TYPES)
  validates :account_code, Mos::Validation::STANDARD_TEXT
  validates :account_name, Mos::Validation::STANDARD_TEXT
  validates :account_number, Mos::Validation::STANDARD_TEXT
  validates :bank_name, Mos::Validation::STANDARD_TEXT
  validates :bank_branch, Mos::Validation::STANDARD_TEXT_OPTIONAL
  validates :bank_bsb, Mos::Validation::BSB
  validates :merchant_bpay_method, Mos::Validation.presence_depend_on(:merchant_biller_code)
  validates :payment_reference_type, Mos::Validation.presence_depend_on(:payment_number)
end

class BankAccountRepository
  include Datamappify::Repository

  for_entity BankAccount

  default_provider :ActiveRecord

  map_attribute :branch_address_line_1, 'ActiveRecord::Address#address_line_1'
  map_attribute :branch_address_line_2, 'ActiveRecord::Address#address_line_2'
  map_attribute :branch_address_line_3, 'ActiveRecord::Address#address_line_3'
  map_attribute :branch_state, 'ActiveRecord::Address#state'
  map_attribute :branch_postcode, 'ActiveRecord::Address#postcode'
  map_attribute :branch_country_code, 'ActiveRecord::Address#country_code'

  map_attribute :branch_phone_number, 'ActiveRecord::Contact#phone_number'
  map_attribute :branch_fax_number, 'ActiveRecord::Contact#fax_number'
  map_attribute :branch_email, 'ActiveRecord::Contact#email'
  map_attribute :branch_website, 'ActiveRecord::Contact#website'

  map_attribute :merchant_number, 'ActiveRecord::Merchant#number'
  map_attribute :merchant_name, 'ActiveRecord::Merchant#name'
  map_attribute :merchant_address, 'ActiveRecord::Merchant#address'
  map_attribute :merchant_biller_code, 'ActiveRecord::Merchant#biller_code'
  map_attribute :merchant_bpay_method, 'ActiveRecord::Merchant#bpay_method'

  map_attribute :payment_number, 'ActiveRecord::PaymentAccount#number'
  map_attribute :payment_merchant, 'ActiveRecord::PaymentAccount#merchant'
  map_attribute :payment_reference_type, 'ActiveRecord::PaymentAccount#reference_type'
end

Pretty clean and straightforward. What do you think?

Datamappify - Keeping Your Domain Layer Sane

Datamappify’s concept isn’t new, but there simply isn’t anything in the ruby community that solves everything Datamappify tries to solve.

I hope that this post has not only introduced you to Datamappify, but has also made you think about how to make your application’s domain layer more decoupled.

Thanks for reading, and if you have any feedback for Datamappify please get in touch with me!

]]>

Would you be interested in reading such a book?

2013-04-29T12:25:00.000000Z

Would you be interested in reading such a book?

]]>

ActiveRecord and DB Migration Ate My Model Attributes!

2013-02-27T00:12:00.000000Z

Update: You might also want to check out reset_column_information.

So a few days ago we started seeing the following errors on our Jenkins builds (swapped with fictional model and attribute names):

NoMethodError:
  undefined method `attack_power=' for #

attack_power is a new attribute we recently added to the Ironman ActiveRecord model.

I was baffled, as the table column is clearly there but ActiveRecord couldn’t see it.

This weird behaviour is confirmed by debugging the model:

Ironman
# => Ironman(id: integer, created_at: datetime, updated_at: datetime)

Ironman.column_names
# => ["id", "created_at", "updated_at"]

And by debugging the schema:

ActiveRecord::Base.connection.structure_dump
# => "CREATE TABLE `ironmans` (\n `id` int(11) NOT NULL AUTO_INCREMENT,\n `created_at` datetime NOT NULL,\n `updated_at` datetime NOT NULL,\n `attack_power` int(11) NOT NULL,\n PRIMARY KEY (`id`)\n) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;\n\n"

So apparently somehow ActiveRecord ate our attack_power attribute!

After some struggling and head-scratching, finally I’ve discovered that during the migration if we call the Ironman class, then any subsequent attribute changes to the class will not be recognised by ActiveRecord.

It turns out, because we run our test suites by invoking rake tasks:

Rake::Task['spec:something'].invoke
Rake::Task['spec:something_else'].invoke

And we manually reset our database beforehand too within the same Rakefile:

Rake::Task['db:reset'].invoke

ActiveRecord will cache its attributes as soon as a model class (Ironman) is called during the migration.

The fix? Simple! Simply use another process to run the database reset:

system 'rake db:reset'

]]>

"Become A Better Developer You Can" - Video of My RubyConf China 2012 Talk

2012-12-03T11:44:00.000000Z

It was the first time I presented in front of 400+ people so I was really nervous - I blasted through the talk in under 30 minutes even though I was supposed to talk for 45 minutes, oh well. ;)

video: https://www.youtube.com/embed/DeBsmdDmB9A

Also, my slides are available too.

P.S. If you’re located in China, you may view the talk at Railscasts China.

]]>

An Interview with Yukihiro "Matz" Matsumoto

2012-11-25T06:45:00.000000Z

Update: Please check out this new interview with Matz, done by Engineer Type.

A week ago I went to Shanghai, China to attend and to give a talk at RubyConf China. The day before the conference’s first day a bunch of us were invited to a VIP dinner where we met with Matz and got to play with a device running MRuby. And I heard that earlier on that day Matz was ‘adopted’ by a book publisher to do an interview.

I have found the interview (in Chinese), and found it to be really useful. So I translated it to English. Hope more people will like it. :)

On November 16th, 2012 - a day before RubyConf China, a Chinese book publisher Turing Book has done an interview with Matz on his new book The Future of Computing as well as about a few topics interested to Chinese readers. The interview was conducted by _The Future of Computing_’s Chinese translator Zi Heng Zhou.

Zhou: Mr. Matsumoto’s new book The Future of Computing is published earlier this year and the Chinese version is being translated by yours truly, and that is going to be released some time next year in China. Your last book The World of Code has received high praises amongst the readers in China, so what are the differences between the last and the new book?

Matz: The World of Code has 14 topics in total, and each topic covers the basics - they covered more breadth than depth. The new book on the other hand has one set topic - thoughts on the emerging technologies in the future, therefore the ground covered would be narrower and more in depth than the last book. On top of that, the new book discusses several things by timescale, such as the history and the changes since the invention of computing, and computing’s impact on our future lives. Therefore it’s the thoughts of both the past and the future. The computing world is changing rapidly, and this book’s purpose is to discuss the direction of computing heading into the future.

Zhou: Speaking of the history of computing, you have touched on a few things about Moore’s law in the book?

Matz: Moore’s law describes the rule of changes over the history of computing hardware. The book discusses not only the changes on computing itself, but also the changes on its surrounding environment.

Zhou: On the topic of the evolution of programming languages, Paul Graham said in The Hundred-Year Language that the main branches of the evolutionary tree pass through the languages that have the smallest, cleanest cores. In the new book you seem to hold a different opinion, can you tell us why? And what’s your take on the evolution of programming languages?

Matz: Paul loves Lisp, and Lisp perfectly matches the characteristics of the programming languages described in his essay, and so Paul reckons the programming languages in a hundred years from now will look like Lisp. In reality though, Lisp has been around for over 50 years, and to be honest it isn’t one of the mainstream programming languages. In my opinion this may have been because most programmers don’t find Lisp charming enough. In other words, there is a gap between the so-called “smallest, cleanest cores”, “beautiful” languages and the expectation of programmers. It would be understandable if Lisp’s charm had not been accepted by everyone in a year or two, but for 50+ years it hasn’t reached the mainstream, could it be because it fundamentally does not match our expectation? There is a huge difference between human friendly languages and languages that have smallest, cleanest cores , and I am afraid the gap between them might not close even in a hundred years. As for what future programming languages should look like, I think they should have a runtime model similar to that of Lisp and be easily understandable by humans. All of a sudden, Ruby looks a lot closer to that, doesn’t it?

Zhou: Mr. Matsumoto you are the father of Ruby. We know that during the design of a programming language there might be a lot of different choices to be made, e.g. dynamically typed vs statically typed, prototype based vs class based, etc. When you were designing Ruby, what was the most difficult choice you made?

Matz: Before Ruby I actually have designed another language when I was in university, and that language was statically typed, similar to Eiffel. I liked statically typed languages, but the language I designed during university was more for academic purposes. Several years later, when I wanted to design a language as a tool I could use, I preferred to have it dynamically typed as I think they were more practical. So I designed Ruby. And I think it was the correct decision - it might not have been the most difficult decision, but it certainly was the biggest decision. Now that the decision of being a dynamic language is made, languages such as Smalltalk, Lisp and to an extent Perl have all had influences on Ruby. One of the features of Ruby is “mixins”, and mixins were not very common at the time Ruby was created. But because I don’t like multiple inheritance, I always believed that there must be an easier way to achieve similar results, so I designed mixins in Ruby.

Zhou: Looking back, is there anything in Ruby you wish you did differently?

Matz: In the beginning my goal was to replace Perl as my tool therefore I have borrowed many ideas from Perl, such as using the dollar sign ($) to indicate special variables. Looking at it now it seems to be a little bit too much and too similar to Perl. There are a few other things but mainly I think it is too similar to Perl. Back then before the ruby idioms were formed, there were many things that were borrowed from Perl - nowadays I think many of them weren’t necessary thanks to the ruby and rails idioms.

Zhou: Many believe Ruby’s popularity was because of Ruby on Rails, and you agreed with this in the book. So what do you think made Rails so successful?

Matz: The first and foremost, it benefitted from the rapid growth of the web - almost every software development platform is eyeing on the web field. What was developed using the traditional Client-Server architecture can now be implemented on the web. The number of applications that can be developed for the web has grown, and that’s important. Secondly, Ruby is optimised for easier development and higher development productivity. I think these two reasons combined, is what made Rails so successful.

Also, Ruby has many powerful features such as meta-programming and extendability via monkey patching. Through these features, basic classes can be enhanced, and DHH created Rails using these powerful features of Ruby. For people who have never touched Ruby, for example those Java programmers who like “inflexible” languages, they will be like, “Huh? You can do that?” - and I think that’s also one of the reasons why Rails is so popular.

Zhou: It is said that DHH was going to create a web framework using PHP, but eventually moved to Ruby?

Matz: Yes, and I think the reason for that might be the limitation of meta-programming in PHP. After Rails was released there were a bunch of PHP frameworks came out that are inspired by Rails, such as Symfony and CakePHP. PHP however does not have many of the powerful features found in Ruby, and purely from the development perspective I still believe Rails is the more powerful one out of the bunch. By the way, I actually met with DHH before, in Denmark. Back then he hadn’t started learning Ruby yet, perhaps that had a small impact on him too.

Zhou: Ruby is being used widely for the web, but one hot topic amongst Chinese readers is the development direction of Ruby outside of the web. Any thoughts?

Matz: True, many web based projects use Ruby, for instance web frameworks like Rails and Sinatra, etc. The world of programming however is far beyond simply web, and I have always wanted Ruby to break out from the web to other areas. In the near future, I would like to see Ruby being used in three fields:

1.

Scientific computing. In this field, languages like Python, R and Matlab are popular choices. I hope Ruby will soon be one of them. In fact, we are developing a project called SciRuby, and we hope it will help us with the adoption in scientific computing.

2.

High performance computing. This is somewhat related to scientific computing - to use super computers for doing computational work. Compared to C++, Ruby indeed is very slow, and that’s why people think Ruby isn’t suitable for doing high performance computing. In University of Tokyo a research student is working on an academic research project that compiles Ruby code to C code before compiling the binary code. The process involves techniques such as type inference, and in optimal scenarios the speed could reach up to 90% of typical hand-written C code. So far there is only a paper published, no open source code yet, but I’m hoping next year everything will be revealed.

3.

Embedded systems. For micro devices such as cell phones, medical equipment and robots, Ruby isn’t really suitable due to its memory use and APIs. Therefore we need something that’s more suitable for these embedded systems where memory consumption has to be low and APIs have to be optimised. And that’s what MRuby is designed for.

Zhou: Twitter is mainly built using Rails, but I read a news recently about the traffic surge during the US president election period and Twitter is now migrating to other platforms to help with the scaling. What’s your view on this?

Matz: There are many reasons I believe. First of all, nobody could have predicted Twitter’s huge success - who would’ve thought a service that essentially provides 140 characters of text for blogging to become one of the largest social networks. During their rapid growth they have added a lot of new features, and I think Ruby has contributed to that - it allowed many new features to be thought and implemented in very short amount of time. Twitter could not have anticipated the traffic it is now handling, so it’s fair to say that they have now hit the limit of their architecture designed from way back.

To solve the issue, Twitter needs to design a new architecture from the ground up in order to handle the ever increasing traffic. Although, Ruby can still be used for rewriting the architecture (chuckles). More seriously though, when a platform hits its limit, there are a few ways of solving it such as rewriting the architecture and changing to another language/platform, etc. I believe the most effective way is to rewrite the architecture, and this is exactly what Twitter’s been doing. During the rewriting process the Twitter engineers wanted to take on more challenges so they picked Scala. Because Scala is a compiled language it has great performance, so it is a fine choice for the new architecture.

My opinion is that when your system is still in its growing stage, it is far more important to have the ability to react quickly to changes, and that’s what a highly flexible language such as Ruby offers. Once your system reaches to a point of maturity, stability and success, then to have a new architecture that saves on resources makes sense. Twitter only chose to use Scala for its core components, the web front-end and many of their internal tools are still using Ruby. As a matter of fact, I paid Twitter a visit last month and talked to many of their engineers there - Ruby is still in great use (chuckles).

Zhou: Mobile platforms become hotter and hotter in recent years due the increased use of smartphones, tablets and other mobile devices. As far as programming language is concerned, Android uses Java and iOS uses Objective-C. What about scripting languages like Ruby, how do you see them fit in this picture?

Matz: It is no doubt that so far in order to develop for Android you use Java, and for iOS you use Objective-C. But that also creates a huge barrier because you would need to rewrite your application if you wanted to port from one platform to another. Projects like PhoneGap and Titanium try to solve this very issue, via languages such as JavaScript and Lua to offer cross-platform compatibility. For Ruby, there is a project called Rhodes - you can use Ruby to write applications that will run on iOS, Android and Blackberry.

In the old days, mobile devices are much less powerful than PCs, and if applications can’t run in full speed then they are useless. Nowadays though, mobile devices have gotten significantly faster, and the use of cross-platform frameworks becomes far more practical than ever before. I think this way of developing for mobile platforms will become more and more popular. And by the way, we spoke about MRuby before, and using MRuby for iOS and Android is already underway, hopefully more applications will get developed using MRuby in the near future.

Zhou: You mentioned MRuby, and the topic of your keynote presentation tomorrow is Ruby 2.0. Could you talk about MRuby and Ruby 2.0’s highlights a little bit?

Matz: As we discussed earlier, MRuby is designed to run on embedded devices, it does not have everything from Ruby. As a result, many devices which traditionally can’t use Ruby, such as vendor machines, controllers and robots can soon utilise MRuby. Perhaps in a few years time, we could see Ruby used in televisions and cars.

Ruby 2.0 indicates two messages. Firstly, by next year it will be 20 years since the development of Ruby started. What better version number can there be to celebrate this special moment! Contrary to what it might seem like, in terms of the changes, from 1.9 to 2.0 is not as big as from 1.8 to 1.9. In fact, our goal for 2.0 is to be fully backward compatible with 1.9. Your 1.9 applications should run just fine on 2.0.

Secondly, Ruby 2.0 does offer a few new features. For instance, use monkey patching to add new or replace features has global effects and it might cause conflicts and bugs. In order to avoid situations like this, Ruby 2.0 limits the scope of monkey patching via Refinements. As bigger teams and bigger projects are adopting Ruby, scoped monkey patching will increasingly show its importance. There are many other features in Ruby 2.0 that are designed to cater for bigger teams and projects.

Zhou: As far as I know most of the mainstream programming languages are from America and Europe - though there are Lua from Brazil and Ruby from Japan. You too mentioned this in your book and you said this feels “lonely”. So what is the cause of this, and what can we do about it?

Matz: Well, for Lua you can include it in Europe/America too because Brazil is part of South America (chuckles). In the south eastern Asia region though there is only Ruby, and it is lonely. Europe and America still remain the most powerful regions as far as programming languages go. Asia, although has massive population, does not compete in this regard, that indeed feels lonely.

I am not sure about other countries, but at least in Japan there are many people working on programming languages, unfortunately other than Ruby none of them are well known. If more people are interested in programming and designing programming languages, there bound to be one or two that’ll break out, right? There is another hurdle in Japan - language. Most Japanese people only speak Japanese and they cannot speak English well. Funnily enough there are programming languages written entirely in Japanese. ( Zhou: In China there also are programming languages written entirely in Chinese.) In China too? I knew it! No matter how interesting these programming languages are, they will never influence anyone beyond the ones in their own country.

On a side note, I once received an email from an American. He said that you are Japanese, but Ruby looks like English because it’s written in English, why isn’t there any Japanese-written languages? I replied saying that there are, you just don’t know them, and even if you did, you wouldn’t be able to use them.

In Japan, more and more people are interested in programming, maybe because both online and in my books I always talk about how fun programming can be. Many people are now taking on the challenge of designing new programming languages. Out of these new languages, even only 0.1% of them ever get any success, I think it’s a win. I don’t know how many people want to take on the same challenge in China, Korea and other countries in Asia, but if people could look beyond programming languages are created for us, we just passively accept them, and think to create a new programming language can also be fun, then I am sure some of them will succeed.

Talking about open source projects, not many of them are from Japan, China and Korea, and I think this could be an entry point for many. There are many reasons why this is the case though, for example English is hard to learn… ( Zhou: And GitHub is also difficult to use?) Haha, is GitHub usable in China? ( Zhou: It is, it is…) Oh, that’s not too bad then. But, China’s Great Firewall still has a huge impact, many resources can’t be accessed here, right? ( Zhou: That’s right, for instance the Go programming language’s website is blocked.) Ah really? Is it because it’s made by Google? (Chuckles.) In any case, I think there are still many difficulties to face. Also, in Japan many programmers still spend most of the time at work (to put food on the table), it’s very difficult for them to contribute to open source projects. Ten years ago nobody cares about open source in Japan, but nowadays people start to realise the importance of open source, and the number of open source projects is growing. I believe China will soon follow this pattern as well, I am looking forward to it.

In the beginning no one knows what will succeed. When I started with Ruby I could not possibly have predicted its success. So I think for a programming language, timing is really important - and you’ll never know until you tried. I think in China there might also be languages that emerge from the right time that will eventually be a global success.

Zhou: In the book when you were talking about Dart, you mentioned that the ecological environment is really important for a programming language. So what do we need to do to ensure a good ecological environment?

Matz: From a user’s perspective, the most important thing is what benefits a programming language can offer. Before Rails came along, many Ruby users including myself, believe Ruby is a user-friendly language, and that’s the reason we like to use it. After Rails was born, more and more people like to use Ruby because making websites using Rails is really productive. So I think if you could communicate to your users what the benefits are by using your new programming language, it will help with its adoption and increase the chance of success.

Zhou: Thank you very much Matz! Do you have anything else to say to Chinese programmers?

Matz: In the book The World of Code I have mentioned that programming will likely to develop and evolve in the form of open source. New generations of programming languages and software are likely to emerge as open source projects. People are happy to freely use the software developed by others, and to take from that, and work on your own open source projects and make a little impact on the world - then you would become a world class software engineer. The software engineers I met in China are all very hard working and enjoying learning. I hope more of these people will step up and take on the challenge, and become world class programmers who will make a decent impact.

]]>

"Become A Better Developer You Can" - Slides of My RubyConf China 2012 Talk

2012-11-25T05:43:00.000000Z

Contrary to what the title of the talk suggests, I am by no means a superstar developer myself. Though I would like to think that I am extremely passionate about what I do, therefore I love to share my thoughts and tips on web development and software engineering.

Below is the version I used for my actual talk:

https://speakerdeck.com/fredwu/2012-become-a-better-developer-you-can

Since I had been preparing for my talk for over a month, I’ve done many revisions to my slides. Here’s an uncut version that contains every single slide since day one:

https://speakerdeck.com/fredwu/2012-uncut-become-a-better-developer-you-can

Hope you will enjoy them! :)

Update: The video of my talk is up! :)

]]>

Photos from RubyConf China 2012

2012-11-24T15:47:00.000000Z

I had such a wonderful time at RubyConf China! So here are some photos from the conference. :)

Gallery of photos of yours truly on stage:

Gallery of photos I took mostly on the VIP dinner event the day before the conference:

]]>

SimpleCov: Test Coverage for Changed Files Only

2012-11-13T07:34:00.000000Z

The other day a colleague asked whether or not it’s possible to have SimpleCov return a group that only contains uncommitted changes.

The answer is yes! After some digging around, we found the following way:

# in spec_helper.rb
SimpleCov.start 'rails' do
  add_group 'Changed' do |source_file|
    `git ls-files --exclude-standard --others \
      && git diff --name-only \
      && git diff --name-only --cached`.split("\n").detect do |filename|
      source_file.filename.ends_with?(filename)
    end
  end
end

Basically use git ls-files --exclude-standard --others for untracked files, git diff --name-only for unstaged files and git diff --name-only --cached for staged files.

]]>

Skinny Coffee Machine - A Simple State Machine with Observers

2012-09-06T11:47:00.000000Z

After a few nights of working on the 2.0 rewrite of jQuery Endless Scroll, I am now releasing one of the tools I built for the project: Skinny Coffee Machine.

Skinny Coffee Machine is a simple JavaScript state machine written in CoffeeScript.

It is fairly simple to use, with the flexibility of adding and removing observers for state transitions.

Define State Machines

@coffeeMachine.power = new SkinnyCoffeeMachine
  default: 'off'
  events:
    turnOn:
      off: 'on'
    turnOff:
      on: 'off'
  on:
    turnOn: (from, to) -> "#{from.toUpperCase()} to #{to.toUpperCase()}"
    turnOff: (from, to) -> "#{from.toUpperCase()} to #{to.toUpperCase()}"
  before:
    turnOff: (from, to) -> "Before switching to #{to.toUpperCase()}"
  after:
    turnOn: (from, to) -> "After switching to #{to.toUpperCase()}"
    turnOff: (from, to) -> "After switching to #{to.toUpperCase()}"

@coffeeMachine.mode = new SkinnyCoffeeMachine
  default: 'latte'
  events:
    next:
      latte: 'cappuccino'
      cappuccino: 'espresso'
      espresso: 'lungo'
      lungo: 'latte'
    last:
      latte: 'lungo'
      lungo: 'espresso'
      espresso: 'cappuccino'
      cappuccino: 'latte'

Switch/Change States

@coffeeMachine.power.currentState() #=> "off"
@coffeeMachine.power.switch('turnOn')
@coffeeMachine.power.currentState() #=> "on"

@coffeeMachine.mode.currentState() #=> "latte"
@coffeeMachine.mode.change('next', 3)
@coffeeMachine.mode.currentState() #=> "cappuccino"

Observers

@coffeeMachine.power.observeBefore('turnOn').start 'labelA', (from, to) => "Observer A before switching to #{to.toUpperCase()}"
@coffeeMachine.power.observeOn( 'turnOn').start 'labelB', (from, to) => "Observer B on switching to #{to.toUpperCase()}"
@coffeeMachine.power.observeAfter( 'turnOn').start 'labelC', (from, to) => "Observer C after switching to #{to.toUpperCase()}"

@coffeeMachine.power.observeBefore('turnOn').stop('labelA')

Go check out the source code now! :)

]]>

Brewing Node.js on OS X without Xcode

2012-08-25T14:43:00.000000Z

If you are like me who has no need for a full Xcode installation just to get the command line tools, chances are you are using one of these: Apple’s Command Line Tools or the osx-gcc-installer.

Recently Node.js has made some changes so that it no longer installs on OS X via homebrew if you don’t have Xcode installed.

If you run brew install nodejs, you will get the following error:

\> Error: Failed executing: make install (node.rb:28)

And if you run brew install -v nodejs, you will discover this line:

\> xcode-select: Error: No Xcode is selected. Use xcode-select -switch , or see the xcode-select manpage (man xcode-select) for further information.

The fix? It’s actually quite easy, simply do:

sudo xcode-select --switch /usr/bin

And voila! You can now install Node.js just fine. :)

]]>

Fix OpenSSL Error on Mountain Lion (and RVM)

2012-08-06T13:25:00.000000Z

Don’t you just hate it when you have a fresh intall of Mountain Lion, RVM and some rubies - then all of a sudden you hit this OpenSSL::SSL::SSLError error message:

SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed

The fix is quite simple actually, all you need to do is to download a CA root certificate:

curl http://curl.haxx.se/ca/cacert.pem -o ~/.rvm/usr/ssl/cert.pem

And that’s it! Enjoy!

]]>

API Taster: Visually Test Rails Application API

2012-07-02T04:59:00.000000Z

Like a lot of places, at Locomote we are building a platform that is API-based. As much as I like having comprehensive test suites, I often feel the need to manually test API endpoints to see exactly what the responses are.

Tools such as Postman solves part of the issue: they allow us to quickly test API endpoints without messing with cURL.

But as a lazy developer, I want more. ;)

I want something that:

automatically generates API endpoints from Rails routes definition
defines input params as easy as defining routes
has input params that can be shared with test factories

And so API Taster was born. Please check it out to see how you can use it.

]]>

PHP Devs: Have You Started Playing with More Toys?

2012-06-29T19:03:00.000000Z

TL;DR - PHP is still a useful tool, but as a PHP developer, have you started playing with other useful tools? Here’s my story.

Today Jeff Atwood’s new piece “The PHP Singularity“ and Marco Arment’s “PHP Addiction“ have started another round of heated discussion on PHP.

As someone who started his career as a PHP developer, I feel like sharing my thoughts from a different perspective.

I began my career as a freelancer - since JavaScript and PHP were the two programming languages I learnt at school, they naturally became my weapon of choice for freelancing work.

At that time I knew very little about software development. My desire was simple: I liked making websites, therefore I decided I was going to do it for a living.

Over the years I came across many people who shared a similar background - people who do not have a strong (or any) computer science / software engineering background, were making websites.

Compared to most software developers, the intention of doing programming for me was different. I had no intention (or awareness) to create maintainable software , instead I simply wanted to create an end product that works.

Was I wrong? According to the market, no. There are many clients who want affordable websites, just like there are many people who want affordable Android phones. On one end you have Android phones that compete with the iPhone and take time to get released, on the other end you have Android phones that use readily available, often less desirable components and are quick to market. Each serves its purpose and fulfils customer demands.

As I kept developing websites and educating myself, I became more and more interested in software development. That’s when I started reading books like “PHP Objects, Patterns, and Practice“. But even then, my knowledge was still extremely limited, and my desire at that time was to increase my PHP knowledge and become a better developer.

For the first desire I made it happen by reading more books and articles, but for the second desire - I thought I made it happen, but in reality I didn’t because I did not correctly identify my weakness. It wasn’t just about the depth of knowledge, but also the breadth.

Spending most of my time writing PHP made me too comfortable in a little corner. The discovery of tools such as CodeIgniter made me feel good about being a competent web developer. After all, I was doing quite a fine job churning out websites.

In 2006 I discovered Ruby on Rails, but that did not have too much impact on me, because I was not yet capable to tell the differences it has compared to PHP (frameworks).

In my mind, PHP and Ruby are both scripting languages that can be used for developing websites and web applications. Although the statement is true, not knowing the specifics made me very ignorant. It is almost like not knowing the difference between travelling from one continent to another by an airplane and by a ship - either will get you to the destination, right?

And this is exactly why my first ruby program looked just like a PHP program.

It was (and still is) an interesting and rewarding journey to slowly discover why my Ruby code sucked.

So I started reading on design patterns, software craftsmanship and other programming languages. And I was lucky enough to be surrounded by many brilliant developers who taught me how to write better code.

At that point, my intention for learning more about software development had shifted from simply making an end product, to making an end product that can be maintained.

And that changed everything. From a PHP developer who just wanted to churn out websites quickly, to a developer who wanted to use software development to create value - in the sense of both business value and technical value.

An important thing to remember is though, it’s not about Ruby , and it certainly is not about looking down on PHP. Rather, the point is to see whether or not there’s something out there that might enable you to look at things differently, and do things differently, for better or for worse.

PHP for some is a double-clawed hammer, but there is no doubt in my mind that for some others it is a near perfect tool, for now at least. Having said that, many other kids have started playing with all sorts of toys, why haven’t you? :)

]]>

[Rails Tip] Render views outside of Controllers or Views

2012-06-20T09:09:00.000000Z

Ever wondered how you could utilise the render method outside the context of Rails controllers and views? If you wonder why anyone would do that. Well, imagine you are building an awesome form builder, you need to output and/or store rendered partials in the buffer. How do you do that?

For example, what if you want to do this in your view?

<%=raw Awesome::FormBuilder.new(some_options).html %>

You could do something like this:

module Awesome
  class FormBuilder < AbstractController::Base
    include AbstractController::Rendering
    include ActionView::Context
    include ActionView::Helpers::CaptureHelper

    # set the view paths from your engine or from your application root, i.e. Rails.root
    self.view_paths = Awesome::Engine.root.join('app/views')

    def initialize(params)
      flush_output_buffer
      @_buffer = ''
      add_to_buffer(params)
    end

    def html
      @_buffer
    end

    private

    def add_to_buffer(params)
      # some logic to add rendered content to @_buffer
    end
  end
end

The idea is to mixin the render method, but also ensuring the view buffer is correctly reset with flush_output_buffer.

Hope that helps. :)

]]>

Agile is not a Sham

2012-03-28T11:35:00.000000Z

This blog post titled “Agile is a Sham“ offends me a little bit. The post screams hey, I am a cowboy programmer, and it almost implies that if you employ processes then you are stupid.

In this day and age, I would have thought finding the right tool for the right job is common sense. Apparently not.

Agile, among many other things, is not a silver bullet - it never was and it never will be. The key thing is to experiment and find what works for you, your team and your company. Most software projects are done in a team environment - putting a bunch of talented developers and designers together actually isn’t as simple as many seem to think. One example is right out of that original blog post:

Given that I am not a fan of prescribed process as the solution to our problems, you can infer my opinion of the general quality and effectiveness of those programmers who teach these courses.

From that attitude, I can already see that he is an interesting developer to manage.

A team needs to function efficiently as a team. Having one or two ace developers aren’t going to help a lot if they can’t get along with the others.

Not to mention that people have spent time and effort proving that techniques such as TDD work for certain projects.

As many things in life, there are always people who try to take advantage of others. No doubt there are questionable ‘agile consultants’ who try to make a fortune out of uninformed clients. Just because some people aren’t using agile effectively or even correctly, doesn’t mean agile itself is a sham. Developers constantly make mistakes and write shitty code - should we say all the programming languages out there are a sham?

Keep an open mind is important, and a lot of times even necessary. To dismiss agile all together is in my opinion childish, and offensive to others who try to improve things and create more value.

]]>

Zend Framework - From Extreme Simplicity to Enterprise!

2012-03-25T12:09:00.000000Z

Today an interesting blog post on Zend Framework has made to the Hacker News front page. I have to agree with the author - Zend Framework is an over-engineered piece of software.

A few years ago I was working full time as a PHP developer. Naturally, I had experimented with lots of frameworks, including CakePHP, CodeIgniter, Kohana, Yii, Symfony and obviously, Zend Framework.

Over the years, I had developed many projects primarily using CodeIgniter and later on, Kohana. There was one project that was larger in scale compared to an average PHP project (whatever that means), so I decided to use Zend Framework to take advantage of its component-based structure.

The result? It was dreadful.

At that time there wasn’t any built-in or quick n’ easy way of bootstrapping a Zend Framework project, so we had to cook up our own. The documentation was absolutely horrible despite its completeness viewed from a distance - to find anything useful, Google is more likely to provide you with the answer, even from the results on the documentation site. And of course, the framework felt so heavy and long-winded. It, felt, Java.

In terms of the code structure, Zend Framework is almost the complete opposite of CakePHP - no offense to the CakePHP developers and users, but if you take a look at each framework’s source code, you’ll know what I mean.

Having layers and layers of structures might appeal to certain users (Enterprise baby!), but does anyone still remember back in the day, how Zend was announcing/marketing the Zend Framework?

Extreme simplicity!

That’s right, Zend Framework was marketed as an extremely simple to use web framework. In fact, we can still read Andi Gutsmans’ post on this topic. Let me quote the relevant part (emphasis mine):

As some know, our ultimate goal for the framework is what we call “ Extreme Simplicity “. Some might have heard me use that term in some of the talks regarding PHP, and I’d like to bring this concept over to the framework. I believe this is what the PHP spirit is all about and the idea behind it is that it’s possible to create very simple & easy-to-use languages&frameworks which still remain powerful and flexible.

Can anyone, including the Zend Framework developers, honestly say that Zend Framework is an extremely easy to use framework?

I guess at the end of the day, in order to market PHP and Zend Framework as ‘enterprise-ready’ with trainings and certifications, something has to give.

]]>

On Hiring: Use Kanban for Managing Candidates and the Hiring Process

2012-02-21T04:20:00.000000Z

Hiring is usually a long and difficult process - in order to streamline and simply it, I use Kanban to manage the whole process. I believe, hiring should be as lean and agile as our development process.

My Kanban board for hiring (we just started hiring at SitePoint!):

Using a Kanban board offers a number of advantages:

a clear picture of the candidates with their feedback from code tests and interviews
a straightforward view of where a candidate is at in the hiring process
limited number of candidates in some stages to prevent chaos
visual reminders to get in touch with the candidates, it’s always a good idea to keep them in the loop
a tight WIP limit for shortlisted candidates, there’s no point to shortlist too many candidates

What do you think?

]]>

On Hiring: How To Be a Non-Technical Co-Founder

2012-01-28T06:16:00.000000Z

If you are looking at hiring developers, check out my article on this subject.

The goal or the dream of working on your own startup is always full of excitement. And apart from some rare cases such as Dropbox, you probably need one or more co-founders to work with you on The Next Big Thing ™.

Problem is, how do you (as a non-technical co-founder) find us? Or more specifically, how do you talk us into working with you instead of some other billion-dollar ideas?

To answer this question, we need to first ask, is there a billion-dollar idea? The short answer is: NO.

Idea is worthless.

Well, that’s not entirely true. I believe - idea, by itself, is worthless.

You will be surprised by the number of people contacting us and wanting to build a better Paypal or a better Amazon, without a concrete plan.

A more worthwhile idea should contain not only the end goal of the project, but also a plan to reach the goal. What should we ship in the Minimal Viable Product? What are our marketing channels? What metrics should we look at? How do we use social media to our advantage? etc, etc.

We Are Not Just Wozniak, Are You Like Steve Jobs?

Apple is pretty much started by Wozniak as the technical co-founder and Steve Jobs as the idea/business co-founder.

Let’s think about this for a second.

Steve Jobs did not just have ideas. Very early on, he persuaded Wozniak to produce and sell Apple I so they have some capital. Jobs was building the foundation. Without the foundation, there will be no failure or success to come.

On the other hand, Wozniak had no intention to become an entrepreneur, he was happy to stay as an engineer even after the early Apple success. Nowadays though, most of us techies are much more ambitious than that.

Ideally, as the technical co-founder, I would be doing most of Wozniak’s work, and both you and I would be doing Steve Job’s work.

Drawing from my personal experience, as a technical person, there are a few key attributes I look for in a co-founder (technical or otherwise).

Technical Ability

“Excuse me? Aren’t you the technical co-founder? Why are you looking for my technical ability?” You ask.

That is right. Even if you are not a developer by trade, having a certain degree of understanding of technologies is still crucial to most modern, web-based projects.

There has never been a better time to start learning to code. Why not give CodeYear and Khan Academy a try?

We all learnt physics and chemistry in high school even though most of us don’t require the knowledge in our day to day life. Let’s treat coding the same. Learn how to code will not only give you insights to how we solve problems, but will also close the communication gap between you and your technical co-founder.

Obsession

Wozniak is obsessed with electrical engineering and gadgets, Steve Jobs was obsessed with computer typefaces, good user experience and beautiful hardware.

What are you obsessed with?

Only when you are obsessed with something, can you answer questions like “what annoys you so much?”

As I wrote in an ealier article:

Inventions and innovations aren’t born out of happiness, they are born out of frustration, anger and sometimes, curiosity.

Curiosity

In web-based projects, it is surprisingly easy to have “what if …?” scenarios. Not sure which sign up form will have a higher conversion rate? Easy, just make two or more of them and run A/B tests.

Sometimes, as developers, we are so in the zone that we would keep on building stuff the way we envisioned. You will need to step in, pull us out, and say “hey, have you thought about …? What if …?”

Flickr as it is today would never have existed if the founders didn’t raise the question of “hey, how about doing just the photo uploading and sharing features?”

High Expectation

“This is shit!” “We can’t ship this!” If the product stinks, say so, and find ways to improve it. An MVP should always be half-polished, not half-arsed.

The original iPhone was shipped without 3rd party native apps support, or multi-tasking - it wasn’t ideal, but they didn’t effect the core user experience. Now look at PlayBook, it has the features most Android devices have, but the core user experience is so bad that the product never took off. If someone at RIM’s top management had the same obsession on user experience as Steve Job’s, PlayBook would never have shipped in such a bad shape.

Passion

Are you in this for the money? Or for something else? Wealth is rarely a good motivation for creating great products.

“It can potentially generate massive revenue and profit” is a big red flag to me when someone pitches their projects.

These are the key attributes I look for. Things like people connections and experience are also important but not essential. What about you? Do you look for any particular attributes in your potential co-founder(s)?

]]>

On Hiring: How Not to Annoy Developers

2012-01-26T05:42:00.000000Z

If you are looking at finding technical co-founders, check out my article on this subject.

In recent years developers become hotter and hotter - especially the good ones - they are hard to find, and they have plenty of employment options to choose from.

Some companies (or individuals who are seeking freelancers) go the extra miles to impress developers with attractive salary/rate and perks, which is nice. But surprisingly, many companies and individuals seem to have a habit of keep doing things that will annoy developers.

Over the years, I have personally encountered many situations that annoyed me as a developer. If you are looking for developers, here are a few things that I think you should be aware of.

Budget Gap

Find out what the developer’s salary/rate expectation is, there is no point if the developer is looking for $120k when you could only afford $80k. It’s absurd to think that you might be able to persuade them with a significant pay cut.

Timing of Hiring

Don’t advertise the position until you are fully ready to hire. Put a job ad up then go on holidays immediately after is a bizarre thing to do.

Recruiters

If you use recruitment agencies, please make sure you know them well enough and if not, do your fair share of research on them. Bad recruitment practices will cost you and damage your reputation. I once dealt with a recruiter who did not even bother to pass along my response to the offer back to the employer.

Technical Assessment

Unless you are Google or Facebook who have a huge pool of candidates to filter, it is probably a bad idea to test your candidate’s technical skills with brainfuck coding questions on paper or on whiteboards.

I am in favour of having candidates to do one or two coding tests (on their own, not during the interview) that will demonstrate their technical abilities without spending too much time or be under interview pressure. If a question is going to take a candidate more than a couple of hours to do, it is too much.

The technical questions should focus on the logic behind a candidate’s solutions, not what functions or libraries a candidate may or may not remember from the API documentation.

Also, pair programming is an excellent way for both parties to get a sense of what it is like to work together.

Communication Channel

This applies more to hiring remote workers and freelancers. If you are pitching your project to a developer, please at the very least write a sentence or two to explain what your project is.

“Would you have time to have a quick chat?” is simply not good enough - good developers are all very busy and are likely to have many projects or potential projects to work on. Even if the developers have the time to chat to you (without knowing what your project is), it is still better to have the conversation via emails so the developers can keep track of things.

Developer Relationship

Recently there is a company in Melbourne who burnt a huge bridge with some of the most respected and talented local developers. Now they are having trouble finding talents even with inflated salary/rate. Moral of the story is, please always treat developers properly.

So, these are the few things I can think of for now. How about you? I would love to hear your thoughts and stories on hiring developers.

Hacker News comments here.

]]>

The Lean Startup - The Book Every Entrepreneur Should Read

2011-12-31T08:18:19.000000Z

Holidays period is the perfect time to gear up and learn a thing or two from the masters - and as it turned out, reading Eric Ries’ The Lean Startup is one of the most exciting and joyful things I’ve done during the holidays.

In this book you won’t find long-winded and boring theories, instead the book is full of real world use cases and practical advices.

If you are an entrepreneur, or if you are responsible for product development, I urge you to read this book if you haven’t already. :)

]]>

Blog Redesigned For 2012, And New Challenges Ahead

2011-12-26T18:28:29.000000Z

After four days of sketching, designing and cutting up HTML/CSS, the new design (as you are seeing now) is finally live!

The new design is structurally similar to the old design, but with a fresh header and better use of space.

A new year warrants a fresh start. Apart from the redesign, I have also started heading up the development effort at SitePoint - dozens of interesting and challenging projects ahead!

2012 will be an awesome year! :)

]]>

Startup, VC, and the Things I Learnt from Open-sourcing A 200+ Hour Client Project

2011-09-02T12:49:00.000000Z

A couple of days ago, a blog post titled “First employee of startup? You are probably getting screwed!” has made to the Hacker News front page and has spawned some great discussion and debate.

Since I had just been screwed recently, I thought it might be a good opportunity for me to share what I’ve learnt.

After my story got picked up by Hacker News and some other sites, I was contacted by my client. It was a very interesting and surreal experience which included keywords such as sue, settle, donation and disappearance. I am however going to spare you the details and instead, going to focus on the things I have learnt.

Be Strong, Be Emotionless

The biggest mistake I made, was being too emotional - not in the sense of being emotionally attached to the project, but in the sense of being emotionally attached to the opportunity.

Despite having detected some negative signs (which I will cover later), I willingly chose to continue on with the project because I wanted it to work out. I wanted to be involved early in a startup, I wanted to create a product that will have large impact, I wanted to have uncertainties and excitement, and I wanted to force myself to be busy.

I was blindsided because of my desires - even though deep in my heart, I already felt the failure approaching.

Be Informed Who You Are Dealing With

My client, who was referred to me by a strong and reputable PHP developer, is well educated, has an MBA, and has worked as a VP of a major US corporation before he decided to start his own startup.

But didn’t that just prove doing background-checking is useless, you might wonder? No. It is still a useful exercise - it proves the fact that, even legit clients can do dodgy things. Therefore you would want to pay more attention to picking up negative signs rather than being impressed by their reputation and/or portfolio.

Be Alert, Pick Up Tells

Just like playing poker, you need to stay alert and try to pick up tells. What are the tells? Well, in my client’s case:

He often emphasise the importance of communication, but he communicated poorly via emails.
He had unrealistic deadlines.
He was overly optimistic about the success of his startup.
He tried to lower my compensation expectation by using poor arguments.
He would call me up and talk for 20+ minutes when he was fully aware that I was at work (at my day job).
He was not being transparent and upfront about the VC side of things.

The list goes on.

No one is perfect, of course. A minor tell or two might just be considered quirks, but half a dozen or more - you run.

Be Informed About Market Rates

Before negotiating my rates, I had done quite a lot of research online - it provided me with context.

As a non-full time, principal developer and designer, my compensation package was approximately:

An hourly rate that is equivalent to 50% of my full time salary, or 30% of my contract/freelance rate.
2% of the company, after the seed round.

In return, a 20-40 hours per week sweat investment was expected from me. To me, it was not a bad deal since I don’t rely on the success of the startup in order to feed myself - I still have my day job.

One thing to note, is that the agreed package was actually a result of me being too emotionally invested in the opportunity. The offer from my client was originally:

An hourly rate that is equivalent to 90%+ of my full time salary, or 50% of my contract/freelance rate.
1% of the company, after the seed round.

See what I did there? I was emotional enough to trade immediate return for potential future return.

One of the best advices I got from more experienced people in this field is that, always treat equities (or employee options) as a bonus that is likely to never happen.

Be Cautious About the Information Received

Until written on paper and signed, any information you were told should be taken with a grain of salt. Often than not, people would say things to favour and/or support their cases.

The $2mil seed around valuation figure that was told (but was never proved) by my client has indeed influenced my decision of trading my hourly rate for equity.

Be Positive, No Matter What Happens

Positive thinking - one of the most powerful attitudes to ensure your happiness, is the key and morale of the story. You see, even though I went through a lot of stressful days and nights, I still consider the whole experience being positive.

Why? Because I learnt a lot. I learnt many things that I would not have learnt from working at my day job or freelancing. It gives me more confidence for my next startup adventure (whether it’s for someone else, or for myself).

Be Adventurous

Don’t be scared by other people’s experience. Steve Jobs once said something along the line of - they started Apple because they didn’t know any better.

I can comfortably say that, I am very much looking forward to my next adventure! :)

]]>

Skype.com - A Quick Example of UX Failure

2011-08-31T08:46:00.000000Z

Today I noticed that I don’t have Skype installed, so naturally, I went to Skype.com. Then I was presented with their homepage:

The problem? No “Download” action button above the fold, or below the fold for that matter. That is, quite frankly, shocking.

So I hovered on the “Get Skype” drop down menu and clicked on the one for Mac. On the new page I was presented with, I clicked on the “Download Skype” button. And then …

Oh snap! You’d have to either create an account or sign in before you could download Skype! Worse, by default it shows you the create an account section, and that section took a good 10 seconds to load for me for the first time.

Is this some kind of prank from the Microsoft enterprise? *shakes head*

]]>

Open-sourcing A 200+ Hour Project - The Story Behind It

2011-08-17T11:55:00.000000Z

Update: I have blogged about the things I have learnt from this.

Last night, after some long overdue frustration, I have open sourced Angel Nest - an online platform for connecting entrepreneurs and investors, similar to AngelList.

Shortly after I open sourced my work, the news got picked up by Hacker News and went onto its front page. (Update: I was just informed that the story was picked up by Reddit as well.) The Github repository has since been followed by 250+ people and forked by 60+ times - it became one of the Github’s daily trending repositories.

Since then I have received many warm regards from fellow developers and entrepreneurs. Thank you so much guys! I will reply to each one of you as soon as I have a chance. :)

From the Hacker News comments as well as some of the comments received privately, there were a few questions raised, and I hope to address some of them in this blog post.

Why didn’t you charge upfront or by milestone?

Well, because it wasn’t a simple freelance or contract job. I was approached and asked to become the then to-be-set-up company’s first employee (yes, an employee, not a technical co-founder). At the time of our work agreement negotiation, the company was still to be formed and the investors were still to be signed on - no term sheets or contracts have been signed by any of the investors.

As a result, I agreed to put the invoices on hold for a while (i.e. a couple of weeks, according to him), then I will start invoicing the company fortnightly.

Oh, and many of you think I am a freelancer. Well, yes and no, I freelance, but my income is mostly from my full time job (as a web developer, of course). :)

Still, why did you trust him so easily?

Before I agreed to work on the project, I did some background-checking of this so-called entrepreneur - everything seemed legit - he was a developer himself a few years ago; he worked as a VP in a major corporation in the States before committing himself to this project; and from the few Skype conversations we had, he seemed to understand how to keep a developer happy. As a matter of fact, he had stressed many times that he would pay the developers a fair deal in order to keep them happy.

And how could you have agreed to work for him without any contracts?

Being a person with goodwill, I agreed to work for him on this project - essentially as the principal developer. My contribution was supposed to be compensated by a low hourly rate + equity in the company - the figures were agreed both verbally and via written Email.

Ordinarily I value equity very little. However, the way he described the investment situation - the company being valued at $2M, with investment from several top VCs and angels in China - got me believed that the company had a very strong lineup of investors and a reasonably large sum of first round investment.

To this day I still do not know what the actual investment situation is. It seems to be a sensitive topic whenever I brought it up during our conversations.

Why didn’t you stop working sooner?

I wanted to be professional - I cared more about building an awesome product than receiving a fat cheque (not that the cheque will be fat anyway, given the low hourly rate).

He stressed many times that the project is very time sensitive and we needed to launch as soon as possible.

And in order to speed up the development progress, I even took two weeks off work (with many unpaid leave days) and worked on this project. Looking back, I lost not only the compensation I am entitled to, but also a fraction of the salary that puts food on my table.

Are you upset?

Actually, not really. As I said before, I care more about building fine products than receiving more money - just the fact that I worked on a real Rails 3.1 project is very fulfilling in itself. And the fact that I received so many warm regards and comments - I am :) rather than :(.

As for the non-paying client, well, this isn’t my first time, and I usually just choose to forget about them and move on. I have too many more important things to worry about than chasing after payments.

Can I donate?

I sincerely thank you for your offer, but no I do not need donation.

What about the intellectual property? Any NDA?

We have no contracts or any paper signed whatsoever, despite I requesting them repeatedly - hence why I could release the source code at my will.

Why release the source code in MIT?

Because I am a big fan of total freedom in open source. I do not like the GPL family of licenses. Having said that, I respect the opinion of other peoples’, so I dual licensed the source code under GPL as well.

But open-sourcing the system gives them access?

They (the entrepreneur himself, and another account who I suspect is the new developer) already have access to the Github repository. Their access was revoked just before I open sourced the work.

To be honest, I could not care less what they do with the source code. Just the fact that he chose to shaft me is a strong enough indication that he will not succeed.

Again, why open source now?

I would have waited a little longer if it wasn’t for the fact that his new developers don’t seem to know what they were doing. A while ago I received a bunch of broken test emails sent to my inbox - indicating that he’s already hired someone else to continue the development work. Last night, a bunch of test emails came through again - so that was when I decided enough is enough.

Is it wise for you to do this?

I don’t know, you decide! :) Is gifting the source code to someone who might find it useful a good thing? I sure hope so.

Are you looking for freelance work?

Nope. Not because I just got burned, but because I have my own adventures to pursue.

What about other work and/or startup opportunities?

You may find out by dropping me a line, all my contact details can be found on this blog. :)

I am interested, but who are you?

Scroll down to the bottom of the page and you will find my bio with a couple of links. :)

]]>

It's Year 2011, Why Aren't People More Open-minded?

2011-07-30T12:01:00.000000Z

On a cold windy day, we are comfortably sitting in our chairs beside the fireplace, holding a mug full of hot tasty latte. We are slowly and calmly sipping on our coffee, imagining that things could be far worse outside of our safe and warm house.

Exactly how do we know that things could be far worse? We don’t, because we are too comfortable and too familiar with what we currently have.

Being happy with where you are is one thing, being ignorant is another.

Inventions and innovations aren’t born out of happiness, they are born out of frustration, anger and sometimes, curiosity.

Being open-minded about how things work, in my opinion is one of the key attributes an entrepreneur should have.

All decisions come with implications and consequences, to dismiss an alternative approach, or worse, to not have any alternative approaches at all because of ignorance is often one of the first signs of failure.

Successes are built on many failures, there is no doubt about that. However, failures should be the outcome of experimentation rather than ignorance.

Frustrations and concerns arise when businesses make decisions (or should I say, mistakes) because the decision makers are close-minded - whether it’s a decision about choosing a piece of technology, or it’s a decision about sticking to a fixed launch date.

Blindly following some rare exceptions such as Apple and 37signals is probably not going to work for most of us.

But heck, what do I know, I am just a web developer, right?

]]>

comic because activerecord is slow

2011-07-06T12:12:00.000000Z

Comic: Because ActiveRecord is Slow!

]]>

Moving On From Envato, What's Next?

2011-03-17T03:04:00.000000Z

Today is my last day at Envato. I have been working here for a year and half and it has been, without a doubt the most fulfilling experience I have ever had in my professional career.

I was hired as a PHP developer initially. Who knew, several months into the job I was ‘converted’ voluntarily to a full time Ruby developer - and it has been the highlight of my career progression. It is hard to imagine what I would have become if it wasn’t for Envato’s support behind my conversion.

During my time at Envato, I had the opportunity to work with some of the most talented people in the web industry, from both inside Envato and externally. I am extremely grateful for the experience and I hope to work with them again in the future.

The decision of moving on from Envato was not made easily. As a matter of fact, I have always been passionate about online education (e.g. the Tuts+ network) and I sincerely hope the project will be taken to the next level.

Oh hey, I am still working on a book project with Rockable Press from Envato. If you are a web developer, expect to see a (hopefully useful) web development book this year. ;)

10 days from now I will be joining PlayUp as a senior web developer - to work on more awesome Ruby/Rails stuff! :) The colleagues I have already met at PlayUp are extremely talented and friendly. I cannot wait to start my first day there!

Aside from my day job, I am also working with some very experienced professionals on a startup project - hopefully you will get to hear more about it soon.

Last but not least, I am finally out of some personal and relationship crisis, and I am able to regain my focus and spend more time working on my varies side projects (to be branded under Wuit) as well as on some of the open source projects such as Slim.

Exciting times ahead! :)

]]>

my entry to the rubycommitters design contest

2011-01-18T22:26:15.000000Z

My entry to the RubyCommitters design contest. View it in action at http://heroesofruby.heroku.com/ :-)

]]>

sneak peak of the redesign of slim langcom

2010-12-22T22:15:00.000000Z

Sneak peak of the redesign of slim-lang.com. :)

]]>

[Ruby] Haml2Slim, Convert Your HAML Templates to Slim Templates

2010-12-16T09:08:51.000000Z

So, you are using Slim, right? If not, go check it out because it’s awesome. ;)

Now, I don’t know about you but prior to Slim I use Haml quite a bit, and even though syntax-wise Haml and Slim have a lot in common, it’s still quite a challenge to convert all Haml templates to Slim templates.

As a result, let me present you with a quick n’ dirty Haml2Slim converter! Check out the source code.

]]>

[Ruby] Releasing Ssync - An Optimised S3 Sync Tool Using the Power of Unix!

2010-11-12T08:15:41.000000Z

At Envato we have a few dozen sites residing on multiple servers. The data on a portion of the servers need to be regularly backed up to Amazon S3.

The Envato Mad Scientist Ryan Allen has worked on a script called Sir Sync-A-Lot which syncs the data to S3. This was done after evaluating a bunch of scripts including s3sync.

Today I turned Ryan’s original script into a little Rubygem and added a bit more features.

Go check out the source code!

]]>

Slim, a Fast and Lightweight Rails Template Engine!

2010-10-17T21:40:20.000000Z

For the past few weeks I have started contributing to a small project - Slim.

Slim is a fast, lightweight templating engine for Rails 3. It has been tested on Ruby 1.9.2 and Ruby/REE 1.8.7. Slim is heavily influenced by Haml and Jade.

Andrew Stone who is the author of the project has posted a quick update on the latest feature additions to Slim. Please go check it out.

The source code of Slim is available on Github.

]]>

A Speedier Rails App using Rails 3.1 + Arel 2.0

2010-09-29T00:51:00.000000Z

Before (Rails 3.0.1pre stable branch + Arel 1.0.1):

After (Rails 3.1.0 master branch + Arel 2.0.0dev master branch:

Thanks to the awesome work done by Aaron Patterson (@tenderlove) and others. :-)

UPDATE:

After Aaron Patteron’s tweet, I ran the tests again on Rails 3.0.1pre stable branch + Arel 2.0.0dev master branch, and the result blew my mind:

]]>

`bundle: command not found` or `Could not find RubyGem bundler (>= 0)` During Capistrano Deployment? No Problems!

2010-09-16T05:59:00.000000Z

Don’t you just hate it when you get the following errors during a Capistrano deployment?

bundle: command not found
Could not find RubyGem bundler (>= 0) (Gem::LoadError)

In fact, even if you don’t use bundler, you might still get errors like this:

rake: command not found
Could not find RubyGem rake (>= 0) (Gem::LoadError)

It turns out this has something to do with the $PATH and $GEM_HOME variables.

So here’s the quick fix.

PermitUserEnvironment yes

Don’t forget to restart ssh:

/etc/init.d/ssh restart

Now, log in as the deployment user, and create ’~/.ssh/environment’ with the following content:

PATH=/usr/local/rvm/gems/ruby-1.9.2-p0/bin:/bin:/usr/local/rvm/rubies/ruby-1.9.2-p0/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GEM_HOME=/usr/local/rvm/gems/ruby-1.9.2-p0

* The above paths are for your reference only, obviously you need to work them out for your server environment. The only thing you need to make sure is that the GEM_HOME’s path matches one from the PATH.

Now, to verify this all work, you may use cap shell to start a new shell session and try out your commands.

]]>

wuit trademarked in australia now the

2010-08-27T09:03:41.000000Z

Wuit trademarked in Australia now! :-)

The trademark application in the US is being progressed as well.

If you don’t already know, Wuit is my soon to be launched studio identity.

]]>

[Rails] RailsConfig, A New Iteration of AppConfig for Rails 3

2010-08-11T01:50:14.000000Z

If you haven’t already been using AppConfig in your Rails project, well, you should!

Jacques Crocker has recently released his new version of the original AppConfig - RailsConfig. I was invited to join the development of this new tool, so make sure you go check it out. :-)

]]>

[Rails] Inherited Resources Views Now Supports Rails 2.x

2010-08-07T07:35:00.000000Z

I have just pushed a commit that added Rails 2.x compatibility to Inherited Resources Views. Please give it a spin! :-)

P.S. I’ve only tested it on Rails 2.3.8.

]]>

Reinvigorate - Realtime Website Traffic Analysis

2010-08-06T00:33:00.000000Z

It isn’t the first time realtime website traffic analysis has been introduced. Reinvigorate is one of the services that provides realtime traffic tracking and analysis.

The thing I like it most though, is in fact the heatmap. :-)

Heatmap is a great tool to help identify convoluted interface and improve the user experience.

]]>

Poke me on Github! :D

2010-08-06T00:03:00.000000Z

I have just added a Github ribbon to my blog (look at the top left corner!). Please feel free to poke me! :D

References:

]]>

[Rails] Releasing Inherited Resources Views - DRY Your View Files

2010-08-05T11:16:53.000000Z

Using Inherited Resources is an excellent way to reduce the amount of repetition in your controllers. But what about views? A lot of times resources share the same views, so why not DRY ‘em up using Inherited Resources Views!

Go check out the code! :-)

]]>

released my first ruby gem

2010-08-04T11:32:34.000000Z

Released my first ruby gem. :-)

]]>

looks like its time for a reboot

2010-08-04T07:53:38.000000Z

Looks like it’s time for a reboot…

]]>

[jQuery] Releasing Inline Confirmation, Confirm Actions Done Right

2010-08-03T05:41:00.000000Z

In a web app, it is very common to have actions that destroy (delete/remove) data. These actions, if you don’t already know, should always map to POST methods. On top of that, because these actions are destructive, the UI should always ask the user for confirmation.

But how do we actually implement the confirmation dialogue though? The vanilla JavaScript confirm box would be the easiest but at the same time the ugliest - this thing stalls most web browsers until the user acts on it.

An inline popup/modal box? Perhaps, but it is still obtrusive, in the sense that the popup/model boxes are usually in the way of other tasks.

Meet Inline Confirmation - a jQuery plugin for creating easy, less obtrusive confirmation dialogues!

Feel free to give it a spin. I will add more documentation and a demo when and if I have time. ;)

]]>

I'm Now a Ruby on Rails Contributor

2010-08-02T22:40:46.000000Z

I am extremely happy that my patch was accepted, so I am now one of the 1600 odd people who have contributed to the Rails project! :D

]]>

[PHP] Releasing KThrottler, A Kohana Module for Throttling Actions

2010-07-28T13:29:00.000000Z

Similarly to Action Throttler for Rails, KThrottler is an easy to use Kohana module to quickly throttle application actions based on configurable duration and limit.

Go check out the code now! :)

]]>

#whatdatingislike What Dating Is Like

2010-07-26T14:21:11.000000Z

Dating girls is like calling a callback function without knowing the accepted arguments. Random exceptions would throw right in your face.

Dating girls is like running prerelease ruby gems. Things might not be compatible, there might be memory leaks and other weirdnesses.

Dating girls is like reading a tutorial in a foreign language. You think you got it, but you don’t.

Dating girls is like using a ruby class with lots of mixins. You’re looking right at a class method, but you don’t know where it comes from.

Dating girls is like running rails on Webrick. It works, for the most part, but you don’t know when it will fall into pieces.

Dating girls is like using Zend Server for PHP - it might have impressive features, but it’s expensive and the shit still runs on Apache!!

Dating girls is like using CakePHP. It might look like Rails, but it’s not, unless you’re happy with seeing arrays everywhere.

Dating girls is like running production PHP sites on IIS. Sometimes you wonder, why bother?

Dating girls is like writing a Capistrano recipe, you don’t know what it’s going to happen until you run it.

Dating girls is like buying an iPhone 4. You know it has flaws, but you are gonna buy it anyway.

Dating girls is like coding in Obj-C. It’s hot right now, but you wonder how long is it going to last?

Dating is like using Zend Framework. It looks so impressive from distance, then when you actually started using it…

Dating girls is like making a huge deployment. As confident as you are, you still need to cross fingers!

Dating girls is like developing for Wordpress. It looks so pretty on the surface, but the roots are broken beyond repair.

]]>

[Rails] Introducing Datamappify - ActiveRecord Without DB Migrations

2010-07-22T12:30:50.000000Z

Introduction

ActiveRecord is without doubt the de facto ORM library for Rails and many Ruby web frameworks. Many developers however, do not like database migrations and prefer to use DSL for data mapping. Datamappify is created with the sole purpose of getting rid of the DB migration headaches.

Why Not DB Migrations?

Well, depending on your specific project, DB migrations might create more trouble than it’s worth. Besides, your code is already version controlled, so why create a separate version control for your DB schema?

Why Not Use DataMapper, Sequel, etc?

As stated in the introduction, ActiveRecord is the most popular ORM in the rails community, it is actively developed and battle-tested. If your only grief with ActiveRecord is the DB migrations, why not just eliminate it be happy? ;)

Go check out the code!

]]>

Fuck GPL!

2010-07-15T13:56:17.000000Z

Fuck GPL!

]]>

just some late night sketches done with a finger

2010-07-14T14:21:00.000000Z

Just some late night sketches, done with a finger, an iPad and some wine.

]]>

Wuit.com Now Runs on Padrino

2010-07-01T21:59:28.000000Z

A few days ago I discovered Padrino, an excellent ruby framework built on top of Sinatra. What can be a better than experimenting with it? Build an actual website with it!

So last night I ported Wuit.com from using vanilla PHP + Flourish to using Padrino + DataMapper + Haml.

The entire porting experience was positive, albeit the website only has two pages.

I did encounter a strange problem with Capistrano though - both current_release and latest_release were giving me strange results. In the end I had to modify my deployment recipe to overcome this.

For building small to medium sized projects, I think Padrino is an excellent choice, as it offers a more complete foundation than Sinatra, and is not as heavy as Rails.

]]>

[PHP] Releasing PHamlP Module for Kohana, Use Haml and Sass with Kohana 3.0!

2010-06-27T14:42:52.000000Z

Haml and Sass are two extremely useful template engines. They are very popular amongst the Ruby and Rails community.

I am happy to release a module for Kohana v3 that uses the PHamlP library to offer Haml/Sass support for Kohana.

Right now the module only supports Haml but I will be adding Sass support in the very near future.

Head over to GitHub to check out the source code. :)

]]>

Deploy PHP Websites Using Capistrano (and Git)

2010-06-21T04:45:00.000000Z

The Rails deployment flow is really smooth thanks to the powerful (and easy to use) Capistrano.

Capistrano not only works with Rails and other Ruby code bases, but also code bases in any programming languages, such as PHP!

Here is an overview of what I did to get one of our PHP production sites up and running with Capistrano.

If you don’t already have Ruby and Rubygems installed, install them!

After you got ruby and rubygems, install Capistrano and its related gems -

gem install capistrano-ext railsless-deploy

Now, navigate to your application’s directory and run capify ., this will generate some necessary files for Capistrano to recognise your app.

Open up the generated Capfile file and replace the content of the file with:

require 'rubygems'
require 'railsless-deploy'
load 'config/deploy'

Okay, now we need to set up the actual deployment recipe in the config/deploy.rb file. But before doing so, we need to set up our deployment server with proper user and deployment permission.

We chose to set up a new user called deploy specifically for deployment purpose, for example -

useradd deploy

Then of course, we need the SSH keys for this deployment user, so switch to this user (su deploy) and -

ssh-keygen

We use GitHub, so simply copy the content of the public key (~/.ssh/id_rsa.pub) and add it to the deploy keys section of the Github repository.

Don’t forget to also add GitHub to the .ssh/known_hosts file, you can do this by manually cloning your repository on the deployment server.

Great! Now let’s do the last step - create the actual deployment recipe!

Open up config/deploy.rb file and you will see some default deployment tasks. Everyone has different needs, so I’m going to paste our deploy.rb file (masked with added comments) for your reference.

set :user, "deploy"
set :application, "YOUR_APPLICATION_NAME"
set :domain, "YOUR_APPLICATION_DOMAIN_NAME"
set :repository, "THE_ADDRESS_OF_THE_APPLICATION_REPOSITORY"
set :deploy_to, "/var/www/#{domain}"
set :shared_path, "#{deploy_to}/shared"
set :use_sudo, false

set :scm, :git
set :branch, 'master'
set :deploy_via, :remote_cache

role :web, "ADDRESS_OF_YOUR_WEB_SERVER"
role :app, "ADDRESS_OF_YOUR_APP_SERVER" # this can be the same as the web server
role :db, "ADDRESS_OF_YOUR_DB_SERVER", :primary => true # this can be the same as the web server

namespace :deploy do
  task :start do ; end
  task :stop do ; end
  task :restart, :roles => :app, :except => { :no_release => true } do
  run "#{try_sudo} /etc/init.d/lsws reload" # we use LiteSpeed Web Server
  end
end

# The task below serves the purpose of creating symlinks for asset files.
# Large asset files like user uploaded contents and images should not be checked into the repository anyway, so you should move them to a shared location.

task :create_symlinks, :roles => :web do
  run "ln -s #{shared_path}/uploads #{current_release}/uploads"
  run "ln -s #{shared_path}/zb #{current_release}/zb"
end

# Let's run the task immediately after the deployment is finalised.

after "deploy:finalize_update", :create_symlinks

That’s it! Now you should be able to deploy your application by -

cap deploy:setup
cap deploy

]]>

[jQuery] Endless Scroll Updated, Now Works with Any DOM Elements

2010-06-18T06:48:00.000000Z

jQuery Endless Scroll has now been updated to work with any DOM elements, not just $(document).

Click here for the project page (with usage examples).

Click here for a simple demo.

Enjoy!

]]>

[Rails] Releasing Action Throttler, A Rails Plugin for Throttling Actions

2010-06-03T03:21:00.000000Z

Sometimes we would want to throttle certain application actions. For instance, a user should only be allowed to send x amount of emails to other members of the site in order to reduce the spam risk.

Since this is a quite common task, I thought I might as well make it as a Rails plugin for better code reusability.

I am now releasing Action Throttler, an easy to use Rails plugin to quickly throttle application actions based on configurable duration and limit. Go check out the code now! :)

There is currently no tests for the plugin but I will be adding rspec specs to it soon.

]]>

A Whole World of Difference: Phusion Passenger Apache to Nginx, Ruby Enterprise Edition 1.8.6 to 1.8.7

2010-05-27T08:35:00.000000Z

Yesterday we have launched the new design of Creattica. It runs well during the testing and staging phase, unfortunately the server quickly became overloaded and unresponsive after the relaunch was made public.

The site was running fine with its old design, but because the new design has many added features such as search filters, followers and more, system resources were very quickly drained.

After some optimisation of the database (which itself has increased the site performance by up to 300%+), queries as well as upgrading Ruby Enterprise Edition from 1.8.6 to 1.8.7 and switching from using Apache + Passenger to Nginx + Passenger, the site runs so much smoother.

Don’t believe me? Take a look at these screenshots below.

Switching from REE 1.8.7 to Ruby 1.9.1 did not yield any significant result though.

]]>

busy at work

2010-05-27T05:49:58.000000Z

Busy at work…

]]>

[Rails] Use App_Config For Your Application Specific Configuration

2010-05-25T23:50:46.000000Z

A simple Google search will reveal that there are a number of different App_Config plugins for Rails. After comparing them side by side, I have decided to use the one by Christopher J. Bottaro.

It features:

simple YAML config files
config files support ERB
config files support inheritance
access config information via convenient object member notation

I just fixed a bug last night (which was pulled in to the main repository) where it could throw errors when used as a Rails plugin (i.e. via rails plugin install).

Make sure you go read the instructions on how to use this handy plugin.

]]>

[Rails Tip] Model Attributes Not Updating? `reset_column_information` To the Rescue!

2010-05-25T10:58:00.000000Z

So you were wondering why some of your model attributes weren’t updating properly? Well, it is perhaps because the db schema has changed but the changed schema has not been passed onto ActiveRecord, as is often the case in DB migration.

Taken from the ActiveRecord documentation:

Resets all the cached information about columns, which will cause them to be reloaded on the next request.

The most common usage pattern for this method is probably in a migration, when just after creating a table you want to populate it with some default values, eg:

class CreateJobLevels < ActiveRecord::Migration
  def self.up
    create_table :job_levels do |t|
      t.integer :id
      t.string :name

      t.timestamps
    end

    JobLevel.reset_column_information
    %w{assistant executive manager director}.each do |type|
      JobLevel.create(:name => type)
    end
  end

  def self.down
    drop_table :job_levels
  end
end

]]>

[jQuery Tip] Traverse/Parse HTML String

2010-04-28T01:42:00.000000Z

When you are getting an HTML string from an external source (e.g. from an AJAX get result) and you want to rip out a certain part of the HTML source, you need to make sure that the ‘certain part’ is not at the top level of the HTML source.

For example, we have the following HTML string:

Hello
World

If we want to get the first paragraph element by using:

// data is the HTML source
$('span#first', data)

The above code won’t work, because the p tags are at the top level. Instead, we can simply wrap the HTML source with a div tag and that’ll do it. :)

]]>

[Rails Tip] DataMapper M:M Association Bug and Workaround

2010-04-26T23:35:00.000000Z

It was confirmed that DataMapper is incorrectly setting table names in SQL JOINs.

So for instance, the following code would generate an SQL error:

type.jobs.all(:"country.name".like => "%#{params[:location]}%")

There is a workaround, however the workaround requires manual looping of the dataset thus produces N+1 queries.

type.jobs.all.reject do |job|
! job.country.name.downcase.include?(params[:location].downcase)
end

But at least it works. ;)

]]>

[Rails Tip] DataMapper Timestamps Bug and Workaround

2010-04-19T08:22:00.000000Z

It has been confirmed by DataMapper’s core developer Martin Gamsjaeger (snusnu) that it is a bug.

In short, created_[at|on] can be manually overridden, but updated_[at|on] cannot.

The workaround is simple, do it in two steps, for example:

job = Job.create(
  title: "Web Developer",
  created_at: Time.now - 2
)
job.update!(updated_at: Time.now - 1)

Please check the bug ticket to see when it’s getting fixed.

]]>

[Rails Tip] Run Specs Faster!

2010-04-09T04:03:00.000000Z

If you are using rake spec to run the specs. Try using spec spec instead! It avoids doing some preliminary tasks and therefore is quicker to execute.

You can verify the difference using Unix’s time command, i.e. time spec spec and time rake spec.

]]>

[Rails] Run Queued Tasks Using 'delayed_job', Now With Intervals!

2010-03-25T11:53:00.000000Z

If you don’t already know, ’delayed_job’ is a database based asynchronously priority queue system extracted from Shopify.

I was seeking for a lightweight, easy to use solution for queueing tasks, ‘delayedjob’ fits the bill _almost perfectly. ;)

One thing we needed to do for a Rails 2 project at Envato, was to queue the tasks and to run them one after another, at a set interval. ‘delayed_job’ unfortunately doesn’t come with this feature.

So, instead of searching for more complicated solutions or reinventing the wheels, Imade a couple of commits to my fork to add this functionality as well as to fix some rails 3 compatibility issues.

Now you can simply add the following line to your config/initializers/delayed_job_config.rb file to have the tasks run at 1 minute interval.

Delayed::Worker.run_interval = 60

A pull request has been issued, hopefully it’ll come as a standard feature soon. :)

]]>

i need textmate 2 or a 3d monitorprojector to

2010-03-17T09:25:04.000000Z

I need Textmate 2 (or a 3D monitor/projector) to come out sooner, otherwise there will be a day where I will just be totally lost in dozens of windows.

]]>

[Rails] MongoMapper on Rails 3 Master, Undefined 'to_key' Method

2010-03-17T08:54:34.000000Z

Rails 3 has recently renamed key method to to_key, which consequently rendered MongoMapper unusable.

You might see an error that looks like:

ActionView::Template::Error (undefined method `to_key' for #)

After some twiddling I’ve submitted a patch to address this issue. Feel free to pull my fork until the fix makes its way into John Nunemaker’s main repository.

]]>

Use the 'textmate' Command to Quickly Install Bundle Files

2010-03-14T10:46:00.000000Z

Initially developed by Yehuda Katz, the Textmate gem is an extremely handy tool for installing and managing your Textmate bundle files.

The tool works very well, for the most part. But due to the way the GitHub search query was built, the tool failed at finding Textmate bundles that do not honour the ’-tmbundle’ suffix in the name of their repositories.

I was waiting for GitHub to fix their API bug which I reported here, but today I discovered a workaround that solves this issue.

So, here, I present you with the improved code on my fork.

Update: The changes are now merged back to Ddollar’s repo, so you may install the Textmate gem by:

gem install textmate --source git://github.com/ddollar/textmate.git

Enjoy! :)

]]>

i hate it when this happens and i have a dozen of

2010-03-14T02:53:35.000000Z

I hate it when this happens and I have a dozen of Textmate and Terminal windows open…

]]>

[Rails] Use HAML templates with Devise

2010-03-13T14:40:00.000000Z

If you are a Rails developers, chances are you have heard of, or are using either Devise or HAML in your projects.

And if you’re like me who uses both, then surely you’d wish these two worked together, i.e. generate devise views in HAML.

Today is your lucky day! I’ve just committed some changes that enable you to do so! The changes are already merged back to the primary repository.

In order to generate HAML views instead of ERb views, simply do:

rails g devise_views -t=haml

You can use the master branch of Devise, i.e. in your Gemfile:

gem "devise", :git => "git://github.com/plataformatec/devise.git"

You will also need the edge version of HAML, as the stable versions do not parse ruby code correctly.

]]>

first photoshop now textmate whats next

2010-03-09T10:43:35.000000Z

First Photoshop, now Textmate, what’s next?

]]>

Advanced Search Query on GitHub

2010-03-09T08:54:00.000000Z

Did you know, you can perform advanced search queries on GitHub?

For example:

ruby AND (textmate OR tmbundle)

The above query will search everything containing ruby, as well as either textmate or tmbundle.

Unfortunately at this stage the GitHub API does not support it. I’ve opened a ticket here.

]]>

[Rails Tip] Making i18n Forms, the Easy Way

2010-03-08T05:41:00.000000Z

Web applications normally have many forms. Building forms is always a mind-boggling task because it involves repetition and chaos.

A shortcut is to use a form builder / DSL, such as Formtastic.

As I am using Rails 3, and the Rails 3 port of Formtastic isn’t complete yet, I thought I’d just use the plain vanilla Rails built-in form helper.

First of all, I am using Haml instead of ERb. Already, I got the out-of-box clean looking Haml markup.

Some of you might not be aware of the fact that Rails’ built-in form helper already does i18n support.

If you have the following form:

-form_for(@post) do |f|
  =f.label :title
  =f.text_field :title
  =f.label :body
  =f.text_area :body
  =f.submit

You can simply translate the labels as such in config/locales/en.yml.

helpers:
  label:
    post:
      title: 'Post Title'
      body: 'Post Content'

Better yet, I am using r18n instead of i18n, so I can instead translate them in app/i18n/en.yml.

helpers:
  label:
    post:
      title: Post Title
      body: Post Content

Alternatively, instead of placing the translation strings under helpers.label, you may place them directly under your models, i.e. activemodel.attribute.

If you’re using the Rails 2.3 stable branch from Github, you can also use the built-in i18n support. Although instead of helpers.label, you use views.labels, as seen in this commit.

]]>

this is from an email newsletter i received in my

2010-03-05T23:46:43.000000Z

This is from an email newsletter I received in my Gmail inbox from Macheist. Note to all the designers out there, please don’t use fancy CSS, not even some of the basic ones, because they don’t work in email clients! Here’s a great table of contents on what you can and cannot use for different email clients.

]]>