Blog - Fred Wu's Tech

Introducing Rizz.farm - An AI-Assisted Lead Generation Tool for Reddit

2024-01-06T13:10:47.679944Z

After having soft launched Persumi in 2023, I quickly found myself wanting to explore better ways to promote the product. Unfortunately, most of digital marketing products are focused on three main areas:

Keyword bidding, e.g. Google Ads or Facebook Ads
Cold outreach, e.g. Apollo
Search Engine Optimisation (SEO)

They each has its own pros and cons. As an indie developer without a huge budget, none of these tools work effectively.

The Challenges

Keyword bidding and targeted ads are expensive

Bidding wars drive up the cost of targeted ads, sometimes so significantly they price out many small businesses.

Users are increasingly using adblockers, making targeted ads ineffective.

Cold outreach are rarely answered

Highly sought-after leads are bombarded with cold emails every day, making the tactic ineffective.

Most cold emails end up in users’ spam bin, never to be seen again.

Search Engine Optimisation is a lost cause

SEO’ed content are being produced hastily, making them low quality and untrustworthy.

Sophisticated users have learnt to ignore highly SEO’ed content from untrusted sources.

The Epiphany

Reflecting on my own behaviour in choosing a product or service, I realised that over the years I have developed a good sensory to weed out the noise. Namely, I am very skeptical with highly ranked websites, I always use an adblocker (uBlock) to block annoying ads, and both my personal and work email inboxes are overflowed with cold emails I never read them.

So how do I do research, say when I’m buying a new TV, or subscribing to a new software package? I read user reviews. And not just questionable reviews on Amazon or review sites, but on Reddit, where (mostly) real users discuss and review products and services.

Of course, like any other place, Reddit can be filled with bots and fake content too. However, in my experience these are easily distinguishable, especially paired with content’s upvotes and user’s post history. With Reddit’s popularity, this makes it a highly desirable place for people to both look for answers, as well as to provide answers.

So I asked myself, what if there was a smart lead generation tool to help me promote my product on Reddit? What should the tool be capable of?

The Solution

In order to really stand out, and provide tangible value, I’ve focused on the following areas for the lead gen tool.

Warm leads over cold leads

By searching for and engaging with relevant content, the leads the tool finds for you are warm and targeted.

Many of the leads are already highly ranked on search engines, further boosting the content’s reach.

Personalised and public

Content the tool produces for the user are context-aware and personalised, making them highly relevant and trustworthy.

Even better, the content are made available publicly, boosting the user’s reputation and reach even further over time.

Infinite scale at a fixed cost

The tool finds leads and suggest responses for the user 24/7 non-stop, at a fraction of the usual marketing costs.

The pricing is fixed and transparent, so the user can scale their business without worrying about blowing the budget.

The Bottom Line

Ultimately, I want a lead generation tool that really provides value to not only its user (the marketer), but its user’s users (the target audience). Instead of focusing on selling, it should focus on helping people. This is where Rizz.farm comes in.

How Rizz.farm Works

With Rizz.farm, machine learning is put to good use. It’s not just another AI wrapper, it actually leverages many of the latest AI innovation to really take lead generation to the next level.

Smart search

Rizz.farm widens the search radius to find the most relevant or highly ranked content for the user. It may find leads a simple Google search couldn’t. Behind the scenes, keywords are expanded and multiple search queries are submitted to search engines.

Continuous monitor

Rizz.farm continuously monitors the social media platforms (Reddit to start off with) to find new leads 24/7, delivering them right to the user’s inbox. Whether they are highly ranked popular posts or the latest post in a particular subreddit, Rizz.farm is able to quickly deliver them to the inbox without fuss.

Leads inbox, as simple as Gmail

Leads are automatically sent to the user’s inbox, where they can review and decide whether to engage with them. It’s as simple as using email.

“Rizz Score”

As mentioned earlier, the tools should focus on helping people, rather than hard selling. To make it easier for the user, Rizz.farm calculates a score based on how relevant, helpful and spammy the generated or user edited response is. This ensures it always publishes high quality content for the target audience, making it a win-win.

Self-Learning AI

Last but not least, by using state of the art AI technologies that adapts to the users, Rizz.farm is able to help them automate and scale their lead generation and reputation building with ease.

Taking It For a Spin

As soon as I’ve deployed Rizz.farm to production, I started using it to promote Persumi and dare I say I am very happy and impressed with its results. Not only does it find relevant leads right away, it drafts up responses that are super helpful and relevant to the OP’s questions or posts, and is very subtle in pushing the “promotion” agenda. I know a lot of subreddits shun or outright ban self-promotion, so it’s extremely important to have helpful posts.

If you are looking for an affordable way to promote your product, service, event or anything really, please give Rizz.farm a try. It comes with 7 days of free trial and 30 days of no-question-asked money back guarantee. For a limited time, you can also get 50% off for six months using the coupon code LAUNCH2024.

Happy lead generation and growth hacking everyone!

]]>

Comparison of AI OCR Tools: Microsoft Azure AI Document Intelligence, Google Cloud Document AI, AWS Textract and Others

2023-09-17T03:44:24.872189Z

For a project I’m working on, we are looking at different AI OCR (Optical Character Recognition) options that would allow us to import documents with various layouts and extract relevant data from them with high enough accuracy. Due to the nature of these documents and the information contained in them, it is paramount that there would be an easy way for us to train the AI models using our documents.

Without revealing exactly what types of documents we are working with, as they’re commercially sensitive, the basic premise is to:

Import a document that might be one or more pages, the document itself may be high quality PDF, or low quality scanned documents that may or may not be skewed
Each page may or may not contain relevant information we want to extract
Relevant information may be structured in different ways, but usually in a tabular form (but each “line item” can either be horizontal or vertical)

In our case, a traditional OCR solution wouldn’t work as the documents often contain heaps of irrelevant data, and for the relevant data, it needs to understand the context in order to work out which data elements to extract, and how they relate to each other. Therefore, a machine learning based OCR solution that could adopt to our documents is highly desirable.

The success of the project is measured by the end results of imported data:

How much data is detected and recognised
How accurate the recognised data elements are
At a minimum, 80%+ accuracy is needed in order to save time in manual data entry, and to reduce/avoid incorrect data from being checked by a human operator before entering the system

Available OCR Tools

There are a number of OCR tools available, for our project the minimum criteria are:

AI powered solution so it can be further developed and tweaked by us for our documents
A UI solution for data training, preferably easy to use so that non-technical team members can help with data training too
A quick and easy way to test and verify trained models
A solution that is compliant with the ISO standards relevant for us
Pay by usage pricing model that’s not cost prohibitive

The tools I have discovered and considered were:

AWS Textract
Microsoft Azure Document Intelligence (formerly known as Azure Form Recognizer)
Google Cloud Document AI
Rossum.ai
Super.ai
Eden.ai

All six options are AI based OCR solutions. Amazon Web Services (AWS), Microsoft Azure and Google Cloud Platform (GCP) are the obvious “big three”, having their own AI models, whereas Super and Eden are aggregators that farms AI calls to providers like AWS, Azure and Google, and provides a middle layer that could potentially make the end result better (or worse). It’s unclear what model Rossum uses.

After some consideration, Rossum, Super and Eden were all taken off the table due to:

Rossum and Super not having transparent pricing
Eden being an API aggregator built by a small team, it’s unclear on its usefulness and longevity

With the longevity of the company and product in mind, as well as transparent pricing and commercial support, AWS, Azure and Google remain to be the leading choices.

The Big Three: AWS, Azure and Google

AWS Textract

Out of the big three, AWS is our preferred vendor due to its market leading position as a cloud service provider, as well as our existing projects already using AWS.

However, upon closer inspection, it appears Textract does not allow custom data training. Textract is provided “as-is” using Amazon’s pre-trained models, it does not allow customers like us to provide their own training data to improve on the detection and recognition of different types of documents.

AWS’ machine learning blog has published an article on “end-to-end intelligent document processing solution“ in 2020, however it does not explain in detail exactly what the capabilities are, and the solution requires a complex architecture that needs significant amount of time and effort to set up and maintain.

Azure Document Intelligence

Azure’s offering, formerly known as Form Recognizer, is an end-to-end solution that offers custom data labelling and training.

Google Cloud Document AI

Google’s offering, similar to Azure’s, is also an end-to-end solution that offers custom data labelling and training.

Comparison: Azure vs Google

With the initial assessment done, we are left with Azure and Google, so let’s dive into the deep comparison of the two platforms.

We will compare the two platforms in several aspects:

The ease of initial set up, i.e. how quickly can we get up and running right away
Whether a “base” pre-trained model is provided for documents with high enough accuracy so we save time in customisation and training
The detection and recognition – how much data is accurately detected and recognised
The custom training process, how easy is it to label custom data
How fast or slow the training is
The end result – provided with the same training dataset, which platform offers better results

The Initial Setup

Microsoft Azure

Azure’s initial set up is relatively straightforward. There is a four-step dialog window to set up the required Azure resources, etc. After which, documents can simply be uploaded for immediate consumption - labelling and training, which will be touched on in later sections.

Google Cloud

Google works slightly differently in this case. To set up a project it’s very simple, there is literally only the “Processor name” to fill out. However, it does require a separate step to set up the storage bucket required.

Also, instead of allowing files to be uploaded within the same UI like Azure, Google requires files to be uploaded to the cloud storage bucket first, then imported into Document AI.

Auto Labelling Data

Microsoft Azure

“Auto-labelling” refers to the AI OCR’s ability to do the initial labelling automatically, to save time and effort in the manual labelling work. It can be seen as the starting point of training the AI model.

In this case, the Azure experience is significantly better. It allows files to be uploaded, then auto-labelled by “current”, “unlabelled” or “all” documents. This gives the flexibility to users (i.e. us) to choose when a document should be auto-labelled – as it does take a bit of time, sometimes if a document is not very well structured and would require more manual labelling anyway, then it’s best to skip the auto-label step.

Google Cloud

Google on the other hand, has a more cumbersome process. We first need to upload the files to the storage bucket, as mentioned before. Then, we select the path/folder of the uploaded files, and the “auto-labelling” is enabled or disabled for the entire import. Also, sometimes the import takes a long time, whereas on Azure it’s always instant.

More on the accuracy of auto-labelling later.

Text Detection and Recognition

A big part of OCR is obviously text detection and recognition. Even without training, a good OCR solution should detect and recognise clear text effortlessly.

During our initial research, we’ve found this article that compared the text recognition accuracy between AWS, Azure and Google. Their finding was that Azure and Google were comparable with a slight edge to Azure (Azure performed better on 2 out of 3 documents, and Google performed better on 1 out of 3 documents), both were way ahead of AWS in terms of accuracy.

Our own testing has somewhat mirrored their experience. We found that in general, Azure performed better in the number of detected text elements, as well as the correctness of these text.

On Azure, it’s very rare for text to be undetectable, whereas on Google, we’ve found several instances where the text is very clear, yet Google was unable to detect the text at all.

Similarly on accuracy, Azure has rarely detected clear text incorrectly, whereas on Google, a lot of times units such as x10*9/L get recognised incorrectly despite all instances appearing similar in their appearance (i.e. very clear), and sometimes dashes won’t get recognised (e.g. 10.0 - 12.50 gets recognised as 10.0 12.50).

Custom Labelling

During training, a big portion of time is spent on custom labelling, meaning we look for and assign text to our pre-defined data elements such as name and occupation - these are just fictional examples.

Overall, Azure and Google each has its strengths and weaknesses. Let’s go through them in detail.

Auto Layout

On Azure, there is a “Run layout” step that would recognise all the text elements, as well as tabular data in a given document.

Once “run layout” is performed, all the recognised text are highlighted in yellow, giving a quick and easy overview of the usable data elements.

Google on the other hand, does not offer a similar function, therefore labelling needs to be performed with more effort, more on this later.

Schema vs Schema-less

One key difference between Azure and Google’s solutions, is that the Azure solution is schema-less whereas the Google solution is based on defined schemas.

On Azure, we can add a “field”, and then assign the field a type (string, number, etc). A field can then be renamed or re-assigned with a different type at any time.

There is a special field called “table field”, we can create tabular data using this field type.

Google works very differently. Instead of adding fields on the fly, Google requires a schema (a.k.a. field definitions) to be created first. Once a field is created, and data trained, it cannot be edited or deleted.

One advantage of Google’s approach is a more definitive structure of OCR’ed data. When creating a label, not only do we have to choose a type (similar to Azure), we also can choose its occurrence logic, see the screenshot below.

Despite this, we found that the schema-less approach on Azure offers far better versatility without being confined to a pre-defined schema, making training new revisions of models far easier.

Labelling Data

Most of the time spent on training, is the manual labelling of data elements on the documents. Azure’s approach in this case is significantly faster and more accurate, compared to Google’s.

Microsoft Azure

On Azure, because of the “run layout” step, all text elements are already detected. So labelling them as part of the test results is very simple, you click on the yellow elements to select the text you want, then click on the table cell on the right to assign them.

However, if somehow Azure detects the data incorrectly, there is no way for us to provide a correction. In this case, we could choose to either skip the cell, or simply assign the cell with the incorrect data – if more training data is provided over time then these edge cases will not have a big impact on the final accuracy of the model.

In the screenshot above, sensitive data elements are blacked out for the purpose of this blog post.

Google Cloud

On Google, this process is significantly more involved. Due to not having a “run layout”-like step, we need to draw over the text to select it, and there’s no way to tell whether Google has detected the text before using the tool, so sometimes you’ll draw over some text and get an error saying, “Cannot create labels with empty values”.

Again, in the screenshot above, sensitive data elements are blacked out for the purpose of this blog post.

And because we can’t simply click on the highlighted text already detected, like on Azure, this makes the whole process significantly longer as we have to carefully draw over the text to ensure the boundary is accurate (for training purpose), sometimes we have to zoom in and reposition the document for accurate drawing.

The schema assignment is also less intuitive compared to Azure’s. Instead of having a table to easily see tabular data, Google’s schema is more like a series of key-value pairs, making glancing at assigned data much harder, and the key-value elements are ordered alphabetically, instead of logically like on Azure, making assigning data unnatural.

The one advantage of Google’s approach though, is that when the recognised data is incorrect, we can manually override it with our own correct data. However, after using both solutions for a while, we’ve found that Google’s has a much higher error rate compared to Azure’s to begin with.

Auto-Label Accuracy

In order to assess the auto-labelling accuracy, we have trained several of our documents with varying quality and layout. We then uploaded a new document with a layout that has not been trained.

The results are very telling.

Microsoft Azure

Azure has pretty much labelled things perfectly - all the data elements were detected correctly and within the correct contexts. The second comment block is also detected correctly.

In this case, the Azure auto-label has only missed the first comment block.

The screenshot above is blurred to protect the sensitive nature of it.

Google Cloud

Google’s result unfortunately falls short of expectation.

As seen in the screenshot below, Google has missed several data elements, and incorrectly detected elements that should not be part of the result. Again, the screenshot is blurred to protect the sensitive nature of it.

This suggests two possible things about Google’s OCR solution:

The default text recognition is poor, therefore it’s missing data elements even though they are clearly presented, in the same way as other elements
The neural network for language models is poor, therefore it incorrectly detected text that shouldn’t be part of any result

Auto-Label Result Verification

Microsoft Azure

Once Azure’s performed the auto-labelling, it’s very easy to check the results, as seen in the blurred screenshot below, we simply look at the table to ensure the correctness. However, as mentioned before, we can only correct detected regions of text, not the content itself.

Google Cloud

On Google, it is more involved. As seen in the blurred screenshot below, it requires a lot of scrolling to find the auto-labelled data. Alternatively, we can hover over the highlighted text to look at the auto-labelled text, although in this case, the pop-up gets in the way of other data which makes it an annoyance at times.

Curiously, despite having a schema, and setting certain elements to “required once” (per schema), Google still went ahead and falsely detected multiple results per schema.

But as mentioned before, Google does offer the ability to manually correct the text content, which is a big plus.

Data Training Speed

We have found training to be significantly faster on Azure.

With about 45 documents, it takes about 30 minutes to train on Azure.

Google takes about twice as long, about an hour to train, and another several minutes to deploy the trained model. Azure does not need a separate deployment step.

Interestingly, on Google, training a model requires having enough test data. So documents need to be split into training and test groups. It is unclear whether the test data is used for actual training.

There is also a required number of labelled elements (10 minimum, 50 recommended) before a model can be trained. Azure has no such limitations.

Data Regions and Compliance

While both Azure and Google are compliant to many standards (ISO, GDPR, etc), Google’s Document AI can only be hosted out of their US or EU regions. Whereas Azure has no such limitation and can be hosted in any of their available regions.

This has a side effect on the performance of UI operations. Azure is very quick and snappy due to it being deployed in our local AU region, whereas Google due to it being deployed in the US region, is a little slow every time you open and close a document for example.

Conclusion

With the in-depth analysis done, it is no surprise that in the end we went with Microsoft Azure’s Document Intelligence for our AI OCR needs. I hope these findings are useful to other people too.

]]>

How I Built a Mostly Feature-Complete MVP in 3 Months Whilst Working Full-Time

2023-08-09T09:28:12.135201Z

A few weeks ago I soft launched an MVP - you are looking at it right now.

In this post I’ll talk about the features, the tech stack and the globally distributed infrastructure behind building this MVP, and of course, with a sprinkle of learnings too.

The “MVP”

Deciding what makes up an “MVP” is always interesting - I’ve heard some saying if the product isn’t embarrassing you’re releasing it too late, and also if the product is embarrassing, you’re not gonna make it.

For me, I’ve always had the idea of building all the essential features as part of the MVP, with one or two “hero” features that would differentiate the product from the competitions, and then build out more premium features over time.

If you can’t already tell, Persumi is a content creation platform with some social networking features. It may sound bland, but what I believe makes it stand out, is the desire of putting the focus back onto the content, rather than the VC-fuelled, ever increasing appetite for more ads and user hostile features.

The Long Nights and Weekends

There is no magic beans for productivity - especially when I have a full time job. Working on a side hustle means giving up on almost all social and entertainment activities. It’s not for everyone, but I didn’t mind it too much. Being an introvert definitely helped - I was happy to see night by night the MVP gradually taking shape to become more and more real.

Looking back, I spent about three months to build out most of the MVP, then another week or two on infrastructure, and another week or two for polishing, all whilst having a full time job.

It’s been a journey, I’m glad that it “only” took me 3-4 months to get to this stage, as initially I estimated for a 6+ months MVP build.

The Features

With all that in mind, I’ve set out to build the essential features that make a blogging and social networking platform:

Short form content like a tweet
Long form content like a blog post or a book chapter
RSS feeds
Communities similar to forums and sub-reddits
Direct messaging between users
A voting (like/dislike) system
A bunch of CRUD glue pieces to make all these things work

The “Hero” Features

Beyond these seemly unremarkable features, I’ve also had in mind two key features that would differentiate the platform from the rest:

The “persona” concept, whereby each user is allowed to have multiple personas to hold different content or topics of interest, e.g. a persona for professional stuff, a persona for gaming stuff and a persona for travel stuff, etc
AI generated audio content for text (also known as Text-to-Speech)

These two “hero” features are what drove me to build Persumi in the first place. Together, they solve some very real pain points for me, namely:

Following specific topics of interest from people is difficult, with the algorithms taking over people’s home feeds, there are simply way too much noise, thanks to VC-fuelled “user engagement” metrics
Content consumption on the go (e.g. during commute or during workouts, etc) is becoming more and more prevalent, but the traditional platforms haven’t adopted to this new lifestyle other than shoving short form content down our throats

There’s also a third “hero” feature: the Aura system. Unlike the upvote/downvote or like/dislike buttons in many social platforms that only serve the algorithm to push more content to you, Persumi’s Aura system keeps track of user’s content quality over time, and would punish the low quality content and promote high quality content using visual cues - lower quality content has a much lower contrast making them easy to ignore. In the age of social media, self-curating content becomes essential to keep a platform healthy, engaging and usable.

The Non-MVP Features, a.k.a. The Future

There are many features that didn’t make the MVP cut, most of these are value-added features that will eventually make their way into paid subscriptions - if Persumi gains enough traction to attract users who don’t mind paying for premium features.

A prime example of such paid features is ones that help users monetise their content, e.g. ad revenue sharing and paid subscribers (like Patreon).

I also have the ambition of building out Persumi’s features so it can eventually compete against the likes of LinkedIn and Tinder.

Wouldn’t it be better for the world to have a platform like Persumi that doesn’t focus on dark patterns and exploiting users? 😉

The Tech Stack

Over the past decade or so I’ve mainly worked with two tech stacks: Ruby and Elixir. So naturally, Persumi was going to be built using one of them.

After some consideration, I’ve decided to go ahead with Elixir, the main reasons were:

Elixir and Erlang/OTP support distributed systems out of box
I’ve been writing more Elixir than Ruby lately, so I’m more productive in Elixir
I really wanted to try and use LiveView in production
I prefer Phoenix’s application architecture more than Rails’

On top of Elixir, I’ve decided early on a few other things to go with it:

Tailwind for CSS
Postgres for database, preferably a serverless option
A search engine
An easy to maintain infrastructure that doesn’t cost an arm and a leg

Elixir

I first discovered Elixir in 2014 while I was still actively involved in the Ruby and Rails communities, but it was two years later that I had the opportunity to really dive into it. I built a few open source libraries to help me learn Elixir and OTP:

Crawler - a high performance web scraper.
OPQ: One Pooled Queue - a simple, in-memory FIFO queue with back-pressure support, built for Crawler.
Simple Bayes - a Naive Bayes machine learning implementation. Hey, I was doing machine learning before it was mainstream! 😆
Stemmer - an English (Porter2) stemming implementation, built for Simple Bayes.

With a decent amount of experience building Phoenix web apps over the years, I unfortunately never had the opportunity to use LiveView…

I guess that’s finally changed now.

LiveView has been amazing, not only does it drastically reduce the amount of front-end code you have to write, make the entire web app feel super snappy, but it also virtually eliminates any front-end and back-end code/logic duplications. It is such an awesome piece of technology that improves both the user experience and the developer experience.

Petal Pro

During my initial tech research, I came across Petal Pro which is a boilerplate starter template built on top of Phoenix. It handles things like user authentication which almost every web app needs, but is somewhat tedious to build.

Petal Pro isn’t free but it ended up saving me so much time. I also started contributing small bug fixes and features to it too. If you are about to build something in Phoenix, check it out!

Tailwind CSS

As I kept progressing my career, there have been fewer and fewer opportunities for me to write front-end and CSS code. Last time I rebuilt my blog, I used Bulma - that was 2019. Since then Tailwind has gained a lot more traction, so I wanted an excuse to try finally give it a shot.

There is a debate on how many new things you should try for building your MVP - the more you have to learn, the slower your MVP progresses. That said, given CSS is reasonably straightforward, I figured it wouldn’t slow me down too much, if anything, Tailwind’s flexibility might just eventually make up any time lost in learning.

I’m happy to report that it is indeed true - by using Tailwind, it became significantly easier for me to customise my components and elements. I can see why it became so popular. It’s not for everyone, but I like it.

Postgres

Choosing Postgres as a database was a no brainer, given how popular and versatile it is. I did briefly consider NoSQL options like DynamoDB but quickly wrote them off as I needed an RDBMS to get things off the ground quickly, and the DB is unlikely to be the bottleneck for a long time anyway.

In Elixir, the Ecto library works wonders for Postgres.

Later in the post I’ll touch on how I deploy and run Postgres in production.

Search Engine

For a search engine, my requirements were:

The ability to search across multiple fields of a schema
The ability to rank them
The ability to have typo tolerance, word stemming and other similar language features to make search more intuitive
The ability to search multiple languages, including CJK (Chinese/Japanese/Korean) characters
Simple to run
Cheap to run

Using Postgres’ full text search was probably going to be the simplest but it doesn’t offer all the functionalities I need without a bunch of set up and manual SQL queries so I didn’t pursue it.

Elasticsearch on the other hand, offers good search functionalities but takes a bit of effort to set up and maintain, and can be costly to run.

After doing some more research, I found the following three options that would fit my needs:

Both Meilisearch and Typesense are open source, with commercial SaaS offerings, whilst Algolia is SaaS-only.

It’s been an interesting journey. I started with Typesense as I liked what I read, but I quickly discovered that it doesn’t search Chinese characters properly.

I then turned to Meilisearch. I especially liked the fact that they offered a generous free tier SaaS to get you off the ground running. Spoiler: during my implementation they did a bait and switch and removed the free tier.

At the time the Elixir support for Meilisearch wasn’t up to date, so I ended up contributing to a community library to add the features I needed.

Curious timing, after Meilisearch removed their free tier, I discovered that even though they officially support searching for Chinese characters, the implementation wasn’t perfect. I found some edge cases where characters weren’t detected properly, making the search results unreliable.

So, my last hope was Algolia. Despite them being the more expensive option out of the three, it does offer a free tier. It turns out, their search results for Chinese characters were much better than Meilisearch’s. Luckily, re-implementing the search from Meilisearch to Algolia didn’t take too much effort, it was pretty much done in one night.

Infrastructure

Early on during the development I’d already determined I wanted to try Fly and Neon, for web and DB, respectively.

I am in no way associated with either company, I was curious about Fly due to its tie-in with the Elixir community (Phoenix Framework’s author Chris McCord works there), and Neon due to its serverless nature.

Globally Distributed Infra

With Fly, the infrastructure automatically becomes globally distributed as soon as I started provisioning servers in more than one region. As of the time of writing, Persumi is deployed to US West, Australia and EU.

Despite being simple to use, making Fly work initially actually took quite a bit of finessing due to its incomplete official documentation and flakiness. Some of the services were having issues during the course of my MVP development. Worse, they don’t report (or sometimes even acknowledge) the issues unless they are region-wide outages. To this date, I believe their blue/green deployment strategy which was recently introduced, is still buggy, I often have to use their rolling deployment strategy instead. Deployment logs were provided to Fly but I think they’re too busy with other things…

Still, I’m sticking with them for now due to the ease of use after the initial hurdle, and their globally distributed infrastructure without asking for my kidney.

To augment Fly’s web servers, I also use Cloudflare’s CDN as well as R2 to serve asset files and audio files.

Funny tangent, initially I used Bunny for asset files and CDN, as I misread Cloudflare’s terms and thought I couldn’t serve audio files from Cloudflare. Bunny worked okay but their dashboard for some reason was painfully slow - not a good look for a CDN company. Like the search engine switch, it didn’t take me too long to switch over to Cloudflare.

Serverless Postgres

There are a few options to run Postgres:

Run on a standard server for maximum portability, but it requires more server maintenance overhead
Run on AWS RDS/Aurora or a similar managed service, easy but can be costly
Run on a serverless option such as Aurora Serverless or Neon

For my use case, I think option 2 or 3 are better fitting. As I mentioned earlier, I started the experiment with Neon.

Neon worked well initially, until I started deploying Fly instances in multiple regions. Due to Neon being only available in one region (I chose US West), and I live in Australia, the round trips between Fly’s Australian instance and Neon’s US instance were a show stopper - especially when complex DB transactions were involved. Actions sometimes took seconds to complete, yikes.

Despite Fly not offering a managed Postgres service, I ended up trying it anyway due to its distributed nature. After incorporating Fly Postgres in the app, all DB operations immediately became more responsive. Paired with LiveView, it feels like running the application locally.

The current Persumi infra looks like:

1 x Fly instance in US West, always on
1 x Fly instance in Australia, auto-shutdown when there’s no traffic
1 x Fly instance in Netherlands, auto-shutdown when there’s no traffic
1 x Fly Postgres writer instance in US West, always on
1 x Fly Postgres read replica instance in Australia, always on
1 x Fly Postgres read replica instance in Netherlands, always on

With this setup, I think I’m quite happy with the cost and scalability balance - it costs ~$20/m to run, with the potential of both vertical and horizontal scaling with ease.

The Missteps

The search engine and CDN swaps mentioned earlier certainly took away some of my time, but they were nothing compared to a major misstep I encountered.

And that was: the choice of how machine learning is done.

Let me explain.

Machine Learning, and Inference

Even before I started the first line of code, I already painted a picture in my head on the machine learning needed: a TTS (text-to-speech) model that I could run inference locally on the instance.

The reason being I believed it was the more flexible approach to gradually improve the inference and therefore the end result by training my own AI models over time.

Given I didn’t want to rent expensive GPU instances, I opted for fast TTS models that could do near real-time inference on CPUs. I used Coqui TTS.

The resulting out-of-box audio wasn’t great, but I kept pressing on.

The show stopper came when it was time to deploy everything onto Fly. Due to Fly’s architecture (they deploy small-ish Docker images, < 2GB each, onto their global network), I struggled to keep the Docker image file small enough to be able to deploy. With Coqui TTS, I would need Python and all the dependencies that resulted in a Docker image around 4-5GB in size.

With my tunnel vision, I then chose to offload the entire Python and Coqui TTS dependency tree onto Fly’s persistent volumes. I knew it wasn’t a great option, as that meant my infrastructure (other than the database) was no longer immutable.

Sometimes it’s necessary to take a step back, re-evaluate, and then press on in a different direction. Which thankfully I did.

The new direction is quite simple really: instead of performing inference locally, use an external service instead.

After doing a quick comparison between the offerings from AWS, Azure and GCP, I ended up using Google’s TTS. Honestly I think I would’ve been happy with any of the options, they all seem to have decent neural based TTS.

In hindsight, these giant corporations have much more resources and expertise to train better models than I ever could on my own.

The end result:

The TTS sounds significantly better than before
It’s just as cheap to run (Google offers a certain amount of free TTS API calls per month)
It no longer needs complex Python calls and FFmpeg calls to make local TTS work
The Fly infrastructure is simple and immutable again

In hindsight, I never should’ve even entertained the idea of running ML locally on CPUs, no matter how simple and efficient a model might be.

That said, with TTS, it wasn’t as simple as just calling the APIs and getting the perfect resulting audio back. Some pre and post processing were needed, but that’s a topic for another time.

More Machine Learning

The cherry on top - now that Google’s APIs were integrated into the app, I ended up also using Google’s PaLM 2 to do text summarisation (it was initially done locally too) as well as for a ChatGPT-like AI prompt service, to power Persumi’s AI writing assistance feature.

The Closing

If you read this far, thank you! I hope you enjoyed reading (or listening) to this post. Please look around and kick tyres, I would love your feedback on how to improve Persumi.

]]>

Welcome to Persumi - A Modern Platform for Content Creation

2023-07-17T04:18:07.386547Z

Like many software engineers and indie hackers, I dream of and experiment with different product ideas every now and then.

Five years ago, in 2018, I toyed with an idea where personas and expressions are the enablers for showcasing people’s talents and interests.

I called it, Persumi.

The Origin

The original idea was to break the pattern of modern day social media platforms like Facebook, Instagram and Twitter, where every person is hidden behind a two-dimensional facade. Words and photos are often used to skew the way people see who we are.

After a while, everything becomes bland as information is being overloaded and people’s attention span starts to decrease drastically. The signal to noise ratio is so low that more and more algorithms and tracking are invented to help curate content so advertisers would keep spending their dime on those social media platforms.

It’s a vicious cycle. There is now even a term for it: enshittification.

Enshittification is when an online platform becomes more monetized and less user-oriented the longer it lasts.

The Goals

With Persumi, personas and expressions are the key concepts to modernise content creation, curation and consumption in a simple yet meaningful way.

I set out a few goals for Persumi, in the hope to change the social media landscape for the better:

Find ways to encourage the creation and consumption of long form content
Make it easy for people to create vastly different topics without confusing their audiences
Make it easy for people to follow different topics, rather than relying on following users
Allow people to decorate their pages with their own personality and style via HTML + CSS, bring back MySpace!
An API-driven, open marketplace that allows developers to build and extend advanced functionalities

Some example use cases outlined in my original “pitch deck”:

social personas for friends, family, work…
interest personas as a gamer, traveller, photographer…
functional personas for travel plan, marketing campaign…

It kind of makes sense, but there is one problem.

The Problem

The age-old “chicken-and-egg” problem of starting a platform that thrives on user-generated content - how do I get early-adopter users without a decent amount of users and content already?

These goals might be nice, but they probably aren’t enough to convince users to drop what they are used to for something experimental and without users or traction.

So I parked this idea…

Revisit the Problem

A few years have gone by, and with a whole pandemic thrown into the mix, this Persumi idea has all of a sudden awakened me yet again.

In early 2023, after wrapping up my last freelancing project I decided to tackle Persumi’s chicken-and-egg problem once more.

I reflected on how my own content consumption behaviour has changed over the past few years. Instead of reading physical books, e-books or blogs, I consume more and more podcasts and audio-books. Instead of consuming content fully concentrated, I often consume them while on the go or during my workouts.

Then a light bulb moment occurred - I don’t need to build a massive social network platform to compete with the likes of Twitter. I just need to build a “simple” blogging platform for my own use and my own content. If I find it useful, other people will find it useful too. It doesn’t need to go viral, it just needs to be steadly discovered by like-minded people who share my frustration and vision in content creation and consumption.

Iterate the Idea

With a few more years of experience of leading teams and building products under my belt, I thought about how to iterate on the original idea, and came up with three main “twists” if you will:

Long form text content needs to be consumable in audio form, to suit the modern lifestyle
An API-driven open marketplace is way too ambitious for a new platform, build the core concept first, then iterate
An aura system that helps in both curating content as well as driving away bad actors

The rest of the original goals are still as relevant as before. In hindsight, these are pretty obvious product direction pivots, the result is an MVP that builds on a simple core concept - to turn long form text content into audio. And to make it easy to organically grow the platform and audience.

Build a Sustainable Business

With the chicken-and-egg problem behind me, it’s time for me to think about how to build a sustainable business to support my product ambition.

Interestingly, in Feb/March 2023, I was thinking about what the VC funding model did to the social media platforms we all love and hate, and why it’s a bad idea. The fact that then during the short few months of developing Persumi, both Twitter and Reddit seemed to be actively imploding themselves just proved my initial hypothesis:

To build a truly user-oriented platform, it needs to be built for users and content creators, rather than investors and advertisers.

That means, no obnoxious ads, no annoying pop ups, every visitor can view content without being forced to sign in, etc, etc… We do what VC-backed companies don’t.

If you’ve seen the landing page then you probably would have noticed a few things either available now or later to help create more value for users, such as:

The hero feature - AI audio, will be a paid feature, however, every user will be able to listen to audio posts, as well as to preview their own content with audio
Advanced expressions such as HTML and CSS for those who want to customise their pages to their own style
Ad-revenue share and paid subscribers - coming later, to make Persumi a platform that shares its success with content creators

Keep on Building

The MVP was essentially built in three months in my spare time. It wasn’t easy juggling between having a day job as a CTO, and moonlighting on my side hustle every evening and weekend.

This is just the beginning. I have many things planned for Persumi, some of which are already mentioned on the landing page, but there are much more to come. Early alpha and beta users are also able to help shape what this platform can become.

In the next week or two I will write about how I managed to build Persumi in three months - the tech choices, architecture and feature prioritisation I went through to make this product a reality.

Like what you’ve read or listened to? Why not sign up and try the platform yourself, the Pro plan is free during the alpha and beta testing stages. Also check out Persumi’s founder @fredwu’s profile here to see what personas and expressions look like and can achieve.

Update: Read about How I Built a Mostly Feature-Complete MVP in 3 Months Whilst Working Full-Time, it covers the features, the tech stack, the global infrastructure and the machine learning…

]]>

Tips for Job Interviews as a Junior Software Developer

2020-09-27T05:56:03.000000Z

As coding bootcamps such as Coder Academy and General Assembly churn out more and more software developers, and as more and more people start to realise the importance of software, companies these days are facing an increased amount of candidates applying for junior dev roles.

Recently we had to take down our job ad for a junior full stack React and Elixir role only a few days after posting it due to having received about 300 applications. Suffice to say, the competition is fierce at the entry-level end of the software engineer spectrum.

Software is eating the world. - Marc Andreessen

In the past three months alone I’ve had to sit through over a hundred interviews, many of which were for junior roles. Some candidates did really well but unfortunately the vast majority did quite poorly. In this article I am hoping to offer some tips and insights as a seasoned interviewer and hiring manager, to those who might be struggling landing an entry-level software engineer role.

I’ll break the tips into three sections: before an interview, during an interview and after an interview.

Before an Interview

This is arguably the most important stage of your interview - before it happens! Prepare yourself with enough knowledge and confidence in both software development and conversational skills in order to ace the interview and stand out amongst a sea of candidates.

Study Software Engineering Fundamentals

I get it, bootcamps and many online tutorials and courses focus on teaching the practical things to turn code into products. But just like running or even walking without correct postures, it can be incredibly dangerous over time.

My recommendation is to read on the basics of object-oriented programming, functional programming and some design patterns. For instance, build up some basic understanding of how the JavaScript prototype works or how the Ruby object model works can be extremely beneficial in progressing your technical capability on application design and architecture.

Get More Project Experience

Not having any substantial project experience on your CV is a sure way to significantly lower your chance of getting an interview.

Do junior developers really need substantial project experience, you may ask? Abso-fucking-lutely!

Build your own side projects, participate in open source projects, pick up some freelancing work - the choice is yours. These won’t be your most technically accomplished work, but they will help you accumulate experience, and more importantly to demonstrate your capability to your interviewers with real world experience.

Why Software Development?

I want you to put your hand on heart and answer this question honestly. If your answer is more money or FOMO, that’s okay, but be prepared to be disappointed with the reality - if you are not in this field because of your passion for building products or solving problems, you will find this a long and hard slog.

If you are passionate about building software, then let me tell you this - when you are starting your career you will have to give up your work life balance for a bit. That is, if you want to boost your career and progress quickly beyond being a junior developer.

My interest is in the future, because I am going to spend the rest of my life there. - Charles Kettering

Looking back at when I started my software career many moons ago, I never regretted giving up my social life early on in my career: I worked extremely hard, often did many jobs - a day job as a dev, a night job as a dev, freelancing and doing open source work all at the same time. Some would probably frown and say that I had no life, but I choose to believe that I was making sacrifices so I can become good at what I love doing in the shortest amount of time.

Research the Company You Apply For

If you are one of those who spam your CVs to every job opening, please consider changing your tactic. A well written cover letter will not only increase your chance of getting an interview, but also allow you to ask relevant questions during the interview.

Check out the company’s website and any social media pages on LinkedIn, Facebook and Instagram, etc. Do a bit of “stalking” on the people who work in the company, sometimes this will give you some insights into the type of work environment and culture the company has.

Doing enough research is not only for your own good - knowing whether you will enjoy working there should you end up getting a job offer, but also demonstrates your ability to do research and independent thinking to your interviewers.

Read, Listen and Communicate

This one might not be immediately obvious because it’s very high level and somewhat vague, bear with me.

Some people might be deep thinkers but really struggle to get their points across, and to communicate their ideas and thought process in a clear, concise and useful way. My advice is to read blogs, listen to talks and learn from other people on how they communicate. When you read and listen, pay more attention to how sentences are structured and how titbits of information are given.

The most powerful person in the world is the story teller. - Steve Jobs

And don’t forget to practice! Get comfortable talking to others on various topics in software development.

During an Interview

Hopefully the tips given thus far would help you secure an interview. Now it’s show time!

Turn Up On time, Have Equipment Sorted

Whether you are attending an on-location interview, or more likely during Covid, an online interview, make sure you turn up on time!

If it’s an online interview, also make sure you have whatever software needed (Zoom, etc) installed and tested. I’ve had a few occasions where the first five minutes of the interview was spent on the candidate trying to get their mic or speaker working - it’s not a huge deal, but it does show the lack of preparation and organisational skills.

A Smile Goes a Long Way

Understandably, interviewing is stressful. However, remember to put on a smile when you greet your interviewer and when you do small talks. Everything needs a balance of course, when you’re giving technical answers it’s probably not a good idea to smile.

If you know you get uncomfortable talking to strangers or when under pressure - do more practices! If you can’t find a friend or don’t have a partner to practice with, use a mirror!

Learn to Say “No, I don’t know”

When you don’t know something or can’t recall the details of something, just say it. As an interviewer, when I ask you a technical question I can see through your hmms and ahs. A “sorry, I don’t know” is a much better answer than one that doesn’t make sense.

Bonus point: “Sorry I don’t know, but I will look it up after this interview.”

Read Social Cues

This one can be extremely easy or extremely hard depending on your personality. For those who might struggle with reading social cues, try pay more attention to what type of questions you were asked.

If it was an introduction of yourself to kick off the interview, make it more than just two sentences but also under five minutes - this is usually a warm up for you to find your feet and calm your nerve, and for the interviewer to extract any interesting information as a discussion point later. It doesn’t have to be your memoir and you shouldn’t robotically talk through your entire education and work history.

If it was a technical question, always try to time-box it. Don’t drag on and repeat yourself just because you have already answered it quickly and concisely. When in doubt, ask the interviewers whether they have anything in particular they were looking for.

If it was an open ended question, try to explain your thought process as you explain it. As an interviewer, I’d hate to ask an open ended question and get a short yes/no back. At the same time, give interviewers room to interject and ask follow-up questions by pacing yourself.

Don’t Speak Too Fast

Okay, you might be very nervous, or English might not be your first language, so you tend to speak fast to “get through it”.

If that’s the case, work on your speech pattern, make it concious enough that you will recognise it when you speak too fast. If interviewers can’t get your points clearly, they might not always ask you to clarify - because if you kept doing it repeatedly, you would already be written off as a suitable candidate.

Be Humble

I totally understand that in a competitive job market you want to present yourself in the best light. Just be confident and speak the truths.

As someone who’s been doing interviews for years, I really dislike people who overstate their capability either on their CV or during the interview because these are often very obvious. Claiming to be an Elixir expert but not knowing how the supervision tree works is a sure way to score a black mark, for example.

Show Passion and Drive

To counter the point on over-selling yourself, what works better is to show interviewers your willingness and determination to learn. Remember, you are still a junior therefore we don’t expect you to know everything we know.

What we do expect, is your commitment to learning - not by saying it, but by demonstrating it. Remember the earlier tip on getting more project experience?

Ask Questions

Usually towards the end of the interview, we’d ask you if you have any questions. Don’t say “no”.

Doesn’t matter how much research you’ve done on a company, there are always insights someone who’s working at the company can give you. How do they manage the delivery? How do they support learning? What makes them enjoy working at the company?

Use this opportunity to gain insights into why you should be working at this company, and to demonstrate your interests in working at the company to the interviewers too.

After an Interview

After completing an interview, it’s an opportunity to do a little retrospective and do a follow up when necessary.

Close the Knowledge Gap

Don’t over-analyse on what you could have or should have said. Instead, spend your energy on solidifying the topics you couldn’t or didn’t provide good answers for. If you’ve said “don’t know” or “not sure” during the interview, now is your chance to make it “sure”.

Follow Up

If during the interview there have been things mentioned that you could provide links or more information on, be sure to act on it. You never know, maybe the clarification is just what the interviewers needed to help them make a decision.

To all the junior developers out there - enjoy the journey! You are in for an amazing ride, make every second count.

If you enjoyed this article, checkout my other tips articles:

]]>

Tips for Writing a Good CV / Résumé

2020-05-20T08:59:24.000000Z

Due to COVID-19, not many companies are hiring at the moment. The company I work for therefore is in a very fortunate position to still be thinking about growth and hiring.

As a hiring manager for almost a decade now, I’ve personally reviewed thousands of job applications and CVs, and many hiring managers would probably agree, the vast majority of CVs are terrible. Let’s change that!

During COVID-19 where more and more people are either losing jobs or having their work hours cut, we are experiencing an increased amount of applicants to our job ads. I’d say on average I spend about 30 seconds per applicant due to my busy schedule - most hiring managers are busy people, it is therefore crucial for candidates to realise the importance of having a CV that is clear, easy to read and most importantly sells yourself. And if you have a cover letter, which I highly encourage that you do, congratulations you just bought yourself another 30 seconds. ;)

I’m writing this post mostly from my own perspective - as a hiring manager in a tech company in the western culture (we’re based in Australia). Understandably, different cultural backgrounds and regions may have their own conventions, but certainly in Australia and many similar western cultures, there are things that you do and don’t do on a CV, and there are things that may help your CV stand out. Let’s talk about these things.

At the end of this post I will also share a copy of my own CV to help illustrate my points.

30 Seconds? Surely It’s Unfair to the Candidates

Yes, I agree, to think that you are only given 30 seconds for your perhaps carefully crafted CV and cover letter is definitely soul-crushing. But it is unfortunately the reality. I work for a company where I can still do the first round of vetting myself, many large corporations would use algorithms and/or HR people to reject your applications based on keywords and other things.

Knowing the reality and the constraints, there are a few things I’d like to address in the hope of improving your CV and your chance of scoring an interview, and in turn, helping myself and other hiring managers out there to have a better candidate CV screening experience.

Have a Pronounceable Name or Alias

This one surely would raise some eyebrows - you might think that your name is your identity and you should not change it for anyone. True, however, the reality is that a hard-to-pronounce name discourages your profile to be shared and spoken about. Why not add a pronounceable alias if means there’s an increased chance of getting an interview?

For clarity, I personally would never reject a candidate based on their name (or their cultural background for that matter), but I know some hiring managers might, and for some of them, they are NOT doing it on purpose. However, I have on several occasions had to ask a candidate how to correctly pronounce their name.

A Short Blurb on Who You Are

As a hiring manager, I care about who you are as a person - if you can summarise who you are as a professional in a sentence or two, it will help me determine whether you might be a good fit or not.

As an example, here’s a blurb about me:

A passionate and hands-on software executive with two decades of experience and an entrepreneurial mindset.

A long time open source developer who has created and contributed to a few dozens of projects, including Ruby on Rails.

In two sentences, I explained my industry experience as well as my open source contributions - two things that help define who I am as a working professional. It also invites more questions from hiring managers: what kind of things have I done as an entrepreneur; what other open source projects have I contributed to?

Work Rights

Many companies have restrictions or policies around who they can hire based on their residency and visa status. If you are not a resident or are on a particular visa, make it clear in your job application so you don’t end up wasting time for the employer and for yourself.

List Keywords, But Don’t Overdo It

In the tech space it is important to have keywords visible to highlight your skills. If you are a software developer, your tech stacks should be clearly stated in your CV. As a hiring manager, if I am hiring a PHP developer, I expect to see PHP mentioned in your CV. There are of course exceptions, for example when we were hiring Elixir developers I did not expect to see Elixir as a keyword simply due to the supply constraint.

It is a balancing act however - I’ve seen CVs where candidates put 20-50 keywords on their CVs. I’m sorry but unless you are extremely gifted, you cannot possibly be good at all those things. Do not put keywords on your CV simply because you’ve read an article on the subject.

Oh, and unless you’re going for a data entry role, I honestly don’t care about your Excel skills…

Do Not Overstate Your Capability

Similarly, try to avoid overselling your capability. I once interviewed a candidate who claimed to be an “expert” on Ruby. We were actually hiring for a non-Ruby position, but given the candidate’s CV, I questioned him on some advanced Ruby subjects during our interview and he struggled all the way through and was sweating bullets. Suffice to say that he did not get the job.

Be confident, but also be honest and be humble. Lying on your CV to get an interview is a waste of everyone’s time.

Keep Things Short

As I mentioned in the beginning, I spend on average 30 seconds on each CV. Keep things short and easy to read! I really don’t care about how awesome you were in your last dozens of projects - these will get covered during interviews.

On a CV I expect short and concise blurbs on what you did in each role. Also, take recency into account too - if you’ve been working in the industry for a decade or two, what you did 20 years ago really doesn’t matter as much, so save yourself some time and cut things short.

For example, here’s the blurb for my current role:

Leading a department of 25+ engineers to make great child care and education software. As part of the leadership team and reporting to the CEO, helping building and turning the company into a market leader.

And here are the blurbs for my older roles:

Yes, the blurbs for my older roles are left empty intentionally.

Now, again there are exceptions. If something happened a while ago but is interesting and relevant, do tell! For example, here’s the blurb for my oldest “role”:

Built my first ever website using Microsoft FrontPage Express, on a Pentium 166Mhz computer, uploaded via a 33.6kbps modem.

Explain Over-Qualified Titles

There were a few times where a “CTO” or even a “CEO” applied for a developer role. In most cases it wasn’t about over-qualification, but about what the candidate wanted to achieve professionally. So, either in the CV or in the cover letter, explain what you are looking for in your next role, otherwise you run the risk of being assessed as over-qualified.

Spare the Personal Details That Are Too Personal

This is predominantly a culture thing as I’ve seen it from mostly candidates of certain cultural backgrounds. I really don’t care about your age, gender, marital status or favourite sport. These things do not define who you are as a professional - we might talk about your favourite sport and food during the interview but they are irrelevant on your CV.

Space Things Out

Look up CRAP Principles - make sure your CV has enough white spaces and contrast, and has fonts that are readable! Scrolling through walls of text is no fun and is a sure way to get your CV dismissed.

2-4 Pages

This is not scientific, for me personally I prefer to see CVs of 2-4 pages. Use the length as a constraint to cut things down. There were several occasions where I ran into CVs with 10+ pages. I guarantee you, unless a hiring manager is extremely bored, he or she does not have time to read your War and Peace.

PDF Over Word

When possible, submit your CV in PDF format instead of Word format. Now, sometimes if you use a recruiter you’ll be asked to submit your CV in Word format so they can ~~fuck it up~~ add their branding. A PDF formatted CV ensures the correct formatting and layout always get shown to the hiring managers.

Cover Letter

Always attach a cover letter when possible, but keep it short too. Given the amount of CVs a hiring manager needs to go through, having a crafted cover letter is another way to grab their attention and increase your chance of getting an interview.

Don’t repeat the same information in the cover letter though. Your CV is about the facts of your experiences, your cover letter should be about your thoughts on why the company should hire you. Focus on the value you can bring to the table.

Find A Referral

When possible, find someone who can refer you. A referral gets preferential treatment during the CV screening stage and does not suffer from the same 30-second fate.

Pleasing Design

This one is a “nice-to-have”: if your CV is really well designed, you would earn another 30 seconds of my attention. ;)

These are the main points, hopefully they are helpful. To help illustrate, here is a copy of my own CV, with contact details removed.

If you enjoyed this article, checkout my other tips articles:

]]>

Tips for Becoming a Better Software Developer

2020-05-03T06:34:25.000000Z

Over the past decade or two, both as a software developer and as a manager I have accumulated a few tips for becoming a better software developer.

For context, I’ve been working as a software developer for more than a decade, mostly in small product-based companies; over the last decade or so I’ve worked as a team lead or manager in various capacities; and I’ve moonlighted as a consultant, freelancer and open source contributor in-between. Check out my LinkedIn profile and Github profile if you are interested.

Now, let’s break down the tips into three main areas: mindset, technical and people. The goal of this article is to raise awareness of the topics mentioned, if you want to understand more on a particular topic I encourage you to do your own research.

Mindset

Have a Growth Mindset

This one hopefully should be obvious, if you haven’t heard of it, please read more here.

Have a StarCraft Mindset

This one is “new”, I’ve given a talk that includes more details on what it means, check it out here. In essence, it’s a mindset that requires you to always consider things around you, and forces you to think beyond your narrowly focused task at hand.

Recognise the Dunning–Kruger Effect

I once interviewed a mid-level developer who rated himself as a 9.9 out of 10. The Dunning-Kruger effect is essentially the opposite of the Imposter Syndrome, both are very common in our field.

Agile is Not a Process, It’s a Mindset

Some people are overly obsessed with the processes of “being agile”. In truth, every organisation, every project and every team is different, there is no point to follow a set of processes that might slow you down and not add any value. Discover what being agile means to your team, your project and your organisation is an ongoing effort and should be treated as such.

Different Organisations Require Different Engineering Approaches

Some projects and organisations require tech-driven decisions be made upfront, some on the other hand require product-driven decisions instead. Knowing how your organisation operates and what it values plays a key part in understanding how software development should be approached.

A Sense of Entitlement Only Gets You So Far

Let’s be honest, as a software developer we are extremely lucky to be in a high demand field. As a result, many have started building a sense of entitlement - some ex-colleagues of mine even complained about the lack of tissue boxes in the office. Starting and running a business is extremely hard work, let’s all appreciate what we have before throwing a tantrum.

9-5 is Not Evil, Nor is Crazy Work Hours

Every now and then I see conflicts between people who share different philosophies on what work and career mean to them. We should embrace the different ways people prefer to spend their time and energy on, rather than force our own ideology on them. Discovering and knowing the difference between working at a giant corporation versus working at a small start-up is key to our happiness.

Pain and Gain, There’s Always a Trade-off

An extension of the 9-5 vs crazy work hours: adjust your expectation based on the effort you put in. You can’t have everything, prioritising family life over career building is 100% okay, as long as you know what you are giving up on, vice versa. In my 20s, many of my similarly aged friends and colleagues would go out and have fun after work and on weekends while I was in front of my computer building open source projects, is an example of a conscious trade-off I made.

Speed vs Quality

Internalising the need for speed and the need for quality should always be on our mind. Pursuing only quality or only speed will significantly limit your problem-solving capability as well as your career progression.

Think, Then Do

It’s a balance - don’t overthink it, but also don’t do without think first. When in doubt, ask more experienced people for advice and guidance.

Technical

Avoid Second System Syndrome

I’ve seen Second System Syndrome over and over again throughout my career. Some people always assume a rewrite is a much better approach than alternatives, more often than not though, it is not the case.

Micro-Services is Neither New nor the Holy Grail

There’s the age-long debate of monolith vs micro-services architecture, and SOA (Service-Oriented Architecture) has been around for decades. Each has its own pros and cons, don’t get bought into the hype because a blog post says so, or a particular product and company found it successful. Use the right tool for the right job.

TDD (Test-Driven Development) is not the Holy Grail

Just like micro-services, TDD is not the holy grail and should not be treated as much. Chasing the TDD dream without knowing its pros and cons is just as counter-productive as not doing TDD when necessary.

Code Aesthetics Matter

A lot of people treat coding as more of a scientific endeavour: they focus purely on the algorithms and results. Software development to me is a creative endeavour, and I often use white spaces and the general look and feel of the code base to determine how well-organised a project is - deeply nested code and long functions for instance are obvious ways to determine potential code smell.

You’ve all heard of the SOLID principles, but have you heard of the CRAP principles?

Read As Much Code As Possible

Don’t get me wrong, writing code is important, practice is important, but time and time again I come across developers who clearly have little exposure to established patterns and conventions.

Tech Stack Exposure and Diversity are Always a Good Thing

No matter whether you are in a highly specialised field or being a generalist, having exposure to different tech stacks and paradigms is always a good thing - it widens your field of vision and increases the boundary of your technical understanding. Don’t be the guy or gal who is known as “a \ developer”.

Have a Deeper Technical Understanding Helps

This is strictly speaking a “nice to have”, but I lost count of the number of candidates I interviewed who claimed to be a Ruby expert yet cannot explain the Ruby object model and have never heard of eigenclasses, or senior developers who cannot explain the difference between git merge and git rebase. Sure, you may not need the deep understanding in order to do your job, but it will certainly help!

People

Always Assume Incompetence over Malice

If you haven’t, read more about it here. Knowing this, and paired with having a growth mindset, you should therefore be encouraged to teach, educate and influence, rather than being defensive and participate in “us vs them”.

Truth = Visibility x Tolerance

This is another topic covered in my talk. If you find yourself regularly dissatisfied with the information you receive, ask yourself: what information is and isn’t available to you, and are you sure you have the stomach to tolerate the new information?

Rapport and Productivity are Built Over Time

Some people assume a team’s productivity is determined largely by team member’s individual capability, some also assume a 10x engineer is a reflection on one’s technical capability. It’s not. Productivity evolves over time - building rapport with those around you is often the unsung hero in productivity.

Champion Team Over Champion Individual

This one hopefully is obvious, but I just have to emphasis its importance. Please, if you struggle to work with your team, ask for help! If you have a team mate who is difficult to work with, reach out and help them!

Final Thoughts

Software development is evolving rapidly, some fundamentals however are always useful and relevant. I hope you find this article useful.

I’ve mentioned my talk a few times, but if you haven’t already, check it out below, it covers a few topics mentioned in this article, and some more. Also check out my tiny Youtube channel for more tech talks I’ve done in the past.

video: https://www.youtube.com/watch?v=MBczdO7RgNo

If you enjoyed this article, checkout my other tips articles:

]]>

Launching Focussist Landing Page - An Upcoming Agile Project Management Tool

2020-03-23T11:04:55.000000Z

Over the past few years as I gain more and more experience in not only building software and products but also in leading teams and projects, I realised that my desire of wanting an agile tool that’s super simple to use yet super flexible to adapt to different needs has grown stronger and stronger.

Given the COVID-19 situation that’s going on at the moment, I’ve finally decided to spend a few nights working on a pitch deck for myself to validate the value proposition, and some high level wireframes to visualise the ideas.

Over the weekend I’ve put together a landing page to “market” my ideas. It’s a quick job, but the underlying thinking has been on the back of my mind for years now.

Introducing Focussist, or rather Focussist’s landing page: Focussist.com.

Please check it out and give me your feedback, and don’t forget to sign up. ;)

]]>

Conference Talk: Adaptable Human @ RubyConf China 2019

2019-09-14T08:05:09.000000Z

Seven years ago in 2012 I spoke at RubyConf China 2012. It was a technical talk on how to become a better developer, if you’re interested you can check out the video recording.

Seven years later in August 2019, I headed back to Shanghai again to speak at this year’s RubyConf China. And this time around, it was a non-technical talk on how to develop one’s career. Check out the video recording below:

video: https://www.youtube.com/watch?v=MBczdO7RgNo

And, here are some photos of yours truly. :)

]]>

New Blog with New Design and Gatsby.js - Loving JavaScript Again

2019-01-20T11:27:22.000000Z

In 2010 I set up this blog on Tumblr mostly due to the ease of publishing and not having to worry about the hosting. I also went through two design iterations done in Photoshop:

You can probably tell that these designs were done before the flat and minimalist design trend we are seeing in recent years.

As I decided to revamp my blog once again, I thought I’d use Sketch this time around and aim for a simpler, cleaner and more mature design approach that’s quicker to design and to build. And this is exactly what I did:

Thanks to Sketch, the design process was much better than I would’ve had in Photoshop.

Why the Revamp?

A big part of the reason for the blog revamp was to move away from Tumblr, now that it is part of the dying Yahoo!, and to invest in modern technologies to make the blog function better by adding responsive design and a proper grid system, etc.

As I was reading up on and researching technologies as I often do, I came across Gatsby, and I was immediately convinced that this was going to power my new blog as it appears super fast (by building static content and pre-fetching), is based on React and has a huge collection of useful plugins.

At work I often joke about how I hate JavaScript (rm -rf node_modules anyone?), the truth is I started off my career doing a lot of JavaScript, my most popular open source library by GitHub star count is in JavaScript, and I have always worked as a full stack dev until recent years.

Due to the shift of my responsibilities more into leadership over the past few years, I have opted to focus more on the backend, but deep down I love software in general and given the frontend scene has seen some major improvements in the past few years, not to mention that at work we have several React code bases, it’s time I dig my teeth into it again.

Disclaimer: until recently I had only done two React projects before and my exposure to ES6 has been somewhat limited.

Gatsby

The thing about Gatsby that impressed me the most is how fast it is due to its out-of-box support for pre-fetching. With a wealth of different starter templates it is also very easy to hit the ground running. I used gatsby-starter-netlify-cms as it supports Netlify and is fairly simple.

Export Tumblr Posts

First thing first, I will need to export all my tumblr posts into Markdown files so that Gatsby can consume them. There are heaps of such tools including Tumblr’s own export functionality. I ended up having to use these two:

Jekyll importer for converting all Tumblr posts into Markdown
tumblr-utils for downloading all images in the posts (Tumblr’s own export tool and many others only download images hosted on Tumblr)

Of course, many search-and-replace as well as manual massaging were required to get the new Markdown files to a useable state.

Add More Gatsby Plugins

The starter template was great, but I needed more functionalities so I ended up adding these over time:

gatsby-plugin-styled-components for supporting Styled Components
gatsby-remark-embed-video for embedding videos
gatsby-remark-prismjs for syntax highlighting
gatsby-awesome-pagination for pagination
gatsby-remark-reading-time for adding reading time
gatsby-plugin-feed for RSS feed
gatsby-plugin-google-analytics for Google Analytics
gatsby-redirect-from for page redirects (of old Tumblr pages)
gatsby-plugin-sitemap for sitemaps
gatsby-plugin-robots-txt for robots.txt

Using many of these plugins only requires you to add the plugin itself and some config, then off you go - it could not be any easier!

GraphQL

One other reason why I was interested in Gatsby was because of its out-of-box support for GraphQL. I came across GraphQL a while back but never had the chance to work with it. At work one of our projects started using GraphQL and the team behind it rates it highly, so I really wanted to get some first hand experience using it.

With the inclusion of GraphiQL the in-browser GraphQL IDE, the GraphQL experience in Gatsby is extremely simple and satisfying.

Querying Data using Gatsby’s Page Components and StaticQuery

It’s been a bit of journey.

The starter template I used has GraphQL queries in the page components, but I wanted the GraphQL queries in shareable components so I opted to using Gatsby’s StaticQuery.

Until I decided to add gatsby-awesome-pagination which requires the GraphQL queries to have variables therefore incompatible with StaticQuery.

Despite the back and forth between page components and StaticQuery, I still loved the GraphQL experience in Gatsby.

It’s All Just React (In a Good Way)

The great thing about Gatsby is that despite making things easier by including a bunch of useful functionalities and plugins, down to its core, everything is based on React Components.

What this means is that as long as you have a working knowledge of React and React Components, extending Gatsby’s capability is simple and straightforward.

CSS Powered by Bulma

The starter template also includes Bulma - a modern and popular CSS framework.

Until now I’ve always relied mostly on Bootstrap so it’s nice to get to experience another CSS framework. And it turns out, Bulma is quite nice and easy to use too.

Thanks to Bulma, supporting responsive design was a breeze.

Gotcha: CSS Rules and “gatsby-plugin-purgecss”

gatsby-plugin-purgecss is included as part of the starter template to reduce the size of the CSS files by removing unused CSS rules. It’s wonderful as it saves about 200Kb of CSS (before minifying and gzipping) from unused rules from Bulma.

One gotcha I soon realised though, is that it also purges any CSS rules that are only rendered in the Markdown components. My guess is that the purging happens before the static contents were rendered therefore it cannot detect them.

The fix was quite simple, given that by default it only purges the main css file (in my case, all.scss), and all the specific rules in this file were defined by me, all I had to do was to add the comment lines to tell the plugin not to purge these rules:

@import "~bulma";

/* purgecss start ignore */
.my-own-css-rules-here {
}
/* purgecss end ignore */

Gotcha: CSS Conflicts Between Bulma and Prism.js

One more issue with CSS was the conflicts between Bulma and Prism.js due to the way some of the CSS rules are defined in Bulma.

What I had to do was to “reset” those CSS rules so they can inherit from Prism.js instead:

.content {
  .number,
  .tag {
    align-items: inherit;
    background-color: inherit;
    border-radius: inherit;
    color: inherit;
    display: inherit;
    font-size: inherit;
    height: inherit;
    justify-content: inherit;
    line-height: inherit;
    margin-right: inherit;
    min-width: inherit;
    padding: inherit;
    text-align: inherit;
    vertical-align: inherit;
    white-space: inherit;
  }
}

Netlify: Deployment and Hosting Made Easy (and Free!)

Before Netlify and similar services were available, many people used GitHub Pages to host their static content sites. It worked reasonably well but in the end this isn’t GitHub’s core competency and the features are limited.

Netlify takes deployment of static content sites to a whole new level and I cannot be happier to have finally jumped on the bandwagon too.

Closing

Suffice to say, I am very satisfied with this revamp experience - Sketch, Gatsby, GraphQL, Netlify and many other software have made it a walk in the park.

It only took me around 50 hours to get to 99%, and a few hours more to add in things like Disqus and social sharing.

Oh, and for those who are curious, the source code of this blog is available on GitHub here.

Let me just say:

The last time I fell in love with JavaScript was when JQuery first came to the scene many moons ago. Ever since then I dread working on the JS stack mostly because I’ve been spoiled by the Ruby and Elixir ecosystems. It’s nice to fall in love JavaScript again after so many years of simply “getting by”.

I hope you like my new blog, and if you have a similar experience to share, I’d love to hear from you!

]]>