Part 1/3 | Uncertainty
By Sooraj Rajmohan
Indonesia — a constellation of 17,000 islands sprinkled between two oceans. A melting pot of ethnicities and cultures, home to the fourth largest population in the world, and ripe with business opportunity. This dazzling island nation is where GOJEK was born. Its chaotic streets were the playground where a scrappy young startup grew, and the love of its people saw GOJEK grow into a Super App.
For GOJEK, Indonesia is home. Comfortable. Familiar. Safe.
The story of a Super App, however, is about more than one country. It is about growing up, stepping out the door, and journeying into the unknown. This three-part series details GOJEK’s journey beyond Indonesia’s borders, and the story of the team that guided it through roads untraveled.
The mandate. The madness.
In late 2017, at a meeting of product leads at GOJEK’s Jakarta HQ, then COO Rohan Monga confirmed the rumour that had been floating around hallways.
We were going abroad.
On paper, the idea was simple: GOJEK was targeting new markets in Southeast Asia — starting with Vietnam, Thailand and Philippines.
The reason for choosing these geographies went beyond the obvious (proximity). These countries have similar dynamics to pre-GOJEK Indonesia. The population numbers are in the same ballpark, broadband Internet penetration is high (and growing rapidly), the food delivery market is nascent, and there was no dominant player in the digital wallet space.
The conditions were ripe for disruption.
The plan was to launch swiftly and stealthily, and take competition by surprise. Each targeted geography even got its own codename. Vietnam was nicknamed Vogue, Thailand dubbed Tatler, and Philippines — Fortune (notice a pattern?)
This was only the beginning, however. The plan was more elaborate.
It was important to recreate this feeling of pride in the brand in the new markets as well. To achieve this, it was decided to launch the new offerings with completely new branding — GO-VIET in Vietnam, and GET in Thailand. Everything about these new local brands was devised to emphasise the values of that country.
The goal: to build a local champion — one that the people, as well as commercial and regulatory agencies — could get behind. The vision was solid. Everyone was pumped.
But there was a problem:
Our existing tech stack was already buckling under the strain of ever-increasing demand, and would not be able to handle new geographies.
We would need a different solution. GOJEK had crossed the borders of the familiar, into the unknown.
One of the options that made sense was to decouple from Indonesia and create new environments for the new countries. This way, the scaling issues affecting Indonesia would not affect the new geographies, and GOJEK developers also got greater flexibility to customise and localise the app and products for the new markets.
But creating new environments/clusters (read mini-GOJEKs) came with its own set of complexities. This process also needed to be repeated in quick succession in each country.
The answer to this problem was using automation for cluster creation… but not on VM’s. To do something new, new technologies would have to be adopted. The path chosen was to use Kubernetes to automate cluster creation and service deployment on containers.
Unfortunately, no one at GOJEK had the insights or the bandwidth required to do this at the time. We had the vision. We knew what needed to be done.
We needed the right people.
Enter Team Trifecta
It all started with a Product Manager, who was tasked with putting together a group of people to accomplish the task ahead. About a week in, three QAs joined, followed in another week by an eight-member crack team who would develop the initial Proof Of Concept (POC). The members of this team were cherry-picked from across GOJEK product teams (with a fair bit of negotiation and bargaining involved ).
It would be this team’s duty to first run an efficient proof-of-concept (POC) for the launch, and then develop frameworks, create playbooks, do product customisation and automate infrastructure creation and deployment. The idea was to create a scalable model and replicate it in any new country GOJEK wanted to launch in.
International expansion was a big deal though, and an exciting challenge. As Trifecta’s work picked up steam, more people would put up their hands to work on this exciting new project, over and above their existing responsibilities.
While the core Trifecta team went about its business to create a POC, a specialist four-person team was brought in. Their job was to work in parallel on our big play — automate cluster creation.
We had a fellowship. It was time for the journey.
Trifecta’s first task was to figure out all the changes that had to be made to build a new app for each country. As soon as this project started, it became apparent that a long uphill climb lay ahead.
For starters, the driver app was never built in a way to support any other language besides Bahasa Indonesia. This had to be re-engineered so it could support localisation/i18n.
And then there was Stan Marsh.
Stan Marsh is a legendary name in the GOJEK mythos. It refers to our late legacy codebase. Stan Marsh was the beast of burden that carried GOJEK through the early years, and each new commit made to it was often a desperate attempt to douse a fire and keep GOJEK up and running. Understandably, a lot of legacy services and Stan Marsh were not designed to support GOJEK’s journey into new geographies.
Trifecta members found stumbling blocks everywhere — there was no provision for new currencies and time zones, for example. The pack mule was struggling to keep up with the party.
One thing was obvious — GOJEK’s entire mobile codebase had to be re-architected to be able to efficiently build multiple apps and designs on top of it.
Automation on Kubernetes would have to wait.
We had a detour to take.
Descending into the mines
First things first — Trifecta needed an environment to play around in and do the testing without affecting the BAU. The team dusted off an old User Acceptance Testing (UAT) environment, which was not in use at the time, and proceeded to do experimentation and development within it. Slowly, the GOJEK app was dismantled and rebuilt into a new avatar that would service new markets.
This turned out to be a lot more complicated than it first sounded (which is a common plot point throughout this story, as you will see ?). Major changes needed to be made to the driver onboarding portal, for instance, and the team needed to figure out ways to transfer money from the wallets of driver partners into their bank accounts.
GOJEK was always used to having multiple support teams, and well-established bank integrations to help with such matters. In this new territory, it all needed to be figured out from scratch. Add to the fact that GOJEK itself was rapidly evolving, making it difficult for the team to figure out which services should be transitioned and which would soon be deprecated, and the job became more complex. As Trifecta PM Sourabh Gupta puts it:
“It was like changing wheels on a moving car.”
While the main team focussed on transforming existing services and mobile architecture to build a working POC, a parallel team undertook the long and painful journey to understand inter-service dependencies that would need to be tackled.
Ironically, they were themselves depending on multiple GOJEK product teams to understand the dependencies of over 400 microservices. This was crucial in order to chart out a plan for automating the deployments and cluster creation.
Hours turned into days…
Documentation was scarce, and the team had to undertake the painstaking process of talking to service owners, and map out configurations. This also meant many GOJEK team members had to take time out of their daily routines to spend on knowledge transfer.
Identify, document, dismantle, develop. Over and over.
An emergence. A transformation.
It took some doing, but finally, there was a glimmer at the end of the tunnel. Trifecta emerged into the light of a new day with a finished POC. A relatively complete driver and consumer app that were worthy of upload to the Play store.
The deadline the team had given itself was three months.
It had only taken two-and-a-half.
Trifecta’s eight-member crack team had accomplished what it set out to do, and built a working POC. Their role in this journey was at an end, and so they disbanded and went back to their product teams. The original fellowship was no more.
The team working on dependencies however, was facing rough weather.
Sifting through those 100s of microservices was taking longer than expected, and the knowledge transfer meetings were slowly losing attendance. Progress was imbalanced, and the parallel approach was not yielding results. There was only one move that made sense: absorb the engineers into the core Trifecta team.
The birth of a new party for the next stage of the journey.
The constitution of the team had changed, but the goal remained the same. The first months of 2018 saw Trifecta work tirelessly to untangle the intricate dependency map of GOJEK’s microservices. The process was tedious, but it did have a silver lining. We now had a ton of documentation about GOJEK’s product and service architecture — a path had been cut through the woods for others on similar journeys.
Finally, a clear picture emerged.
It… looked a little like this:
Months of work, depicted in a single image. Complicated as it looks, this was a major breakthrough. The key to the codex.
Initially, it was estimated that in order to launch with two key products (GO-RIDE and GO-SEND) GOJEK would need to deploy 40 microservices for the new clusters. With each new dependency unearthed, that number kept inflating. Integration of GO-PAY services (which were required for maintaining a driver wallet, and to handle incentives and subsidies) brought its own set of complexities, as its services and tech architecture was modelled differently from the rest of GOJEK.
As the launch timeline drew closer, there were over 100 microservices to deploy. Not an easy task, but the team finally knew what needed to be done.
The fog of war was finally parting. Now, to build a working environment.
Gunning for the finish line
The work on automation and deployment scripts now continued as originally planned. The wheels were in motion again, and passing through terrain GOJEK knew well. It was time to build stuff.
OpsTech, Transport, GO-SEND, Marketplace, Driver Platform, GO-PAY and Platform teams provided backup, and the products for Vietnam launch were developed (we would launch with GO-RIDE and GO-SEND). The team was working efficiently, and as the cool grip of winter began to give way to the warm summer months, a basic environment was up for app development.
Then came the next wave of trouble.
As the environment took shape, our inexperience in dealing with new geographies came back to haunt us. Configuration mismatch across clusters and deployment drift reared their heads. Suddenly, there were more questions that needed answering:
1.Where would we manage configuration of services for all the new markets?
2. How would we deploy on containers and ensure that the same version of code was running on all environments?
3. What about logging and monitoring?
The goal seemed an eternity away, again.
Chaos ensued.
For the continued story of how GOJEK grappled with the challenges brought on by a new geography, read part 2 of this 3 part series.
As evidenced by this post, GOJEK is an ever-changing entity. Constantly improving and adapting to the circumstances prevalent in some of the world’s fastest-growing markets. We’ve been able to thrive thanks to two core principles: lean engineering, and hiring the best people and giving them freedom to innovate. We’re still looking for sharp minds. If you’ve got one, and answers to the questions like those Trifecta faced, visit gojek.jobs, and join us for the ride.
