I’ve been involved with two things these past few years that, in my mind, have very similar execution success characteristics: creating a startup company (SC) and a recommender system based on Supervised Learning (ML).
Looking at how to implement each from scratch reveals interesting parallels.
The Cold Start
The challenge that is faced when first implementing a recommender system (recsys) in a new environment is that the ML engine is not very smart about what items to recommend. This makes some sense in that without rich historical data about how an item has performed (e.g. purchases, shares, ratings) there can be no concrete statistic on an items success probability. Historical data can be analyzed and used to help train the engine but that data’s usefulness is quickly obviated as new information is gathered from the live system.
A specific lack of information means that out of the gate there are some postulates guesstimated and models built which are based on the best information available. It is well understood that these models will change, features will be dropped or added and a next generation model should be in the works as soon as possible.
The same issue typically faces a new startup. Not much is really known about the best plan for success. Which products will win users over or land clients with eager, open checkbooks? Forecasting revenues for a brand new venture seems nearly an exercise in mental masturbation. New products envisioned by the ambitious team have no proof points.
Much like the ML implementation, understanding that the initial products and business models are literally a jumping off point is important. Not betting it all on an unproven theory will save time, energy and money.
Learning And Adapting
The ML engine learns how to provide more accurate recommendations over time by getting feedback from the ecosystem. The engine will suggest a variety of items, based on statistical analysis, that it believes will match a users interest at a particular time and for a specific context. The results of this endeavor are collected as log data and are returned to the ML system. This data set represents the success and failure of the previous set of predictions. As real time passes, the ML engine consumes the latest available data sets and (if properly designed) the accuracy of the engine’s future predictions becomes more acute.
Likewise the model through which these data sets pass may need to be updated to reflect new learning. Testing a new model against the previous model to reveal its efficacy (A/B testing) is an absolute prerequisite. Only after a rigorous testing will you know if the new direction you have chosen will provide better quality or if you should stick with the status quo.
Building a company on a hunch or in a complete vacuum is a recipe for failure. As one of my favorite startup bloggers Steve Blank said in no uncertain terms, “Get out of the building”. It is imperative to canvas the market you are trying to address to see if the widget you want to produce has value.
Getting feedback about the direction you are taking the venture toward or the features that you want to add to the product will be worth more than any blue-sky session at a whiteboard. Being agile, testing and iterating quickly over product ideas and features as data comes in from the field will produce dividends and should help steer you toward a successful path.
It is important to the success of the ML project to establish KPIs before coding begins. These metrics should be meaningful and not academic or esoteric. Typically measuring the users reaction to the recommendations is the best metric; e.g. is your revenue per user or page view going up?
Based on clear, measureable success criteria the model can be refined and tuned to best perform.
Peter Drucker said it best, “What gets measured, gets managed”. Without measurable success metrics for your fledgling business how will you know if you are succeeding? You should create KPIs that have actionable meaning for your business. These KPIs should necessarily cover more than just your product or service. Internal KPIs for marketing, IT, business development etc. can be captured and presented in a company dashboard. Clear goals and metrics align the people getting the work done and provide the impetus for acceleration or course correction.
Rinse And Repeat
For both ML and SC, relevance is determined by maintaining a contemporary view of the ecosystem that it serves. Hence: