DIY Predictive Modeling: The Hidden Pitfalls and the Four E’s

At Predictable we talk to many companies and see varied approaches for leveraging AI-driven predictive outcomes. From time to time, we come across enterprises opting for in-house teams to craft their predictive models. The allure of a DIY approach is clear: retaining IP, leveraging an intimate understanding of business nuance, and enjoying endless model customization. From what we’ve observed, however, these perceived advantages go mostly unrealized due to several less apparent and significant challenges.  We refer to these the Four E’s: effort, experience, expense and efficacy.

Time-Consuming Effort

Building and deploying predictive AI models takes both time and resources – resources that are generally in short supply.  Unless you have a robust in-house ML engineering team with several data scientists on staff and the bandwidth to take on new projects, the work to implement and activate AI models in a production environment material and complex: model development, infrastructure deployment, data pipelines and outbound data integrations are all necessary components to the endeavor.  We estimate the effort is about 800 hours for the first model – minimum.  That’s half a year of effort before your initial results are available.  And that would be a successful project, moving quickly, with an experienced team.  For most businesses the effort invested will be much, much longer, with no guarantee of a useful output.

The Steep Learning Curve of Experience

Predictive modeling, especially when designed to enhance marketing strategies, isn’t a beginner’s game. Grasping the intricacies of model development, deployment, and maintenance—keeping the models updated, ensuring uninterrupted data flow—becomes a full-time job.   Additionally, identifying the right data sources, both internally and externally, that drive model lift is a learning process that takes time and iteration.  Organizations lacking prior experience in these domains, and not prepared for continued investment in upkeep and monitoring are bound to struggle.  

Hidden Costs and Expense

The DIY option may seem like a low-cost solution, but there are many hidden costs that make the economics highly unpalatable.  Most apparent are the costs of the team required to build and maintain these models.  The team to build, and then manage the modeling, data pipelines and ML operations involved is likely 1-2 FTEs.  Additional costs for cloud hosting, ML tooling, and software for inbound and outbound data transfer are also a factor.  We estimate the DIY option has a floor “price” of $250k per year in hard costs.  And that doesn’t account for additional organizational overhead, and the opportunity costs inherent in taking on a new internal initiative.

Uncertainty of Efficacy

The whole point of predictive modeling is to drive better outcomes.  But if you are building a predictive model for the first time, how do you know it’s going to work?  And how do you know it’s working as well as it could, or should?  If you’ve built models for dozens of similar brands, tested them across channels and executions, iterated and improved them over time, you are in a much better position to drive efficacy and meaningful results than if you are rolling out your first model off-the-shelf.  

So while the DIY option may seem appealing, consider the hidden pitfalls described above before going down the DIY path.  Consider too, solutions like Predictable, which offer rapid deployment, rapid time-to-value, and highly performance models with no coding, data science, or engineering overhead required.