Most insurers are using some form of predictive modeling, but it can be difficult to know if it will remain effective over time. Evaluating a predictive model can be tricky because, while there are many ways data can be measured, there is no accepted standard. With the considerable investment that’s involved in predictive analytics, the C-suite understandably wants to hold certain yardsticks to the models and see if they are performing well, and to make sure every stakeholder is using it correctly. Having a forward-looking evaluation can make all the difference when making key decisions, especially if there is trust in the measuring mechanism.
Below are three new ways that insurers can evaluate the impact of predictive models, based on a model currently in production for a regional workers’ compensation insurer. The graphs below provide real-time insights that can help predictive modeling avoid becoming a black box, meaning that you can only see the output of the predictive model, not the input or how that output came to exist. The first two graphs separate out 10 equal portions of either premium or policy count, with each portion referred to as a "bin."
1. Monitoring that a model is still current and accurate
You need to be able to regularly check if the model you have in production is still up-to-date and providing accurate scores. This graph illustrates the overall model lift on the book for a regional workers’ comp insurer in 2015 and 2016. The insurer’s model is generating a low score on business that’s running very profitably -- the lower-risk bins 1, 2, 3 are approximately 30% better than average. Policies getting a score in the higher-risk bins 8, 9, 10 are all running at twice the average loss ratio. This provides a clear indication of what to target and what to avoid.
Bottom line: This model is still current and accurate.
See also: Top 6 Myths About Predictive Modeling
2. Tracking the impact of a model on decision-making
To realize the benefits of analytics, your staff needs to leverage the insights to make more informed decisions that create improved results. This is a graph of “decision data” from Valen’s InsureRight Manage application. Orange represents policies that were declined, red is quoted and lost, green is quoted and bound and yellow represents non-renewals. It’s evident that declinations are low on the good business -- less than 10% -- and high on the other end, approaching 50% for bin 10. The insurer is not renewing policies in bins 9 and 10 and, most importantly, retaining more than 50% of business in bins 1, 2, 3.
Bottom line: Underwriters at this insurer are using the model to make more profitable risk selection and pricing decisions.
3. Measuring if the overall risk quality of a portfolio is improving with a model in production.
If you’ve established that your model is accurate and your people are using it, the next question is what kind of impact it’s making to the quality of your portfolio. Are we lowering the risk of our book of business? This view shows the insurer’s risk-selection trends, with an overview of how risk-selection decisions have been influenced by a model and the resulting change to the portfolio. The blue bars represent premium volume by month, and the orange line represents average risk score (i.e., loss ratio prediction) by month. Though there is some variability from month to month, the overall downward trend indicates improvement over the course of the year. There is a small uptick in December 2016, which provides an indication that further analysis is needed.
Bottom line: The risk quality of this portfolio is improving, though still requires careful monitoring.
See also: Survey: Predictive Modeling Lifts Profits
Not only is it crucial to measure before an implementation takes place, it’s vital to do so both during and after, as well. Predictive modeling only works well if it is aligned with stated business goals, and knowing how to measure that is key to an insurer’s bottom line. With these three new ways to measure, insurers now will have different yardsticks to see whether it is successful and if they are using the actionable insights.
You need to be able to regularly check if the model you have in production is still up-to-date and providing accurate scores. This graph illustrates the overall model lift on the book for a regional workers’ comp insurer in 2015 and 2016. The insurer’s model is generating a low score on business that’s running very profitably -- the lower-risk bins 1, 2, 3 are approximately 30% better than average. Policies getting a score in the higher-risk bins 8, 9, 10 are all running at twice the average loss ratio. This provides a clear indication of what to target and what to avoid.
Bottom line: This model is still current and accurate.
See also: Top 6 Myths About Predictive Modeling
2. Tracking the impact of a model on decision-making
To realize the benefits of analytics, your staff needs to leverage the insights to make more informed decisions that create improved results. This is a graph of “decision data” from Valen’s InsureRight Manage application. Orange represents policies that were declined, red is quoted and lost, green is quoted and bound and yellow represents non-renewals. It’s evident that declinations are low on the good business -- less than 10% -- and high on the other end, approaching 50% for bin 10. The insurer is not renewing policies in bins 9 and 10 and, most importantly, retaining more than 50% of business in bins 1, 2, 3.
Bottom line: Underwriters at this insurer are using the model to make more profitable risk selection and pricing decisions.
3. Measuring if the overall risk quality of a portfolio is improving with a model in production.
If you’ve established that your model is accurate and your people are using it, the next question is what kind of impact it’s making to the quality of your portfolio. Are we lowering the risk of our book of business? This view shows the insurer’s risk-selection trends, with an overview of how risk-selection decisions have been influenced by a model and the resulting change to the portfolio. The blue bars represent premium volume by month, and the orange line represents average risk score (i.e., loss ratio prediction) by month. Though there is some variability from month to month, the overall downward trend indicates improvement over the course of the year. There is a small uptick in December 2016, which provides an indication that further analysis is needed.
Bottom line: The risk quality of this portfolio is improving, though still requires careful monitoring.
See also: Survey: Predictive Modeling Lifts Profits
Not only is it crucial to measure before an implementation takes place, it’s vital to do so both during and after, as well. Predictive modeling only works well if it is aligned with stated business goals, and knowing how to measure that is key to an insurer’s bottom line. With these three new ways to measure, insurers now will have different yardsticks to see whether it is successful and if they are using the actionable insights.
