As mentioned yesterday, I gave a talk, this afternoon, entitled From Contemplative to Predictive Modeling (in actuarial science and risk management). Slides are available online, but maybe I can take some time to explain what I talked about…

It is usually claimed that actuaries build ‘predictive models’ but most of the time, what they consider would be simply ‘contemplative modeling’, in the sense that they use past information and hope that the future will be more or less the same (corresponding to the idea of generalization in machine learning). In the context of climate change (but also when modeling insurance market competition) it is not the case, data used to train models do not have the same distribution as the one we will have in the future.

As a starting point, it is my second talk at the Actuarial Contact Program (ACP) meetings, the last one was in June 2015, for a talk entitled Data Science: from Small to Big Data. My take-away slide mentioned two points. The first one was a mention of a quote, claiming that we should have more fun. As far as I remember, it’s when actuaries shifted from accounting problems (and Solvency II) to data science, and computational actuarial science (in 2015, the textbook was just out). The second one is a comment in Nate Silver’s The Signal and the Noise, claiming that data won’t speak for themselves. And actually, that’s the starting point of my talk this year…

Back in 2015, Chris Anderson’s The End of Theory was still the mainstream idea

The new availability of huge amounts of data, along with the statistical tools to crunch these numbers, offers a whole new way of understanding the world. Correlation supersedes causation, and science can advance even without coherent models, unified theories, or really any mechanistic explanation at all.

But I guess we could mow agree on the fact that data can’t talk without a model,  to avoid getting misleaded. Now just a brief disclaimer, to explain the title of the talk, From Contemplative to Predictive Modeling. I did borrow the idea from an actuary I was in graduate school with. His point was that actuaries claim to do “predictive modeling”, but most of the time, it’s more “contemplative” than really “predictive”. I should stress that I did not use “contemplative” in devotious, or religious sense, but in the sense “the act of regarding steadly” that is mentioned in the dictionary. But in many applications, that’s not enough. Of course, history sets the context for the present, but still, using past observed data is usually not enough to make accurate predictions.

A classical point is John Stuart Mill’s popular “black swan” example. How could we “think outside the box”, predict events that were never observed before ? Because, somehow, that is the core of insurance.

More formally, it is based on the idea of “generalization” used in machine learning, and actuaries. Because they still give a lot of important on accuracy of models, in the sense that they should perform well on a validation dataset, which is an i.i.d. version of the training one. The i.i.d. (or stationary) assumption is also very important in actuarial science, and that’s what I want to discuss here

I mention a couple of references stressing that idea, in the context of risk, that the data we have to train a model is probably not exactly what we would love to have so that our model has good properties (usually accuracy on new and possibly quite different data, but also legal requierements, such as non-discriminatory)

Statistics and Machine Learning

First, we can’t talk about “contemplation” without mentioning the rebus sic stantibus statement that is so important in statistical modeling, usually translated as with thing thus standing,

This concept (related to the idea of i.i.d. samples) is an important assumption, that appears in the very first page of Statistics of Extremes, by Emil Gumbel

It is used to define the concept of “return period“, popular tool in the toolbox of actuaries, when studying extremal events

If is a key assumption because the geometric distribution (that is underlying that average return period) is based on that i.i.d. assumption

It is also important as we will see, when discussing generalization properties of machine learning algorithms. I should stress here that when I started teaching the use of generalized linear models in insurance, twenty years ago, it was not really an issue. We spent more time on the interpretation and the assumption, but really on predictive properties. Well, somehow, there were ideas related to this generalization idea. For instance, there were residuals that were obtain by comparing the true observation y_i and the prediction, but actually not the standard prediction \widehat{y}_i, the one where we fitted the very same model (with the very same algorithm) on the dataset but observation i, and we get our prediction on a brand new observation, not used to estimate the model, \widehat{y}_{(i)}.

That sound very much like Jackknife and cross validation

Cross validation is a popular idea to avoid under and over-fit

The entiere idea can be summarized in the following graph, where the complexity should be not too small, not too large, but just right. That’s the Goldilocks principle.

A related concept is the bias-variance tradeoff, but that usually requires mathematical computations of biases and variances,

In machine learning, we rely on computational techniques, where generalization properties can be investigated. Of course, there are mathematical foundations, such as the law of large numbers

or some concentration inequalities

that gives orders of magnitude of the generalization error, under the standard i.i.d. assumption, and the fact that the underlying distribution of observations used to fit a model is actually the one we expect to have for prediction (that we will call “target” later on)

Of course, it is a simplfied introduction, and it is possible to adapt techniques used on i.i.d. on non stationary time series, for example. It is possible to adapt the k-fold tool

to time series (and there’s nothing really new here, the idea was suggested a long time ago)

Beyond generalization in actuarial science

In actuarial science, the rebus sic stantibus assumption (with thing thus standing) is usually very standard. For example, in claims reserving, the Chain Ladder methodology assumes that past historical patterns of payments can be used to predict future payments,

In triangles, we observe historical payments (top left in red) and want to predict future payments (bottom right in green). Somehow, here, we “let the data speak”. Nevertheless, it could be important to introduce some judgement or model, if there could be an additional column, of payments very far away in the future (never observed, so far).

For that “tail factor”, we really have a “predictive model”, in the sense that it is unforseen. But as long as we stay in that rectangle format, everything is fine,

Things can be more complicated if we consider not a development year factor by use calendar year factor, for instance if we suppose that there is some inflation

Then we face a real “predictive modeling” problem, because we need to extrapolate future inflation (which is usually not an easy exercise)

I can also talk a little bit about “transfer learning” where we add additional information to improve of predictions. In a small line of business related to liability, it is possible to use larger lines of business also related to liability, with similar patterns etc. It also possible to transfer additional information, taking into account not only financial payments, but also estimates of incurred losses by claims manager

Since claims reserving and mortality are very similar problem, we can also mention life insurance, and mortality. In standard approaches, things are fine, and again, “data can speak by themselves”

But if we want to make proper predictive inference, we need to predict future death rate. Even to compute a simple quantity such at the life expectancy. Demography is a predictive problem, so we need models to do that, real predictive models

Here again, it is possible actually to transfer knowledge from related groups

Climate modeling  is also a predictive model problem

And possibly a complicated one. In California for example, insurers are not allowed to use “climated-informed catastrophe models“, that are real predictive models

In France, regulation mentions return periods of climate events, acknoledging also that climage is changing (and is no longer stationary)

Finally, I could mention the one I have been working on for the past year, discrimination and fairness. The way I see it, it could be related to the “is-ought” problem.

The is–ought problem, as articulated by the Scottish philosopher and historian David Hume, arises when one makes claims about what ought to be that are based solely on statements about what is.

This means that it is difficult to have what ought to be (related to an ethical “norm”) based on data we collected, that reflect what is (the statistical “normality”). To illustrate I get back to an example discussed recently on this blog, where I prove that even if we exclude the gender from the training dataset, to price motor insurance policies (we ought to have gender-free premiums), the model still discriminated, because of the generalization property : the model’s objective is to replicate what is in the data. If there is significant difference (here men have 10% more chance to claim a loss than women), the model will reproduce if (even without focusing on the gender since the information is not available)

This point is very important in predictive modeling. In this “is-ought” problem, it means that we cannot get fairness without explicitely asking for it, and of course it will come with a price (on the accurary, but since the accuracy is based on some generalization that is probably not the best metric, maybe it is not so bad)

Mitigating distribution will be translated, mathematically, as distorting, transporting, distributions.

Predictions with biases data

If we get back to general thoughts, we should remember that we should not let the data speak because of possible baises…

(to be continued….)

---

---