Cet été, les étudiants relancent leur séminaire. Ça a commencé ce midi (avec de l’estimation non paramétrique dans des modèles de cladogénèse) et c’était passionnant
Tag Archives: StatQAM
Exposé au séminaire de statistique (StatQAM)
Tomorrow, Ewen Gallic will present some recent work at the StatQAM statistical seminar, on calibration, with Agathe Fernandes Machado, François Hu, and Emmanuel Flachaire. It will substantially be based on our recent paper From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration
The assessment of binary classifier performance traditionally centers on discriminative ability using metrics, such as accuracy. However, these metrics often disregard the model’s inherent uncertainty, especially when dealing with sensitive decision-making domains, such as finance or healthcare. Given that model-predicted scores are commonly seen as event probabilities, calibration is crucial for accurate interpretation. In our study, we analyze the sensitivity of various calibration measures to score distortions and introduce a refined metric, the Local Calibration Score. Comparing recalibration methods, we advocate for local regressions, emphasizing their dual role as effective recalibration tools and facilitators of smoother visualizations. We apply these findings in a real-world scenario using Random Forest classifier and regressor to predict credit default while simultaneously measuring calibration during performance optimization.
To illustrate, consider predictions about the gender of the person on the picture, including probabilities (confidence), obtained from https://www.picpurify.com/demo-face-gender-age.html, with fake pictures, from https://www.nytimes.com/interactive/2020/11/21/science/artificial-intelligence-fake-people-faces.html.
Talk at StatQAM on Counterfactuals and Optimal Transport
Next Thursday, I will present our recent work at the StatQAM seminar, with Emmanuel Flachaire ajd Ewen Gallic, on Optimal Transport for Counterfactual Estimation: A Method for Causal Inference
Many problems ask a question that can be formulated as a causal question: “what would have happened if…?” For example, “would the person have had surgery if he or she had been Black?” To address this kind of questions, calculating an average treatment effect (ATE) is often uninformative, because one would like to know how much impact a variable (such as skin color) has on a specific individual, characterized by certain covariates. Trying to calculate a conditional ATE (CATE) seems more appropriate. In causal inference, the propensity score approach assumes that the treatment is influenced by x, a collection of covariates. Here, we will have the dual view: doing an intervention, or changing the treatment (even just hypothetically, in a thought experiment, for example by asking what would have happened if a person had been Black) can have an impact on the values of x. We will see here that optimal transport allows us to change certain characteristics that are influenced by the variable we are trying to quantify the effect of. We propose here a mutatis mutandis version of the CATE, which will be done simply in dimension one by saying that the CATE must be computed relative to a level of probability, associated to the proportion of x (a single covariate) in the control population, and by looking for the equivalent quantile in the test population. In higher dimension, it will be necessary to go through transport, and an application will be proposed on the impact of some variables on the probability of having an unnatural birth (the fact that the mother smokes, or that the mother is Black).
Slides are now online.