"Why Should I Trust You?": Explaining the Predictions of Any Classifier · HackerPapers discussions

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by pona-a

Wed, 05 Jun 2024 12:58:45 GMT

That's what model interpretability research is. You can train an interpretable model from the uninterpretable teacher, you can look at layer activations and how they correspond to certain features, or apply a hundred other domain-specific methods depending on your architecture. [0] Sadly, insight is always lost. In a noisy world where even with the best regularization, some fitting on it, or higher order features that describe it, is inevitable for maximizing prediction accuracy, especially if you don't have the right tools to model it (like transformers adapting to lacking registers [1]) and…

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — HN submission

Fri, 30 Jul 2021 04:30:15 GMT

“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by spekcular

Mon, 30 Nov 2020 04:42:01 GMT

One reason to want to understand what some "black box" is doing is distribution/dataset shift, especially in medical applications. For example, suppose you're building a neural net to detect early-stage lung cancer on medical imaging, and you test/train it on patients in a small set of hospitals. Often the hospital name is given on the image, and this can be used as a covariate to improve accuracy (due to the hospitals serving different populations with different demographics). But a model that does this may suffer when put into production at other hospitals. Some real-life examples with this…

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — HN submission

Tue, 20 Oct 2020 16:37:50 GMT

Explaining black box models with LIME

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by rangerranvir

Thu, 30 Apr 2020 22:15:13 GMT

Great Points. Really appreciated. Will have to put extra effort to learn about the feature engineering part of the problem. Also, if you know a few things about the data it becomes a little easier to explain what your model is doing and why it is producing those results. Found a good resource which explained the trust component: https://arxiv.org/pdf/1602.04938.pdf

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by Der_Einzige

Thu, 11 Jul 2019 06:16:03 GMT

This is plain wrong. https://arxiv.org/abs/1602.04938 A simple counter-example. A really neat library called "eli5" and specifically, it's white-boxing functionality for NLP and CV applications does a great job of helping one figure out if their model is learning "incorrect" information in its prediction pipeline. The paper I link show why this does a good job of improving user trust in models, which is your second point on human-computer interaction https://eli5.readthedocs.io/en/latest/tutorials/black-box-te... For instance, one can run the same TextExplainer algorithm on a word embedding…

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by jamessb

Mon, 26 Mar 2018 14:12:02 GMT

There's a whole field studying machine learning interpretability. This recent Distill article is a good introduction [1]. The lime [2] package, based on this paper [3], is also notable. [1]: https://distill.pub/2018/building-blocks/ [2]: https://github.com/marcotcr/lime [3]: https://arxiv.org/abs/1602.04938

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by standyro

Tue, 19 Dec 2017 07:59:39 GMT

Explain in detail all of the minute muscle movements that you make to throw a ball. Not all things that humans learn are easily articulated or explained, not because those humans aren't articulate, but because they learned actions that are more easily intuited. Also, on this topic, explainability of machine learning models is a hot field: https://www.youtube.com/watch?v=KP7-JtFMLo4 https://arxiv.org/abs/1602.04938 https://github.com/marcotcr/lime

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by nl

Mon, 11 Dec 2017 00:51:35 GMT

LIME: Local Interpretable Model-Agnostic Explanations; https://www.oreilly.com/learning/introduction-to-local-inter... "“Why Should I Trust You?” Explaining the Predictions of Any Classifier": https://arxiv.org/pdf/1602.04938.pdf https://homes.cs.washington.edu/~marcotcr/blog/lime/ https://github.com/marcotcr/lime Anytime anyone makes snide HN comments like "oh you can't understand why neural networks make predictions" the correct response should always be "why doesn't LIME work in your specific case". LIME is being used within the EU to explain credit decisions and fraud detection flagging…

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by drieddust

Tue, 01 Aug 2017 08:11:40 GMT

Agreed and I am not arguing to regulate the obvious but now as AI can potentially touch and transform every walk of life, guidelines are needed to keep big players from going too far. There won't be any ground for plausible deniability if principals are set. For example I think when deep learning is bound to touch every aspect of our lives, explainable models should be a must[1]. Look at section 3.5 where classifier was predicting with 94% accuracy for completely unrelated reasons like a person's name. >Although this classifier achieves 94% held-out accuracy, and one would be tempted to trust…

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by halwa

Sat, 29 Jul 2017 08:34:35 GMT

While I wouldn't say that the problem has been "well studied", the research community has been paying interest, and some progress has been made, most notably- as pointed out by benjaminjackman- LIME[1]. Roughly speaking, LIME learns a locally approximate model which can be interpreted. It will work will any black box model, not just neural networks. [1] https://arxiv.org/abs/1602.04938

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by nl

Tue, 28 Mar 2017 11:28:54 GMT

That's not what "explainable ML is". That project is about explaining how (say) a given classifier came to a decision on a given set of data. This is fairly easy on a tree-based classifier, but on a deep neural network is extremely difficult. The most impressive piece of work I'm aware of in this area is LIME, from “Why Should I Trust You?” Explaining the Predictions of Any Classifier[1] The acknowledgement section of that paper read, in part: This work was supported in part by ONR awards #W911NF-13-1-0246 and #N00014-13-1-0023, and in part by TerraSwarm, one of six centers of STARnet, a…

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by yoav_hollander

Thu, 08 Sep 2016 10:13:41 GMT

Right. This paper ([0]) is actually mentioned in the DARPA BAA ([1]) as an example of a possible direction. A somewhat-similar scheme is [2]. Both seem to do some kind of sensitivity analysis, so as to show the user which parts of the input were most important for coming up with the decision: For instance, [2] "explains" an ML system (which answers questions about pictures), by telling you which pixels were most important for the decision. It does that by essentially hiding pixels and seeing how that influences the ML system's decisions. So this produces not so much an explanation as "hints"…

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by vonnik

Thu, 08 Sep 2016 04:45:30 GMT

That's what I was thinking of, too. If any one wants the direct link to the arXiv paper, it's here: https://arxiv.org/abs/1602.04938 . One of the authors is Carlos Guestrin, one of the co-founders of Dato/Turi/Graphlabs that was recently acquired by Apple, fwiw.

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — HN submission

Sun, 04 Sep 2016 17:34:01 GMT

“Why Should I Trust You?” Explaining the Predictions of Any Classifier [pdf]

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — mentioned by brudgers

Thu, 21 Apr 2016 13:20:30 GMT

Blog post: http://homes.cs.washington.edu/~marcotcr/blog/lime/ Paper: http://arxiv.org/pdf/1602.04938v1.pdf

Paper and discussion history. A mention is not an endorsement.

"Why Should I Trust You?": Explaining the Predictions of Any Classifier — HN submission

Tue, 12 Apr 2016 16:51:24 GMT

[pdf] Explaining the Predictions of Any Classifier

1 point; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.