Underspecification Presents Challenges for Credibility in Modern Machine Learning · HackerPapers discussions

Underspecification Presents Challenges for Credibility in Modern Machine Learning — mentioned by pms

Fri, 24 Mar 2023 01:49:25 GMT

Have you provided the visual novel as a prompt or just asked about it? ChatGPT does seem to know more about books than songs, maybe because they are more often described in texts, while songs not, not sure. I wished more parameters gave a generalizable solution, but research suggests that's not the case [1, 2, 3]. OpenAI won't tell you this, cause they have a product to sell. [1] http://arxiv.org/abs/2211.08411 [2] http://arxiv.org/abs/2011.03395 [3] http://arxiv.org/abs/2005.04345

Paper and discussion history. A mention is not an endorsement.

Underspecification Presents Challenges for Credibility in Modern Machine Learning — mentioned by andy99

Tue, 04 May 2021 00:35:05 GMT

You've probably seen it, but a more recent, related paper (that I think has some of the same authors) about inherent features of modern ML that make models so fragile, even if they test OK: Underspecification Presents Challenges for Credibility in Modern Machine Learning, D'Amour et al., https://arxiv.org/abs/2011.03395

Paper and discussion history. A mention is not an endorsement.

Underspecification Presents Challenges for Credibility in Modern Machine Learning — HN submission

Tue, 01 Dec 2020 01:19:23 GMT

Underspecification Presents Challenges for Credibility in ML

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Underspecification Presents Challenges for Credibility in Modern Machine Learning — mentioned by throwaway888abc

Mon, 30 Nov 2020 23:26:32 GMT

Here is the paper. "ML models often exhibit unexpectedly poor behavior when they are deployed in real-world domains. We identify underspecification as a key reason for these failures. An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain. Underspecification is common in modern ML pipelines, such as those based on deep learning. Predictors returned by underspecified pipelines are often treated as equivalent based on their training domain performance, but we show here that such predictors can behave very…

Paper and discussion history. A mention is not an endorsement.

Underspecification Presents Challenges for Credibility in Modern Machine Learning — HN submission

Sat, 21 Nov 2020 19:21:35 GMT

Underspecification Presents Challenge for Credibility in Modern Machine Learning

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Underspecification Presents Challenges for Credibility in Modern Machine Learning — HN submission

Thu, 19 Nov 2020 15:15:46 GMT

Underspecification Presents Challenges for Credibility in Machine Learning

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Underspecification Presents Challenges for Credibility in Modern Machine Learning — HN submission

Mon, 16 Nov 2020 16:54:22 GMT

Underspecification Presents Challenges for Credibility in Modern MachineLearning

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Underspecification Presents Challenges for Credibility in Modern Machine Learning — HN submission

Sat, 14 Nov 2020 20:41:27 GMT

Underspecification Presents Challenges for Credibility in Machine Learning

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Underspecification Presents Challenges for Credibility in Modern Machine Learning — HN submission

Mon, 09 Nov 2020 14:56:48 GMT

On generalization problems of ML models in real-world applications

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.