On the Fragility of Active Learners for Text Classification · HackerPapers discussions

On the Fragility of Active Learners for Text Classification — mentioned by abhgh

Sun, 29 Sep 2024 23:37:57 GMT

Working on a library that helps benchmark Active Learning (AL) techniques [1]. This is a form of Machine Learning, where you want to learn a supervised predictor but you don't have labels to start with, and labeling comes at a cost that you need to account for (the term AL can have broader connotations but this is the popular one). We feel the area suffers quite a bit from poor benchmarks, which my colleague and I wrote up about in a paper [2]. To run the many experiments in the paper we had to write a fairly comprehensive codebase that makes it convenient to swap out different bits and…

Paper and discussion history. A mention is not an endorsement.

On the Fragility of Active Learners for Text Classification — mentioned by abhgh

Tue, 09 Apr 2024 21:38:03 GMT

It becomes valuable if it either compensates for the lack of hyperparam tuning in all or a broad class of models, or the result is precisely presented to say something like "better than random if the model is a linear svm with C=1". Right now we are somewhere in between where the results are presented to seem somewhat generally useful, but when you tweak the hyperparams you realize that the utility is potentially quite narrow. We either need broad utility or increased rigor in reporting - we cannot abandon both. Yeah that's an interesting paper [1], and label imbalance is a case we have not…

Paper and discussion history. A mention is not an endorsement.

On the Fragility of Active Learners for Text Classification — mentioned by abhgh

Tue, 09 Apr 2024 05:58:18 GMT

Thanks for the good discussion! * Reg. model selection/calibration: I acknowledge that in interactive setups cross-validation (CV) is not practical, but that is a particular kind of model selection. And faster alternatives exist for specific model families, such as approx. gradients [1,2], out-of-bag error (Random Forests), Infinitesimal Jacknife [3]. The bigger problem to me (for no model selection) is that there is a philosophical gap here: why would we fit the model with a default parameter blessed by a specific library? For ex., for LinearSVC in scikit, what is so great about C=1? And…

Paper and discussion history. A mention is not an endorsement.

On the Fragility of Active Learners for Text Classification — mentioned by abhgh

Tue, 09 Apr 2024 00:52:27 GMT

Appreciate the reply! We have 2 binary datasets in there [1], but of course, those results are not separately presented - you see the averages. The choice of datasets here is based on what other papers report, to have comparable results. In our experience with real data, AL doesn't fare any better on average - our pipelines use different representations (not just BERT based, includes some legacy ones like n-grams+tf-idf) and classifiers. We do see some strong peformance with RoBERTa (Fig 6) and the margin strategy, which is an uncertainty based strategy like least confidence (Fig 7) - these…

Paper and discussion history. A mention is not an endorsement.

On the Fragility of Active Learners for Text Classification — mentioned by abhgh

Sun, 07 Apr 2024 16:22:53 GMT

I second the point about learning a classifier over them - it is practically quite useful. But I'd caution against believing Active Learning (AL) would be unequivocally be useful; most positive performances seem to arise in very specific circumstances [1]. Esp. see Table 1: a Linear SVM with MPNet (one of the embeddings sbert supports) has indeed strong performance, but, in general, random sampling (over an AL strategy) performs quite well! [1] https://arxiv.org/pdf/2403.15744v1.pdf

Paper and discussion history. A mention is not an endorsement.