Beyond Accuracy: Behavioral Testing of NLP models with CheckList — mentioned by jayalammar

Mon, 03 Jan 2022 14:50:48 GMT

The size of the database, the training set, the details of the architecture, as well as results on benchmark tasks should all be considered in the comparison. I'm also a fan of Behavioral Testing [1]. Parameter count is not very accurate measure of model performance. Mixture of Expert models like the Switch Transformer [2] can be 1 trillion parameters in size, but are not 5X the performance, for example. They clock the retrieval at 10 ms, unclear if that includes the BERT inference, however. My assumption is that it does not. [1] https://arxiv.org/abs/2005.04118 [2]…

Paper and discussion history. A mention is not an endorsement.

Beyond Accuracy: Behavioral Testing of NLP models with CheckList · HackerPapers discussions

Beyond Accuracy: Behavioral Testing of NLP models with CheckList — mentioned by jayalammar