Explaining and Harnessing Adversarial Examples · HackerPapers discussions

Explaining and Harnessing Adversarial Examples — mentioned by andy99

Wed, 09 Jul 2025 00:07:38 GMT

> prompt injection attacks are the LLM equivalent of social engineering, That's anthropomorphizing. Maybe some of the basic "ignore previous instructions" style attacks feel like that, but the category as a whole is just adversarial ML attacks that work because the LLM doesn't have a world model - same as the old attacks adding noise to an image to have it misclassified despite clearly looking the same: https://arxiv.org/abs/1412.6572 (paper from 2014). Attacks like GCG just add nonsense tokens until the most probably reply to a malicious request is "Sure". They're not social engineering…

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by Imnimo

Sat, 20 Feb 2021 20:46:05 GMT

I think it's unlikely that the noise will generally have any relation to the target class. I can't find anywhere they say exactly which attack method they use, so it's hard to say for certain, but all of the attacks I'm aware of don't generate noise that has any human-interpretable structure. See for example Figure 1 in the seminal paper on adversarial attacks: https://arxiv.org/pdf/1412.6572.pdf

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by moyix

Wed, 26 Jul 2017 01:39:55 GMT

Yes: > Adversarial training seeks to improve the generalization of a model when presented with adversarial examples at test time by proactively generating adversarial examples as part of the training procedure. This idea was first introduced by Szegedy et al. [SZS13] but was not yet practical because of the high computation cost of generating adversarial examples. Goodfellow et al. showed how to generate adversarial examples inexpensively with the fast gradient sign method and made it computationally efficient to generate large batches of adversarial examples during the training process…

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by mhue

Fri, 21 Jul 2017 12:57:12 GMT

There are results showing that machine learning can be completely fooled by images that are slightly modified, so little that the human eye can not perceive any difference. Have a look at the images in this paper on "adversarial examples": https://arxiv.org/abs/1412.6572

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by dbecker

Sat, 04 Feb 2017 21:50:06 GMT

Adversarial noise isn't specific to deep learning models. Most model working on high-dimensional input will confidently misclassify noise with high confidence if you construct the noise right. The original "adversarial pixel" paper demonstrates this with logistic regression. https://arxiv.org/pdf/1412.6572v3.pdf

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by nl

Thu, 03 Nov 2016 10:28:45 GMT

https://arxiv.org/abs/1412.6572 is a Goodfellow et. al. (ie, the person who's Tweet you note above) paper where they show adversarial examples do apply across architectures (as you state above), but they show that using the adversarial examples during training has a regularization effect and improves performance: https://arxiv.org/pdf/1511.05432v3.pdf takes a similar approach: We show that adversarial training of ANNs is in fact robustification of the network optimization, and that our proposed framework generalizes previous approaches for increasing local stability of ANNs. Experimental…

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by sherjilozair

Sat, 29 Oct 2016 18:52:24 GMT

Short answer: No. Computing these perturbations requires an expensive optimization with multiple passes through the dataset, and this would be prohibitively expensive to do in the inner-most loop of training. There are other work in the literature describing faster algorithms to compute these perturbations, which makes it possible to use them while training. See, eg.: https://arxiv.org/abs/1412.6572

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — HN submission

Wed, 16 Dec 2015 04:30:53 GMT

Explaining and Harnessing Adversarial Examples

3 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by Houshalter

Mon, 09 Nov 2015 05:55:05 GMT

Humans can be easily fooled too: https://en.wikipedia.org/wiki/Optical_illusion Anyway this issue has been explained and fixed somewhat in this paper: http://arxiv.org/abs/1412.6572 And karpathy has a post which explains it: https://karpathy.github.io/2015/03/30/breaking-convnets/

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by abrichr

Sun, 21 Jun 2015 18:31:51 GMT

We spend so much effort trying to engineer intelligence, when we would get a lot farther reverse engineering intelligence. Whenever AI makes a big advance the analog was already known my neuroscientists. The problem with attempting to understand intelligence by reverse engineering the human brain is that we cannot know a priori which aspects of the human brain are necessary for intelligence to arise, and which are merely consequences/side effects of biology and chemistry. Once we discover some technique that works in a practical setting (e.g. on ImageNet), then it is fairly straightforward to…

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by Houshalter

Tue, 12 May 2015 08:59:31 GMT

He's also referring to this follow up paper which attempts to explain the issue and show that it exists in other models too: http://arxiv.org/abs/1412.6572

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — HN submission

Fri, 10 Apr 2015 21:15:28 GMT

Adversarial examples – Machine Learning [pdf]

2 points; 0 discussion replies on HN. Paper and source history. A submission is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by Nvn

Fri, 03 Apr 2015 20:20:58 GMT

> If this model can provide an explanation for the small noises impacting NN performance on images (from karpathy.github.io, posted to HN earlier today) then that would be rocking This paper [0] does a pretty good job of giving an explanation for that. [0] http://arxiv.org/abs/1412.6572

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by Houshalter

Fri, 03 Apr 2015 05:25:35 GMT

The original paper found that nets pretrained as autoencoders were the least susceptible to it, but still failed. IIRC the follow up paper (http://arxiv.org/abs/1412.6572) tested it or at least claimed it wasn't a solution.

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by Houshalter

Sun, 29 Mar 2015 23:47:28 GMT

Those are images incredibly optimized to fool and exploit those neural networks. However on the actual image recognition tasks they were designed for they beat humans. Anyway this paper has a method for addressing this problem: http://arxiv.org/abs/1412.6572

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by masterfres

Mon, 23 Mar 2015 19:06:13 GMT

Very interesting. First of all here's a youtube video associated with the paper -- https://www.youtube.com/watch?v=M2IebCN9Ht4 . Second some on here have posted about the Svegedy, Goodfellow, and Shlens paper http://arxiv.org/abs/1412.6572 which discusses the opposite effect. The Svegedy research is mentioned in the Nguyen paper and specifies that given an image that is correctly classified in a DNN, you can alter that image in a way imperceptible to a human to create a new image that will be INCORRECTLY classified. The Nguyen, Yosinski, et al. work that's the subject of this post states that…

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by Houshalter

Mon, 23 Mar 2015 17:27:51 GMT

Overfitting has nothing to do with it. See this paper: http://arxiv.org/abs/1412.6572 I believe the original paper tried ensembles and even got the images to work on different networks.

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by Houshalter

Mon, 23 Mar 2015 17:25:32 GMT

This paper tried that with positive results: http://arxiv.org/abs/1412.6572

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — mentioned by Houshalter

Mon, 23 Mar 2015 15:13:17 GMT

A paper came out that explains this effect and a method of minimizing it: http://arxiv.org/abs/1412.6572 Basically neural networks and many other machine learning methods are highly linear and continuous. So changing an input just slightly should change the output just slightly. If you change all of the inputs slightly in just the right directions, you can manipulate the output arbitrarily. These images are highly optimized for this effect and unlikely to occur by random chance. Adding random noise to images doesn't seem to cause it, because for every pixel changed in the right direction…

Paper and discussion history. A mention is not an endorsement.

Explaining and Harnessing Adversarial Examples — HN submission

Tue, 23 Dec 2014 17:26:32 GMT

Explaining and Harnessing Adversarial Examples [problem with deep networks]

1 point; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.