Embed Size (px)
Citation preview
TOWARDS FEATURE SPACE ADVERSARIAL ATTACK
Qiuling XuDepartment of Computer Science
Purdue [email protected]
Guanhong TaoDepartment of Computer Science
Purdue [email protected]
Siyuan ChengDepartment of Computer Science
Purdue [email protected]
Xiangyu ZhangDepartment of Computer Science
Purdue [email protected]
ABSTRACT
We propose a new adversarial attack to Deep Neural Networks for image classification. Differentfrom most existing attacks that directly perturb input pixels, our attack focuses on perturbing abstractfeatures, more specifically, features that denote styles, including interpretable styles such as vividcolors and sharp outlines, and uninterpretable ones. It induces model misclassfication by injectingimperceptible style changes through an optimization procedure. We show that our attack can generateadversarial samples that are more natural-looking than the state-of-the-art unbounded attacks. Theexperiment also supports that existing pixel-space adversarial attack detection and defense techniquescan hardly ensure robustness in the style related feature space. 1
1 Introduction
Adversarial attacks are a prominent threat to the broad ap-plication of Deep Neural Networks (DNNs). In the contextof classification applications, given a pre-trained model Mand a benign input x of some output label y, adversarialattack perturbs x such that M misclassifies the perturbedx. The perturbed input is called an adversarial example.Such perturbations are usually bounded by some distancenorm such that they are not perceptible by humans. Sinceit was proposed in [1], there has been a large body ofresearch that develops various methods to construct adver-sarial examples with different modalities (e.g., images [2],audio [3], text [4], and video [5]), to detect adversarialexamples [6, 7], and use adversarial examples to hardenmodels [8, 9].
However, most existing attacks (in the context of imageclassification) are in the pixel space. That is, boundedperturbations are directly applied to pixels. In this paper,we illustrate that adversarial attack can be conducted in thestyle related feature space. The underlying assumption isthat during training, a DNN may extract a large numberof abstract features. While many of them denote criticalcharacteristics of the object, some of them are secondary,
1The code is available at https://github.com/qiulingxu/FeatureSpaceAttack.
for example, the different styles of an image (e.g., vividcolors versus pale colors, sharp outlines versus blur out-lines). These secondary features may play an improperlyimportant role in model prediction. As a result, featurespace attack can inject such secondary features, which arenot simple pixel perturbation, but rather functions overthe given benign input, to induce model misclassification.Since humans are not sensitive to these features, the re-sulted adversarial examples look natural from humans’perspective. As many of these features are pervasive, theresulted pixel space perturbation may be much more sub-stantial than existing pixel space attacks. As such, pixelspace defense techniques may become ineffective for fea-ture space attacks (Section 5). Figure 1 shows a numberof adversarial examples generated by our technique, theircomparison with the original examples, and the pixel spacedistances. Observe that while the distances are much largercompared to those in pixel space attacks, the adversarialexamples are natural, or even indistinguishable from theoriginal inputs in humans’ eyes. The contrast of the benign-adversarial pairs illustrates that the malicious perturbationslargely co-locate with the primary content features, denot-ing imperceptible style changes.
Under the hood, we consider that the activations of aninner layer represent a set of abstract features, includingthose primary and secondary. Distinguishing the two typesof features is crucial for the quality of feature-space at-
arX
iv:2
004.
1238
5v2
[cs
.LG
] 1
6 D
ec 2
020
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
(a) Spaniel`∞:121/255`2:25.92
(b) Espresso192/255
24.47
(c) Balloon149/255
20.75
(d) Llama183/255
28.55
(e) Printer252/255
21.80
(f) Lizard225/25540.88
(g) Guitar216/255
25.60
(h) Race car248/255
29.67
Figure 1: Examples by feature space attack. The top rowpresents the original images. The middle row denotes theadversarial samples. The third row shows the pixel-wisedifference (×3) between the original and the adversarialsamples. The `∞ and `2 norms are shown on the bottom.
tack. To avoid generating adversarial examples that areunnatural, we refrain from tampering with the primaryfeatures (or content features) and focus on perturbing thesecondary style features. Inspired by the recent advance instyle transfer [10], the mean and variance of activationsare considered the style. As such, we focus on perturbingthe means and variances while preserving the shape of theactivation values (i.e., the up-and-downs of these valuesand the relative scale of such up-and-downs). We use gradi-ent driven optimization to search for the style perturbationsthat can induce misclassification. Since our threat model isthe same as existing pixel space attacks, that is, the attackis launched by providing the adversarial example to themodel. An important step is to translate the activationswith style changes back to a naturally looking pixel spaceexample. We address the problem by considering the dif-ferences of any pair of training inputs of the same class as
the possible style differences, and pre-training a decoderthat can automatically impose styles in the pixel spacebased on the style feature variation happening in an innerlayer. We propose two concrete feature space attacks, oneto enhance styles and the other to impose styles constitutedfrom a set of pre-defined style prototypes.
We evaluate our attacks on 3 datasets and 7 models. Weshow that feature space attacks can effectively generateadversarial samples. The generated samples have natu-ral, and in many cases, human imperceptible style differ-ences compared with the original inputs. Our compar-ative experiment with recent attacks on colors [11] andsemantics [12] shows that our generated samples are morenatural-looking. We also show that 7 state-of-the-art detec-tion/defense approaches are ineffective to our attack as theyfocus on protecting the pixel space. Particularly, our attackreduces the detection rate of a state-of-the-art pixel-spaceapproach [13] to 0.04% on the CIFAR-10 dataset, and theprediction accuracy of a model hardened by a state-of-artpixel-space adversarial training technique [14] to 1.25%on ImageNet. Moreover, we observe that despite the largedistance introduced in the pixel space (by our attack), thedistances in feature space are similar or even smaller thanthose in `-norm based attacks. Note that the intention ofthese experiments is not to claim our attack is superior,but rather to illustrate that new defense and hardeningtechniques are needed for feature space protection.
2 Background and Related Work
Style Transfer. Huang and Belongie [10] proposed totransfer the style from a (source) image to another (target)that may have different content such that the content of thetarget image largely retains while features that are not es-sential to the content align with those of the source image.Specifically, given an input image, say the portrait of actorBrad Pitt, and a style picture, e.g., a drawing of painterVincent van Gogh, the goal of style transfer is to producea portrait of Brad Pitt that looks like a picture painted byVincent van Gogh. Existing approaches leverage varioustechniques to achieve this purpose. Gatys et al. [15] uti-lized the feature representations in convolutional layersof a DNN to extract content features and style featuresof input images. Given a random white noise image, thealgorithm feeds the image to the DNN to obtain the corre-sponding content and style features. The content featuresfrom the white noise image are compared with those froma content image, and the style features are contrasted withthose from a style image. It then minimizes the above twodifferences to transform the noise image to a content im-age with style. Due to the inefficiency of this optimizationprocess, researchers replace it with a neural network thatis trained to minimize the same objective [16, 17]. Furtherstudy extends these approaches to synthesize more thanjust one fixed style [18, 19]. Huang and Belongie [10]introduced a simple and yet effective approach, which canefficiently enable arbitrary style transfer. It proposed anadaptive instance normalization (AdaIN) layer that aligns
2
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
the mean and variance of the content features with thoseof the style features.Adversarial Attacks beyond Pixel Space. The explo-ration beyond `-norm based attacks is rising. Inkawhichet al. [20] found that simulating feature representation oftarget label improves transferability. Hosseini and Pooven-dran [11] proposed to modify the HSV color space togenerate adversarial samples. The method transforms allpixels by a non-parametric function uniformly. Differently,our feature space attack changes colors of objects or back-ground and the transformation is learned from images ofthe same object with different styles. It is hence moreimperceptible. Laidlaw and Feizi [21] proposed to changethe lighting condition and color (like [11]) to generate ad-versarial examples. Prabhu and UnifyID [22] producedart-style images as adversarial samples. It does not restrictthe feature space such that the generated samples are notnatural looking, especially compared to ours. Bhattad et al.[12] generated semantic adversarial examples by modify-ing color and texture. It advocates not to restrict attackspace and is hence considered unbounded. As such, it isdifficult to control the attack to avoid generating unreal-istic samples. In contrast, our attack has a well-definedattack space while being unbounded in the pixel space. Itimplicitly learns to modify lighting condition, color andtexture, it tends to be more general and capable of trans-forming subtle (and uninterpretable) features (Section 5.1).Unlike in Song et al. [23], where a vanilla GAN-basedattack generates samples over a distribution of limited sup-port, and has little control of the generated samples, ourencoder-decoder based structure enables attacking indi-vidual samples with controlled content. Stutz et al. [24]proposed to perturb the latent embedding of VAE-GANto generate adversarial samples. Since it does not distin-guish primary and secondary features, the perturbation onprimary features substantially degrades the quality of gen-erated adversarial samples. In contrast, our feature spaceperturbation is more effective.
We empirically compare with these attacks later in thepaper.
3 Feature Space Attack
Overview. We aim to demonstrate that perturbation inthe feature space can lead to model misbehavior, whichexisting pixel space defense techniques cannot effectivelydefend against. The hypothesis is that during training, themodel picks up numerous features, many of which do notdescribe the key characteristics (or content) of the object,but rather human imperceptible features such as styles.These subtle features may play an improperly importantrole in model prediction. As a result, injecting such fea-tures to a benign image can lead to misclassification. How-ever, the feature space is not exposed to attackers such thatthey cannot directly perturb features. Therefore, a promi-nent challenge is to derive the corresponding pixel spacemutation that appears natural to humans while leading tothe intended feature space perturbation, and eventually the
Encoder Deco
der
EncoderImg
Img
Img
Encoder Deco
der
Target MEncoder
Img
Img
1 23 4
5 B
D
AC
12 3 4
5
6
7
BA
E
{ {μp
c , σpc
μp0 , σp
0
…
{ {μq
c , σqc
μq0 , σq
0
…
{ {μp
c , σpc
μp0 , σp
0
…
(a) Decoder training phase
Encoder Deco
der
EncoderImg
Img
Img
Encoder Deco
der
Target MEncoder
Img
Img
1 23 4
5 B
D
AC
12 3 4
5
6
7
BA
E
{ {μp
c , σpc
μp0 , σp
0
…
{ {μq
c , σqc
μq0 , σq
0
…
{ {μp
c , σpc
μp0 , σp
0
…
(b) Feature space attack
Figure 2: Procedure of feature space adversarial attack.Two phases are involved during the attack generation pro-cess: (a) decoder training phase and (b) feature space attackphase.
misclassification. In particular, the attack comprises twophases: (1) training a decoder that can translate featurespace perturbation to pixel level changes that look naturalfor humans; (2) launching the attack by first using gradientbased optimization to identify feature space perturbationthat can cause misclassification and then using the decoderto generate the corresponding adversarial example. In-spired by style transfer techniques, we consider a muchconfined feature perturbation space – style perturbation.Specifically, as in Huang and Belongie [10], we considerthe mean and variance of the activations of an inner layerdenote the style of the features in that layer whereas theactivations themselves denote the content features. Wehence perturb the mean and variance of content featuresby performing a predefined transformation that largely pre-serves the shape of the features while changing the meanand variance. The decoder then decodes the perturbedfeature values to an image closely resembles the originalimage with only style differences that appear natural tohumans but causing model misclassification.
Fig. 2 illustrates the workflow of the proposed attack. Inthe Decoder training phase (a), a set of image pairs witheach pair from the same class (and hence their differencescan be intuitively considered as style differences) are fedto a fixed Encoder that essentially consists of the first afew layers of a pre-trained model (e.g., VGG-19) (step1©). The Encoder produces the internal embeddings of the
two respective images, which correspond to the activationvalues of some inner layer in the pre-trained model, e.g.,conv4_1 (step 2©). Each internal embedding consists of anumber of matrices, one for every channel. For each em-bedding matrix, the mean and variance are computed. Weuse these values from the two input images to produce theintegrated embedding A© (step 3©), which will be discussedin details later in this section. Intuitively, it is generated byperforming a shape-preserving transformation of the uppermatrix so that it retains the content features denoted by the
3
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
upper matrix while having the mean and variance of thelower matrix (i.e., the style denoted by the lower matrix).We employ a Decoder to reconstruct a raw image from A©at step 4©, which is supposed to have the content of theupper image (called the content image) and the style ofthe lower image (called the style image). To enable goodreconstruction performance, two losses are utilized for op-timizing the Decoder. The first one is the content loss.Specifically, at step 5© the reconstructed image is passedto the Encoder to acquire the reconstructed embedding B©,and then the difference between the integrated embeddingA© and the reconstructed embedding B© is minimized. Thesecond one is the style loss. Particularly, the means andvariances of a few selected internal layers of the Encoderare computed for both the generated image and the origi-nal style image. The difference of these values of the twoimages is minimized. The Decoder optimization process isconducted on the original training dataset of target modelM (under attack). Intuitively, the decoder is trained to un-derstand the style differences so that it can decode featurestyle differences to realistic pixel space style differences,by observing the possible style differences.
When launching the attack ((b) in Fig. 2), a test input imageis fed to the Encoder and goes through the same processas in the Decoder training phase. The key differences arethat only one input image is required and the Decoderis fixed in this phase. Given a target model M (underattack), the reconstructed image is fed to M at step 6© toyield prediction E©. As the attack goal is to induce M tomisclassify, the difference between prediction E© and atarget output label (different from E©) is considered theadversarial loss for launching the attack. In addition, thecontent loss between A© and B© is also included. The attackupdates the means and variances of embedding matricesat step 7© with respect to the adversarial loss and contentloss. The final reconstructed image that induces the targetmodel M to misclassify is a successful adversarial sample.
3.1 Definitions
In this section, we formally define feature space attack.Considering a typical classification problem, where thesamples x ∈ Rd and the corresponding label y ∈{0, 1, . . . , n} jointly obey a distribution D(x, y). Givena classifier M : Rd → {0, 1, . . . , n} with parameterθ. The goal of training is to find the best parameterargmaxθ P(x,y)∼D[M(x; θ) = y]. Empirically, peopleassociate a continuous loss functionLM,θ(x, y), e.g. cross-entropy, to measure the difference between the predic-tion and the true label. And the goal is rewritten asargminθ E(x,y)∼D[LM,θ(x, y)]. We use LM in short forLM,θ in the following discussion. In adversarial learning,the adversary can introduce a perturbation δ ∈ S ⊂ Rdto a natural samples (x, y) ∼ D. For a given samplex with label y, an adversary chooses the most maliciousperturbation argmaxδ∈S LM (x+ δ, y) to make the clas-sifier M predict incorrectly. Normally S is confined as an`p-ball centered on 0. In this case, the `p norm of pixel
space differences measures the distance between adversar-ial samples (i.e., x+ δ that causes misclassification) andthe original samples. Thus we refer to this attack model asthe pixel space attack. Most existing adversarial attacksfall into this category. Different from adding bounded per-turbation in the pixel space, feature space attack appliesperturbation in the feature space such that an encoder (toextract the feature representation of the benign input) anda decoder function (that translates perturbed feature val-ues to a natural-looking image that closely resembles theoriginal input in humans’ perspective).
Formally, consider an encoder function f : Rd → Re anda decoder function f−1 : Re → Rd. The former encodesa sample to an embedding b ∈ Re and the latter restoresan embedding back to a sample. A perturbation functiona ∈ A : Re → Re transforms a given embedding toanother. For a given sample x, the adversary chooses thebest perturbation function to make the model M predictincorrectly.
maxa∈ALM [f−1 ◦ a ◦ f(x), y]. (1)
Functions f and f−1 need to satisfy additional proper-ties to ensure the attack is meaningful. We call them thewellness properties of encoder and decoder.
Wellness of Encoder f . In order to get a meaningful em-bedding, there ought to exist a well-functioning classifierg based on the embedding, with a prediction error rate lessthan δ1.
∃g : Re → {0, 1, . . . , n},P(x,y)∼D[g(f(x)) = y]
≥ 1− δ1, for a given δ1.(2)
In practice, this property can be easily satisfied as onecan construct g from a well-functioning classifier M , bydecomposing M =M2 ◦M1 and take M1 as f and M2 asg.
Wellness of Decoder f−1. Function f−1 is essentially atranslator that translates what the adversary has done onthe embedding back to a sample in Rd. We hence requirethat for all possible adversarial transformation a ∈ A,f−1 ought to retain what the adversary has applied to theembedding in the restored sample.
∀a ∈ A, let Ba = a ◦ f(x), E(x,y)∼D
||f ◦ f−1(Ba)−Ba||2 ≤ δ2, for a given δ2.(3)
This ensures a decoded (adversarial) sample induce the in-tended perturbation in the feature space. Note that f−1 canalways restore a benign sample back to itself. This is equiv-alent to requiring the identity function in the perturbationfunction set A.
Given (f, f−1,A) satisfying the aforementioned proper-ties, we define Eq. (1) as a feature space attack. Under thisdefinition, pixel space attack is a special case of featurespace attack. For an `p-norm ε-bounded pixel space attack,
4
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
i.e., S = {||δ||p ≤ ε}, we can rewrite it as a feature-spaceattack. Let encoder f and decoder f−1 be an identityfunction and let A = ∪||δ||p≤ε{a : a(m) =m+ δ}.
One can easily verify the wellness of f and f−1. Notethat the stealthiness of feature space attack depends onthe selection of A, analogous to that the stealthiness ofpixel space attack depending on the `p norm. Next, wedemonstrate two stealthy feature space attacks.
3.2 Attack Design
Decoder Training. Our decoder design is illustrated inFig. 2a. It is inspired by style transfer in [10]. To train thedecoder, we enumerate all the possible pairs of images ineach class in the original training set and use these pairsas a new training set. We consider each pair has the samecontent features (as they belong to the same class) andhence their differences essentially denote style differences.By training the decoder on all possible style differences(in the training set) regardless the output classes, we havea general decoder that can recognize and translate arbitrarystyle perturbation. Formally, given a normal image xp andanother image xq from the same class as xp, the trainingprocess first passes them through a pre-trained Encoder f(e.g., VGG-19) to obtain embeddings Bp = f(xp), B
q =f(xq) ∈ RH·W ·C , where C is the channel size, and Hand W are the height and width of each channel. For eachchannel c, the mean and variance are computed across thespatial dimensions (step 2© in Fig. 2a). That is,
µBc =1
HW
H∑h=1
W∑w=1
Bhwc
σBc =
√√√√ 1
HW
H∑h=1
W∑w=1
(Bhwc − µBc)2 .
(4)
We combine the embeddings Bp, Bq from the two inputimages using the following equation:
∀c ∈ [1, 2, ..., C], Boc = σBqc
(Bpc − µBpcσBpc
)+ µBqc , (5)
where Boc is the result embedding of channel c. Intuitively,the transformation retains the shape of Bp while enforcingthe mean and variance of Bq. Bo is then fed to the De-coder f−1 for reconstructing the image with the content ofxp and the style of xq (steps 3© & 4© in Fig. 2a). In orderto generate a realistic image, the reconstructed image ispassed to Encoder f to acquire the reconstructed embed-dingBr = f ◦f−1(Bo) (step 5©). The difference betweenthe combined embedding Bo and the reconstructed embed-ding Br, called the content loss, is minimized using thefollowing equation during the Decoder training:
Lcontent = ||Br −Bo||2. (6)
Note that the similarity between the input and the outputis implicitly ensured by the fact that the encoder is rela-tively shallow and well-trained. In addition, some internal
layers of Encoder f are selected, whose means and vari-ances (computed by Equation 4) are used for representingthe style of input images. The difference of these valuesbetween the style image xq and the reconstructed imagexr, called the style loss, is minimized when training theDecoder. It is defined as follows:
Lstyle =∑i∈L||µ(φi(xq))− µ(φi(xr))||2
+∑i∈L||σ(φi(xq))− σ(φi(xr))||2
(7)
where φi(·) denotes layer i of Encoder f and L the set oflayers considered. In this paper, L consists of conv1_1,conv2_1, conv3_1 and conv4_1 for the ImageNet dataset,and conv1_1 and conv2_1 for the CIFAR-10 and SVHNdatasets. µ(·) and σ(·) denote the mean and the vari-ance, respectively. The Decoder training is to minimizeLcontent + Lstyle.
Two Feature Space Attacks. Recall in the attack phase(Fig. 2b), the encoder and decoder are fixed. The stylefeatures of a benign image are perturbed while the con-tent features are retained, aiming to trigger misclassifica-tion. The pre-trained decoder then translates the perturbedembedding back to an adversarial sample. During pertur-bation, we focus on minimizing two loss functions. Thefirst one is the adversarial loss LM whose goal is to in-duce misclassification. The second one is similar to thecontent loss in the Decoder training (Eq. 6). Intuitively,although the decoder is trained in a way that it is supposedto decode with minimal loss, arbitrary style perturbationmay still cause substantial loss. Hence, such loss has to beconsidered and minimized during style perturbation.
With two different sets of transformations A, we devise tworespective kinds of feature space attacks, feature augmen-tation attack and feature interpolation attack. For featureaugmentation attack, attacker can change both the meanand standard deviation of each channel of the benign em-bedding independently. The boundary of increments ordecrements are set by `∞-norm under logarithmic scale(to achieve stealthiness). Specifically, given two perturba-tion vectors τµ for the mean and τσ for the variance, bothhave the same dimension C as the embedding (denotingthe C channels) and are bounded by ε, the list of possibletransformations A is defined as follows.
A = ∪||τσ||∞≤ε and ||τµ||∞≤ε, τσ and τµ∈RC{a : a(B)h,w,c = eτ
σc (Bh,w,c − µBc) + eτ
µc µBc
} (8)
Note that µB denotes the means of embedding B for theC channels. The subscript c denotes a specific channel.The transformation essentially enlarges the variance of theembedding at channel c by a factor of eτ
σc and the mean
by a factor of eτµc .
For the feature interpolation attack, the attacker pro-vides k images as the style feature prototypes. LetSµ,Sσ be the simplex determined by ∪i∈[1,2,...,k]µf(xi)
5
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
and ∪i∈[1,2,...,k]σf(xi) respectively. The attacker can mod-ify the vectors of µB and σB to be any point on the simplex.
A = ∪σi∈Sσµi∈Sµ
{a : a(B)h,w,c = σi ·
Bh,w,c − µBcσBc
+ µi
}(9)
Intuitively, it enforces a style constructed from an interpo-lation of the k style prototypes. Our optimization methodis a customized iterative gradient method with gradientclipping (see Appendix A).
In pixel level attacks, two kinds of optimization techniquesare widely used: Gradient Sign Method, e.g., PGD [8], andusing continuous function, e.g. tanh, to approximate andbound `∞, e.g., in C&W [2]. However in our context, wefound these two techniques do not perform well. Usinggradient sign tends to induce a large content loss whileusing tanh function inside the feature space empiricallycauses numerical instability. Instead, we use the iterativegradient method with gradient clipping. Specifically, Wefirst calculate the gradient of loss L with respect to vari-ables (e.g., τµc and τσc in Equation (8)). The gradient isthen clipped by a constant related to the dimension of vari-ables. ||∇L||∞ ≤ 10/
√Dimension of variable. Then an
Adam optimizer iteratively optimizes the variables usingthe clipped gradients.
For the optimization of Feature Interpolation Attack, stylevectors are constrained inside the polygon. To convenientlyenforce such constraint, a tensor of variables as coefficientsof vertices in the simplex are used to represent the stylevectors. These variables are clipped to be positive andthe sum of which is normalized to be one during everyoptimization step. Therefore, these style vectors stay insidethe simplex denoted by Equation (9).
4 Attack Settings
The two proposed feature space attacks have similar perfor-mance on various experimental settings. Unless otherwisestated, we use feature augmentation attack as the defaultmethod. For the Encoder, we use VGG-19 from the inputlayer up to the relu4_1 for ImageNet, and up to relu2_1for CIFAR-10 and SVHN . To launch attacks, we set the`∞-norm of embedding, ε, in Eq. (8) to ln(1.5) for all theuntargeted attacks and ln(2) for all the targeted attacks.We randomly select 1,000 images to perform the attackson ImageNet. For CIFAR-10 and SVHN, we use all theinputs in the validation set.
5 Evaluation
Three datasets are employed in the experiments: CIFAR-10 [25], ImageNet [26] and SVHN [27]. The feature spaceattack settings can be found in Appendix B. We use 7 state-of-the-art detection and defense approaches to evaluate theproposed feature space attack. Detection approaches aimto identify adversarial samples while they are provided to
0.3 0 1 0.656250.325 2 1.05 0.4375
0.3 4 1.1 0.343750.25 6 1.15 0.234375
0.425 8 1.2 0.06250.35 10 1.25 0.0156250.25 12 1.3 0.015625
0.325 14 1.35 00.375 16 1.4 0
0.25 18 1.45 00.25 20 1.5 0
Preference To Style Attack Accuracy Rate:
ACC PREFRENCE1 0.78125 0.478
1.05 0.609375 0.4041.1 0.515625 0.486
1.15 0.421875 0.491.2 0.234375 0.38
1.25 0.140625 0.461.3 0.09375 0.432
1.35 0.078125 0.381.4 0.0625 0.392
1.45 0.078125 0.3541.5 0.046875 0.358
0.419455
00.20.40.60.81
0
0.2
0.4
0.6
1 1.05 1.1 1.15 1.2 1.25 1.3 1.35 1.4 1.45 1.5
Accu
racy
Pref
eran
ce R
ate
Scale of Feature Space Change
Human Preference
Preference Rate Accuracy
Figure 3: Human preference evaluation. The left y-axis(and blue bar) represents the percentage of user preferencetowards feature space attack images. The right y-axis (andorange line) denotes the test accuracy of models under fea-ture space attack. The x-axis presents the scale of featurespace perturbation eε in Eq. (8). The horizontal red line de-notes where users cannot distinguish between adversarialsamples and original images.
a DNN. They often work as an add-on to the model anddo not aim to harden the model. We use two state-of-the-art adversarial example detection approaches proposed by[13] and [28] to test our attack. Defense approaches, onthe other hand, harden models such that they are robustagainst adversarial example attacks. Existing state-of-the-art defense mechanisms either use adversarial training orcertify a bound for each input image. We adopt 5 state-of-the-art defense approaches in the literature [8, 9, 14, 29,30] for evaluation. Note that while these techniques areintended for pixel space attacks, their effectiveness for ourattack is unclear. We are unaware of any detection/defensetechniques for the kind of feature attacks we are proposing.
5.1 Quality of Feature Space Adversarial Examplesby Human Study and Distance Metrics
In the first experiment, we conduct a human study to mea-sure the quality of feature space attack samples. We followthe same procedure as in [31, 12]. Users are given 50 pairsof images, each pair consisting of an original image andits transformed version (by feature space attack). Theyare asked to choose the realistic one from each pair. Theimages are randomly selected and used in the followingtrials. Each pair appears on screen for 3 seconds, and isevaluated by 10 users. Every user has 5 chances for prac-tice before the trials begin. In total, 110 users completedthe study. We repeat the same study for different featurespace attack scales on ResNet-50 as shown in Fig. 3. Onaverage, 41.9% of users choose our adversarial samplesover original images. This indicates that the feature spaceattack is largely imperceptible.
We also carry out a set of human studies to qualitativelymeasure images generated by different attacks: PGD [8],feature space attack, semantic attack [12], HSV attack [11],manifold attack [24] and art attack [22]. The results areshown in Table 1. The first columns show the two attacksin comparison. The second column presents the humanpreference rate. The third column is the attack success
6
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
Pref. Succ.
PGD 60 58Feature Space 40 88
Pref. Succ.
Semantic 33 100Feature Space 67 100
Pref. Succ.
HSV 20 64Feature Space 79 100
Pref. Succ.
Manifold 27 100Feature Space 73 100
Pref. Succ.
Art 11 100Feature Space 89 100
Table 1: Human preference and success rate for differentattacks.
rate. In the first table, the two attacks are conducted on amodel with the denoise(t,1) defense [14] (in order to avoid100% attack success rate). In the following four tables, theattacks are performed on a ResNet-50. We observe that thequality of feature space attack samples is comparable tothat of PGD attack and the former has a higher attack suc-cess rate. Feature space attack also outperforms semanticattack, HSV attack, manifold attack and art attack in visualquality while achieving a higher/comparable attack suc-cess rate. That is, 67% or more users prefer feature spaceattack samples to the others. The comparison with artattack supports the benefit of leveraging feature-space dur-ing attack.The comparison with manifold attack stressesthe importance of manipulating secondary features. Thegenerated images by these attacks and comparison detailscan be seen in Appendix E.
We study the `-norm distances in both the pixel space andthe feature space for both pixel space attacks and featurespace attacks. We observe that in the pixel space, the intro-duced perturbation by feature space attack is much largerthan that of the PGD attack. In the feature space, our attackhas very similar distances as PGD. Fig. 1 and 4 (in Ap-pendix) show that the adversarial samples have only styledifferences that are natural or even human imperceptible.Detailed discussion can be found in Appendix C. We alsostudy the characteristics of the adversarial samples gener-ated by different feature space attacks and attack settings.Please see Appendix D.
5.2 Attack against Detection Approaches
We use two state-of-the-art adversarial sample detec-tion approaches “The Odds are Odd” (O2) [13] 2 andfeature-space detection method “Deep k-Nearest Neigh-bors” (DkNN) [28].
O2 detects adversarial samples by adding random noise toinput images and observing activation changing at a cer-tain layer of a DNN. Specifically, O2 uses the penultimate
2O2 is recently bypassed by [32], where the attacker alreadyknows the existence of the defense. In our case, however, we areable to evade the detection method without knowing its existenceor mechanism.
layer (before the logits layer) as the representation of inputimages. It then defines a statistical variable that measurespairwise differences between two classes computed fromthe penultimate layer. The authors observed that adver-sarial samples differ significantly from benign samplesregarding this variable when random noise is added. Byperforming statistical test on this variable, O2 is able todetect PGD attacks [8] with over 99% detection rate onCIFAR-10 with bound `∞ = 8/255 and on ImageNet with`∞ = 2/255. It also has over 90% detection rate againstPGD and C&W [2] attacks under `2 metric on CIFAR-10.
Table 2 shows the results of O2 on detecting differentinput samples. The first two columns are the datasets andmodels used for evaluation. The third column denotesthe prediction accuracy of models on normal inputs. Thefollowing three columns present the detection rate of O2on normal inputs, PGD adversarial samples, and featurespace adversarial samples, respectively. The detectionrate on normal inputs indicates that O2 falsely recognizesnormal inputs as adversarial, which are essentially falsepositives. We can observe that O2 can effectively detectPGD attack on both datasets, but fails to detect featurespace attack. Particularly, O2 has only 0.04% detectionrate on CIFAR-10, which indicates that O2 can be evadedby feature space attack. As for ImageNet, O2 can detect25.30% of feature space adversarial samples but at the costof a 19.20% false positive rate3. The results show that O2is ineffective against feature space attack.
Table 3 shows the results for Deep K Nearest Neighbour.Due to memory limits, we only test on the CIFAR-10dataset. The second column denotes models employed forevaluation including the default one used in the original pa-per (CNN+MLP). The third column shows model accuracyon benign inputs. The last two columns present detectionrate on PGD and feature space attacks. We can observethat DkNN has much lower detection rate on feature spaceattack compared to PGD, despite the fact that DkNN usesfeature space data for detecting adversarial samples.
5.3 Attack against Defense Approaches
We evaluate our feature space attack on 5 state-of-the-art adversarial training approaches: Madry [8], TRADES[9], Denoise [14], Adaption [29], and Pixel-DP [33]. ForDenoise, the original paper only evaluated on targetedattacks. We conduct experiments on both targeted anduntargeted attacks. We use Denoise (t,1) to denote the top-1accuracy of hardened model on targeted attack and Denoise(u,5) the top-5 accuracy on untargeted attack. We launchthe PGD `∞ attack as well as our feature space attack onthe four defense approaches. The results are shown inTable 4. The first column denotes attack methods, where“None” presents the model accuracy on benign inputs and“Decoder” denotes the samples directly generated fromthe decoder without any feature space perturbation. The
3The parameters used for ImageNet are not given in the orig-inal paper. We can only reduce to this false positive rate afterparameter tuning.
7
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
Dataset Model AccuracyDetection Rate
Normal PGD Feature Space
CIFAR-10 ResNet-18 91.95 0.95 99.61 0.04ImageNet ResNet-50 75.20 19.20 99.40 25.30
Table 2: O2 detection rate on normal inputs and adversarialsamples.
Dataset Model Accuracy Detection Rate
PGD Feature Space
CIFAR-10 CNN+MLP 53.93 3.92 1.95ResNet-18 81.51 11.32 5.42
Table 3: DkNN detection rate on normal inputs and adver-sarial samples.
AttackSVHN CIFAR-10
Adaption Madry TRADES Pixel-DP 4
None 84.84 77.84 84.97 44.3PGD 52.84 41.43 54.02 30.7
Decoder 84.81 77.35 84.01 50.0Feature Space 2.56 7.05 8.64 0.0
AttackImageNet
Denoise (t,1) Denoise (u,1) Denoise (u,5)
None 61.25 61.25 78.12PGD 42.60 12.50 27.15
Decoder 64.68 64.00 82.37Feature Space 11.41 1.25 1.25
Table 4: Evaluation of adversarial attacks against variousdefense approaches.
latter is to show that the Decoder can generate faithful andnatural images from embeddings. The following columnsshow different defense approaches (second row) appliedon various datasets (first row). We can see that the PGDattack can reduce model accuracy to some extent whendefense mechanisms are in place. Feature space attack,on the other hand, can effectively reduce model accuracydown to less than 12%, and most results are one order ofmagnitude smaller than PGD. Especially, model accuracyon ImageNet is only 1.25% when using untargeted attack,even in the presence of the defense technique.
From the aforementioned results, we observe that existingpixel space detect/defense techniques are largely ineffec-tive as they focus on pixel space. While it may be possibleto extend some of these techniques to protect feature space,the needed extension remains unclear to us at this point.We hence leave it to our future work. For example, it isunclear how to extend O2, which leverages the penulti-mate layer to detect anomaly and hence should have beeneffective for our attack in theory.
Towards Feature Space Adversarial Training. We con-duct a preliminary study on using feature space attack toperform adversarial training. For comparison, we alsoperform the PGD adversarial training and use semantic
attack to perform adversarial training. We evaluate theadversarially trained models against feature space (FS) at-tack, HSV attack, semantic (SM) attack, and PGD attack,with the first three in the feature space. We find that PGDadversarial training is most effective against PGD attack(55% attack success rate reduction) and has effectivenessagainst SM attack too (22% reduction), but not FS or HSVattack. Feature space adversarial training can reduce theFS attack success rate by 27% and the HSV attack by 13%,but not others. Adversarial training using semantic attackcan reduce semantic attack success rate by 34% and PGDby 38%, but not others. This suggests that different attacksaim at different spaces and the corresponding adversar-ial trainings may only enhance the corresponding targetspaces. Note that the robustness improvement of featurespace adversarial training is not as substantial as PGDtraining in the pixel space. We believe that it is becauseeither our study is preliminary and more setups need tobe explored; or, feature space adversarial training may beinherently harder and demand new methods. We will leaveit to our future study. More details (e.g., ablation study)can be found in Appendix F.
6 Conclusion
We propose feature space adversarial attack on DNNs. Itis based on perturbing style features and retaining contentfeatures. Such attacks inject natural style changes to in-put images to cause model misclassification. Since theyusually cause substantial pixel space perturbations and ex-isting detection/defense techniques are mostly for boundedpixel space attacks, these techniques are not effective forfeature space attacks.
7 Acknowledgments
This research was supported, in part by NSF 1901242 and1910300, ONR N000141712045, N000141410468 andN000141712947, and IARPA TrojAI W911NF-19-S-0012.Any opinions, findings, and conclusions in this paper arethose of the authors only and do not necessarily reflect theviews of our sponsors.
8
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
References[1] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob
Fergus. Intriguing Properties of Neural Networks. In International Conference on Learning Representations(ICLR), 2014.
[2] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In Proceedings of38th IEEE Symposium on Security and Privacy (SP), pages 39–57. IEEE, 2017.
[3] Yao Qin, Nicholas Carlini, Ian Goodfellow, Garrison Cottrell, and Colin Raffel. Imperceptible, Robust, andTargeted Adversarial Examples for Automatic Speech Recognition. In Proceedings of the 36th InternationalConference on Machine Learning (ICML), pages 5231–5240, 2019.
[4] Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. Hotflip: White-box adversarial examples for textclassification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL),pages 31–36, 2018.
[5] Shasha Li, Ajaya Neupane, Sujoy Paul, Chengyu Song, Srikanth V Krishnamurthy, Amit K Roy Chowdhury, andAnanthram Swami. Adversarial perturbations against real-time video classification systems. In Proceedings of26th Annual Network and Distributed System Security Symposium (NDSS), 2019.
[6] Guanhong Tao, Shiqing Ma, Yingqi Liu, and Xiangyu Zhang. Attacks meet interpretability: Attribute-steereddetection of adversarial samples. In Advances in Neural Information Processing Systems (NeurIPS), pages7717–7728, 2018.
[7] Shiqing Ma, Yingqi Liu, Guanhong Tao, Wen-Chuan Lee, and Xiangyu Zhang. Nic: Detecting adversarial sampleswith neural network invariant checking. In Proceedings of 26th Annual Network and Distributed System SecuritySymposium (NDSS), 2019.
[8] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deeplearning models resistant to adversarial attacks. In Proceedings of 6th International Conference on LearningRepresentations (ICLR), 2018.
[9] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoreticallyprincipled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML),pages 7472–7482, 2019.
[10] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1501–1510, 2017.
[11] Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. In Proceedings of the IEEE Conferenceon Computer Vision and Pattern Recognition Workshops, pages 1614–1619, 2018.
[12] Anand Bhattad, Minjin Chong, Kaizhao Liang, Bo Li, and David Forsyth. Unrestricted adversarial examples viasemantic manipulation. In International Conference on Learning Representations ICLR Conference, 2020, 2020.
[13] Kevin Roth, Yannic Kilcher, and Thomas Hofmann. The odds are odd: A statistical test for detecting adversarialexamples. In International Conference on Machine Learning (ICML), pages 5498–5507, 2019.
[14] Cihang Xie, Yuxin Wu, Laurens van der Maaten, Alan L Yuille, and Kaiming He. Feature denoising for improvingadversarial robustness. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 501–509, 2019.
[15] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer using convolutional neural networks.In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2414–2423,2016.
[16] Chuan Li and Michael Wand. Precomputed real-time texture synthesis with markovian generative adversarialnetworks. In European Conference on Computer Vision (ECCV), pages 702–716, 2016.
[17] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution.In European Conference on Computer Vision (ECCV), pages 694–711, 2016.
[18] Vincent Dumoulin, Jonathon Shlens, and Manjunath Kudlur. A learned representation for artistic style. InProceedings of 5th International Conference on Learning Representations (ICLR), 2017.
[19] Yijun Li, Chen Fang, Jimei Yang, Zhaowen Wang, Xin Lu, and Ming-Hsuan Yang. Diversified texture synthesiswith feed-forward networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 3920–3928, 2017.
[20] Nathan Inkawhich, Wei Wen, Hai Li, and Yiran Chen. Feature space perturbations yield more transferableadversarial examples. pages 7059–7067, 06 2019. doi: 10.1109/CVPR.2019.00723.
9
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
[21] Cassidy Laidlaw and Soheil Feizi. Functional adversarial attacks. In Advances in Neural Information ProcessingSystems, pages 10408–10418, 2019.
[22] Vinay Uday Prabhu and John Whaley UnifyID. Art-attack ! on style transfers with textures , label categories andadversarial examples. 2018.
[23] Yang Song, Rui Shu, Nate Kushman, and Stefano Ermon. Constructing unrestricted adversarial examples withgenerative models. In Advances in Neural Information Processing Systems, pages 8312–8323, 2018.
[24] David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and generalization. CoRR,abs/1812.00740, 2018. URL http://arxiv.org/abs/1812.00740.
[25] Alex Krizhevsky et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.[26] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej
Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. InternationalJournal of Computer Vision, 115(3):211–252, 2015.
[27] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in naturalimages with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised FeatureLearning, 2011.
[28] Nicolas Papernot and Patrick D. McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robustdeep learning. CoRR, abs/1803.04765, 2018. URL http://arxiv.org/abs/1803.04765.
[29] Chuanbiao Song, Kun He, Liwei Wang, and John E. Hopcroft. Improving the generalization of adversarial trainingwith domain adaptation. In International Conference on Learning Representations (ICLR), 2019.
[30] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness toadversarial examples with differential privacy. In 2019 IEEE Symposium on Security and Privacy (SP), pages656–672. IEEE, 2019.
[31] Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In Bastian Leibe, Jiri Matas, NicuSebe, and Max Welling, editors, Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, TheNetherlands, October 11-14, 2016, Proceedings, Part III, volume 9907 of Lecture Notes in Computer Science,pages 649–666. Springer, 2016. doi: 10.1007/978-3-319-46487-9\_40. URL https://doi.org/10.1007/978-3-319-46487-9_40.
[32] Hossein Hosseini, Sreeram Kannan, and Radha Poovendran. Are odds really odd? bypassing statistical detectionof adversarial examples. CoRR, abs/1907.12138, 2019. URL http://arxiv.org/abs/1907.12138.
[33] Mathias Lecuyer, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, and Suman Jana. Certified robustness toadversarial examples with differential privacy. In Proceedings of 40th IEEE Symposium on Security and Privacy(SP), 2019.
[34] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. CoRR, abs/1605.07146, 2016. URLhttp://arxiv.org/abs/1605.07146.
10
Appendix
A Optimization
In pixel level attacks, two kinds of optimization techniques are widely used: Gradient Sign Method, e.g., PGD [8],and using continuous function, e.g. tanh, to approximate and bound `∞, e.g., in C&W [2]. However in our context,we found these two techniques do not perform well. Using gradient sign tends to induce a large content loss whileusing tanh function inside the feature space empirically causes numerical instability. Instead, we use the iterativegradient method with gradient clipping. Specifically, We first calculate the gradient of loss L with respect to variables(e.g., τµc and τσc in Equation (8)). The gradient is then clipped by a constant related to the dimension of variables.||∇L||∞ ≤ 10/
√Dimension of variable. Then an Adam optimizer iteratively optimizes the variables using the
clipped gradients.
For the optimization of Feature Interpolation Attack, style vectors are constrained inside the polygon. To convenientlyenforce such constraint, a tensor of variables as coefficients of vertices in the simplex are used to represent thestyle vectors. These variables are clipped to be positive and the sum of which is normalized to be one during everyoptimization step. Therefore, these style vectors stay inside the simplex denoted by Equation (9).
B Attack Settings
The two proposed feature space attacks have similar performance on various experimental settings. Unless otherwisestated, we use feature augmentation attack as the default method. For the Encoder, we use VGG-19 from the inputlayer up to the relu4_1 for ImageNet, and up to relu2_1 for CIFAR-10 and SVHN . To launch attacks, we set the`∞-norm of embedding, ε, in Eq. (8) to ln(1.5) for all the untargeted attacks and ln(2) for all the targeted attacks. Werandomly select 1,000 images to perform the attacks on ImageNet. For CIFAR-10 and SVHN, we use all the inputs inthe validation set.
C Measurement of Perturbation in Pixel and Feature Spaces
To measure the magnitude of perturbation introduced by adversarial attacks, we use both `∞ and `2 distances. Inaddition, as we aim to understand how the different attacks perturb the pixel space and the feature space, we computethe distances for both spaces. For the pixel space, the calculation is discussed in Backgrond section. For the featurespace, we normalize the embeddings before distance calculation. For each channel, we use h(x) =
f(x)−µf(x)
σf(x)to
normalize the embedding produced by the Encoder f(·) given input x. The feature space difference hence can becomputed as ||h(x) − h(x′)||p. Table 5, Table 6 and Table 7 illustrate the magnitude of perturbation introduced byadversarial attacks on pixel and feature spaces for different hardened models. It can be observed that in the pixel space,the introduced perturbation by feature space attack is much larger than that of the PGD attack with `∞ and `2 distances.In the feature space, however, our feature space attack does not induce large difference between normal inputs andadversarial samples. Particularly, the difference is similar or even smaller than that by the PGD attack. As shown inFig. 1 and Fig. 4, the introduced perturbation is either insensitive to humans or even imperceptible.
AttackPixel Space Feature Space
l∞ l2 l∞ l2
PGD 0.02 1.04 11.50 53.78Decoder 0.08 1.05 10.12 36.52
Feature Space 0.12 2.01 10.51 41.83
Table 5: Magnitude of perturbation on hardened SVHN models.
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
Attack
Madry TRADES
Pixle Space Feature Space Pixle Space Feature Space
`∞ `2 `∞ `2 `∞ `2 `∞ `2
PGD 0.03 1.51 6.86 41.71 0.03 1.47 6.49 37.89Decoder 0.19 1.85 4.29 25.13 0.18 1.85 4.30 25.15
Feature Space 0.27 4.08 6.88 43.38 0.28 4.72 7.43 46.43
Table 6: Magnitude of perturbation on hardened CIFAR-10 models.
Attack
Targeted Untargeted
Pixle Space Feature Space Pixle Space Feature Space
`∞ `2 `∞ `2 `∞ `2 `∞ `2
PGD 0.06 19.48 12.08 375 0.03 9.24 16.91 227Decoder 0.87 44.98 8.26 193 0.88 44.67 15.24 214
Feature Space 0.89 69.07 9.99 283 0.86 54.89 16.09 236
Table 7: Magnitude of perturbation on hardened ImageNet models.
D Two Feature Space Attacks
We generate and visually analyze the two feature space attacks. The ImageNet model hardened by feature denoising isused for generating adversarial samples. Columns (a), (d), and (g) in Fig. 4 present the original images. Columns (b),(c), and (d) present the adversarial samples generated by the Encoder and the corresponding Decoder, with differentencoder depths. Specifically, column (b) uses conv2_1 layers, column (c) uses conv3_1 layers and column (d) usesconv4_1 layers of a pre-trained VGG-19 as the Encoder, and the Decoders are of neural network structure similar to thecorresponding Encoders but in a reverse order. Observe that as the Encoder becomes deeper, object outlines and texturesare changed in addition to colors. Columns (e) and (f) are adversarial samples generated by the feature argumentationattack (FA) and feature interpolation attack (FI). We observe that they both generate realistic images.
In column (h), we only perturb the mean of embedding whereas in column (i) we only perturb the standard deviationof embedding. The results indicate that mean values tend to represent the background and the overall color tone. Incontrast, the standard deviations tend to represent the object shape and relative color.
E Comparison with Other Attacks
Some examples used in the human studies for comparing between feature space attack and semantic attack, HSV attack,manifold attack and art attack are shown in Fig. 5. The top row presents the original images. The second row showsthe adversarial images generated by feature space attack. The third row denotes the adversarial samples produced bythe semantic attack. And the last row shows the adversarial examples by the HSV attack. We can observe that theadversarial samples generated by the feature space attack look more natural.
In the following, we introduce the settings used for comparing with other attacks. We adopt the source code of theseattacks and utilize the best settings stated in their papers. Specifically, for the HSV attack, we randomly sample 1000times and select the first successful adversarial samples. For the semantic attack, we use their provided color model andthe setting cadv8 in the original paper. Since the semantic attack is unbounded, we use the first successful adversarialsample during the optimization procedure, which is the most similar one to the original image. For the art attack, wegradually increase the interpolation coefficient as defined in the original paper and select the first successful adversarialsample. We repeat this process on the multiple art works provided by the authors. The manifold attack paper does notinclude a suitable setting to scale up to ImageNet. Thus we compare with it using the same network structure as ours.We set the internal `∞ bound in a similar manner as in the paper.
12
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
Figure 4: The adversarial samples from different feature space attack methods.
Model AccuracySuccess Rates
FS HSV SM PGD
Normal 92.1% 100% 62.2% 100% 99.6%PGD-Adv 78.1% 94.0% 65.1% 78.8% 44.2%SM-Adv 76.9% 95.7% 69.9% 66.2% 62.5%FS-Adv 82.4% 73.3% 48.3% 92.7% 92.7%
Table 8: Adversarial training results
F Towards Feature Space Adversarial Training
The ultimate goal of developing new adversarial attack is to enhance model robustness. In this section, we present apreliminary study on using feature space attack to adversarial-train the model. For comparison, we also perform thePGD adversarial training and use semantic attack to perform adversarial training. We evaluate the adversarially trainedmodels against feature space (FS) attack, HSV attack, semantic (SM) attack, and PGD attack, with the first three in thefeature space.
We adopt a standard adversarial training procedure as in [2], where adversarial samples are generated against the modeland used to train the classifier iteratively. We conduct the experiment on the CIFAR-10 dataset and the ResNet-18model. For semantic texture attack that requires a texture template image, we randomly choose one image from thetraining dataset as the template. We use adversarial weight of 500 and optimization step of 3 in a limited-memoryBFGS optimizer as in [12]. For PGD attack, we use `∞ = 8/255 for adversarial training and `∞ = 4/255 duringevaluation. For HSV attack, we set the random trail number to 100 as in [11]. For feature space adversarial training, dueto computational limit, we set the optimization step to 400 and adversarially train the model for 25 epochs. For other
13
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
(a) (b) (c) (d) (e) (f) (g) (h)
Figure 5: Examples by feature space attack, semantic attack, HSV attack, manifold attack and art attack. The first rowpresents the original images. The second row shows the adversarial images generated by feature space attack. The thirdrow denotes the adversarial samples produced by semantic attack. The fourth row shows the adversarial examples byHSV attack. The fifth row shows the results by manifold attack. And the last row shows the adversarial samples by artattack.
Model Steps Normal Accuracy Feature Space Attack Success Rates
ResNet18 50 82.1% 83.2%ResNet18 100 82.4% 78.0%ResNet18 200 81.9% 73.7%ResNet18 400 82.4% 73.3%
WResNet18 200 82.9% 72.5%ResNet28 200 84.5% 66.4%
Table 9: Ablation study for feature space adversarial training
adversarial training, we train for 100 epochs. We use the Adam optimizer with a learning rate 1e-3, and set the batchsize to 64 for all the adversarial training. A typical learning curve of FS adversarial training can be found at Figure 6.
During attack evaluation, we randomly sample 1000 images from the test set for untargeted attack. For the featureargumentation attack, we use the same parameters as discussed in Evaluation section except for the number of attacksteps, which is increased to 2000 during evaluation. It is a common practice when evaluating the performance of attacks.
Note that our settings are similar to those in pixel space adversarial training as we are not aware of existing adversarialtraining practice for feature space.
14
PUBLISHED AS A CONFERENCE PAPER AT AAAI 2021
Figure 6: Learning curve of adversarial training on ResNet18, with the number steps of feature space attack set to 400.We show the accuracy in the presence of feature space adversary and the normal accuracy of the model throughout thetraining process, on both the training set and the test set.
The results are presented in Table 8. The first column denotes the adversarially trained models. The second columnshows the normal model accuracy. The following four columns present the attack success rate of various attacks forthe different models. We observe that PGD adversarial training is most effective against PGD attack and has someeffectiveness against HSV attack, but not FS or SM attack. The semantic adversarial training can reduce the SM attacksuccess rate by 34% and the PDG attack success rate by 38%, suggesting the two may share some common attack space.But it is not effective for FS or HSV attack. Feature space adversarial training can reduce the FS attack success rate by27% and the HSV attack by 13%, but not others. This suggests that feature space attack aims at different attack spacethan PGD and semantic attacks. Note that the improvement of feature space adversarial training is not as substantial asPGD training in the pixel space. We believe that it is because either our study is preliminary and more setups need to beexplored; or, feature space adversarial training may be inherently harder and demand new methods. We will leave it toour future study.
We further study the effect of model capacities and optimization steps during adversarial training on the robustnessof feature space hardened models as shown in Table 9. For adversarial training with optimization steps of 200 ormore, we train the model for 25 epochs (instead of 100) due to the computational limit. The first column denotesdifferent model capacities used for adversarial training. The second column shows the number of optimization stepsused for feature space attack during training. To test how the model capacity affects the result of adversarial training,we correspondingly increase the width and depth of the model by using Wide-ResNet18 [34] and ResNet28. We canobserve that with the increase of optimization steps, the initial improvement of robustness is significant. When the stepsincrease to 200 and beyond, the benefit becomes limited. By using a deeper model, the result can be further improvedby 6%, while a wider model does not lead to evident improvement.
15
