63 V.3 The reachable set of an average pooling layer with an ImageStar input set is another Im-. 64 V.4 Exact reachable set of a maximum pooling layer with an ImageStar input set is a union of.
Motivation
In this thesis, we propose a verification framework for deep neural networks and neural network-based control systems using set-based reachability analysis. In particular, our approach focuses on four major challenging problems: 1) security verification of feed-forward neural networks (FNNs), 2) security verification of neural network-based control systems (NNCS), 3) robustness verification of convolutional neural networks for image classification (CNN) and 4) robustness analysis of Semantic Segmentation Networks (SSN).
Contribution
Star-Based Reachability for Verification of Feedforward Neural Networks
Star-Based Reachability for Verification of Neural Network Control Systems
ImageStar-Based Reachability for Verification of Convolutional Neural Networks . 3
Instead, we demonstrate the robustness of the network to an image attacked by perturbations bounded by arbitrary linear constraints. We can also calculate the percentage of the number of pixels in the image that are correctly classified, which is called the robustness value of the SSN corresponding to the attack.
Thesis Outline
NNV has been successfully applied to verify the security and robustness of many real-word DNNs and learning-enabled CPS. NNV has received positive feedback for industrial users, e.g. the feedback from a researcher in General Motor is given at https://github. com/verivital/nnv/issues/18).
Copyright Acknowledgments
Johnson, "NNV: A Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems," i den 32. internationale konference om computerstøttet verifikation.
Deep Neural Network Verification
- Geometric and Reachability Methods
- MILP Methods
- Satisfiability and SMT Methods
- Other Optimization-Based Methods
- Other Methods
- Summary
In particular, it is applicable to a variety of neural networks, regardless of the specific form of the activation functions. Thus, the verification problem is transformed into a calculation of the minimum or maximum output value of the neural network.
Neural Network Control System Verification
- Extended Polyhedron Approach
- Verisig Approach
- Sherlock Approach
- ReachNN Approach
- SMC Approach
- Summary
This approach calculates the multilevel-based reachable set of the neural network controller at each control step. The major challenge in this approach is the calculation of the error interval, which is solved in three steps.
Testing and Falsification
The SMC approach's central idea is the construction of a finite-state abstraction of the system, which is then analyzed by a standard reachability method to compute the set of safe initial conditions. Finally, an SMC formula is encoded to analyze the behavior of the network under the constraints on the robot dynamics and partitioned workspace.
Preliminaries
Machine Learning Models and Symbolic Verification Problem
Generalized Star Sets
Zonotope
Reachability of FNNs with ReLU Activiation Functions
- Exact and complete analysis
- Over-approximate analysis
- Zonotope pre-filter
- Reachability algorithms (code)
The worst-case complexity of the number of stars in the reachable set of an N-neurons FNN is O(2N). The worst-case complexity of the number of constraints of a star in the reachable set of an N-neuron FNN is O(N).
Dealing with Other Piecewise Activation Functions
Reachability of a satlin layer
The star-based reachability algorithm with zonotope pre-filtering for a satlin layer is given in Algorithm 2. Similar to a ReLU layer, we use a zonotope pre-filter to determine all neurons where splitting cannot occur.
Reachability of a satlins layer
For those neurons, we solve LP optimization to determine their actual ranges to partition the input set (in the exact analysis) or to construct an over-approximated reachable set (in the over-approximated analysis).
Reachability of a leaky ReLU layer
9: [lb,ub] =In.Z.getRanges.estimates the ranges of all input variables 10: map=find(ub≤0).list of neglected neurons. 9: [lb,ub] =In.Z.getRanges.estimates the ranges of all input variables 10: map=find(ub≤ −1).list of neglected neurons.
Evaluation
Safety Verification for ACAS Xu DNNs
Zonotope-based method [74]. Experimental results show that the over-approximate, zonotope-based method is significantly faster than the exact methods. The benefits of accessible cluster computing. Computational sets available are useful to intuitively observe complex network behavior.
![Figure III.3: Vertical view of a generic example of the ACAS Xu benchmark set.[41]](https://thumb-ap.123doks.com/thumbv2/123dok/10732223.0/44.918.339.636.117.340/figure-iii-vertical-view-generic-example-acas-benchmark.webp)
Robustness Ceritification of Image Classification DNNs under adversarial attacks . 40
However, constructing a complete counter input set that contains all counter inputs is a major challenge due to the nonlinearity of a neural network. This counter input set, shown in Figure III.5, is part of the input set that contains all counter inputs that make the neural network unsafe.
System Model and Problem Formulation
System model
Problem formulation
We call this initial state a "safety-critical initial state" of the system and assume that the initial conditions of all states are independent. Then we discuss handling the nonlinear transformation on the computed available set to check the security of the system, i.e. task 1, as well as searching for a safety-critical start state, i.e. assignment 2.
Reachability Analysis of Neural Network Control Systems
- Exact reachability analysis of the neural network controller
- Exact reachability analysis of the discrete linear plant
- Reachability algorithm for NNCS
- Extension to NNCS with nonlinear plants
Therefore, the exact achievable mass of the plant for the next step is also the constellation, Xk+1=AXk+BUk. The last star in the union is the initial set of device states for the next time interval [tk+1,tk+2].
Verification of Neural Network Control Systems
Safety verification
This algorithm first further divides the time betweentkandtk+1 into smaller time steps and then performs a reachable set calculation based on plant-sound linearization together with Np time steps to obtain a reachable set with Npstars (we put (note that a zonotope is also a star). The security verification of NNCS is summarized in Algorithm 6, which solves the above nonlinear optimization problem to obtain the narrowest reachable transformed set interval and uses it to verify the security of the system at each time step.
Characterization of safe initial condition
Evaluation
Advanced Emergency Braking System
- Scenario of Interest
- Safety Specification
- RL-based Controller
- System Identification and Validation
- Safety Verification of the AEBS
- Safe Initial Conditions of the AEBS
Therefore, we will check the safety of AEBS for a distance (between the car and the obstacle) of 10 to 100 meters (assume that the car is at least 10 meters away from the obstacle). From the vmax information, we visualize the safe range of initial conditions for AEBS, as shown in Figure IV.13.
Adaptive Cruise Control System
- System Decription
- Scenario of Interest
- Verification Results
- Timing Performance
The safety of the system can be observed intuitively via Figure VII.2, which shows that the relative distance is greater than the safe distance. Our approach can prove the security of the NNACC system with the fourth controller having a total of 200 neurons in some cases with reasonable verification times (less than 1 hour).
Problem formulation
ImageStar
The A4×4×1 grayscale image with a bounded perturbation b∈[−2,2] applied to the pixel at position (1,2,1) can be described as an ImageStar shown in Figure V.1. We will show later that, an ImageStar is very efficient for the reachability analysis of convolutional layers, fully connected layers, and average pooling layers.
Reachability of CNN using ImageStars
- Reachability of a convolutional layer
- Reachability of an average pooling layer
- Reachability of a fully connected layer
- Reachability of a batch normalization layer
- Reachability of a max pooling layer
- Exact reachability of a max pooling layer
- Over-approximate reachability of a max pooling layer
- Reachability of a ReLU layer
- Exact reachability of a ReLU layer
- Over-approximate reachability of a ReLU layer
Therefore, the number of ImageStars in the reachable set of maximum clustering layers can grow rapidly if each local region has more than one maximum point candidate. Given below is the worst-case complexity of the number of ImageStars in the exact achievable maximum pooling layer set.
Evaluation
Robustness Verification of MNIST Classification Networks
However, the sonotope method produces very large output ranges that cannot be used to prove the robustness of the network. The tables show that our method is less conservative than the polytope approach in all networks since the number of cases that our approach can prove the robustness of the network is greater than the one proved by the polytope method.
Robustness Verification of VGG16 and VGG19
In particular, the verification times and robustness results of the polytope and ImageStar methods are comparable when working with small input sets, except in two cases where ImageStar is faster than the polytope method. It is interesting to note that according to the verification results for the VGG and MNIST networks, deep networks appear to be more robust than shallow networks.
Exact Analysis vs. Approximate Analysis
To optimize the verification time, it is important to know the time consumed by each type of layers in the reachability analysis step. As shown in the figure, the computation of the reachable set in convolutional layers and fully connected layers.
Discussion
Effect of Input Sizes on Verification Performance
Benefit of Parallel Computing
Preliminaries
Semantic Segmentation Networks and Reachability
Adversarial Attacks and Robustness
The robustness value (RV) of a SSN corresponding to a UBAA applied to an input image is defined as . Therobustness sensitivity (RS) of a SSN corresponding to a UBAA applied to an input image is defined as.
Verification
Reachability of a Transposed (Dilated) Convolutional Layer
A reachable set of transposed convolution layer with input set ImageStar I =hc,V,Pi is another ImageStarI0=hc0,V0,Pi where c0=Trans(Dil)Conv(c) applied transpose convolution operation to anchor image, V0={v01,. Similar to the reach convolution layer [80], the transposed (dilated) convolution operation used for ImageStar is a combination of 1) a transposed (dilated) convolution operation with bias on the anchor image and 2) a transposed (dilated) convolution operation with zero bias based on the images.
Reachability of a Pixel-classification Layer
The convenience of computing the output set of a transposed convolution layer comes from the linearity of the input set and the operation itself. analysis). By determining all cross-channel max-point candidates for all pixels in the input set, we can obtain the pixel-classable set of the layer, which is also the available set of the networkRf = [pci j]h×w.
Verification Algorithm
Depending on the values of the predicate variables in the input array, a pixelpi jin array can be assigned to more than one class. To reduce computation time, we estimate the lower and upper bounds of the ImageStar input to the layer with only the ranges of the predicate variables.
Evaluation
Robustness and Sensitivity of Different Network Architectures
Interestingly, N1 is also more robust than N2 as it has a larger average robustness value (Figure VI.1 -a) and more robust pixels (Figure VI.1 -d). Finally, when the input size increases, the robustness of the maximum-pool-based network decreases faster than the average-pool-based networks (Figure VI.2 (a,d)) and its sensitivity increases faster (Figure VI.2 -b).
Verification Performance
On M2NIST networks, by considering the 21-layer (8 ReLU) transposed convolution network N5 and the 24-layer (4 ReLU) dilated convolution network N6, one can see that even when using more layers, N6 is less accurate than N5 (0.72 vs. 0.75 IoU). (see Table VI.1). In terms of robustness, N6 is also less robust and more susceptible to the attack than N5 (Figure VI.5-(a,b,d,e)).
Training robust and verification-friendly networks
Overview and Features
When an overestimated (good) reachability solver is used, such as the zonotop-based scheme or the estimated star-based solvers, the safety checker may return "safe" or "uncertain" (unknown). In this case, the forger automatically calls the evaluator to generate simulation traces to find a counterexample.
Set Representations and Reachability Algorithms
- Polyhedron [83]
- Star Set [84, 81] (code)
- Zonotope [74] (code)
- Abstract Domain [75]
- ImageStar Set [80] (code)
We refer readers to [81, 82] for a detailed discussion of the extensions of the star-based reachability algorithms for NNCSs with linear/nonlinear plant models. Similar to the over-approximation algorithm using star sets, the sonotope algorithm calculates an over-approximation of the exact reachable set of a FFNN.
Evaluation
Safety verification of ACAS Xu networks
We note that the best sequences of states can be calculated by solving the LP optimization. The approximate star method is 547×faster than Reluplex, 374×faster than Marabou, 2151×faster than Marabou-DnC, and 8×faster than ReluVal.
Safety Verification of Adaptive Cruise Control System
When the initial speed of the lead vehicle is less than 27 (m/s), the ACC system with the discrete device model is unsafe. As shown in Table VII.3, verification for the linear model using the overestimation method is on average 22.7× faster than for the nonlinear model.
Conclusion
Future Directions
- Scalability vs. Conservativeness
- Formal specifications and compositional verification
- Verification for Recurrent Neural Networks
- Run-time Verification for Neural Network Control Systems
- Robust and Safe Learning
Deep neural networks for acoustic modeling in speech recognition: Shared views of four research groups. IEEE Journal of Signal Processing. Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems.
Verification approaches for neural networks
Verification approaches for NNCS
Maximum robustness values (δ max ) of image classification networks with different meth-
Hyper-parameters for DDPG algorithm
Reachability analysis times (measured in seconds) of the exact and over-approximate star
Safe initial conditions for the AEBS with RL controller in which d is the distance range
Verification results for the ACC system in which V T is the verification time and controller
Falsification results for NNACC system with different neural network controllers using
Verification results of the small MNIST CNN
Verification results of the medium MNIST CNN
Verification results of the large MNIST CNN
Verification results of VGG networks
Verification results of the VGG16 and VGG19 in which V T is the verification time (in
Example output reachable set computation for a neural network with 3 inputs, 2 outputs,
An example of a stepReLU operation on a layer with two neurons
The star set approach is less conservative than the zonotope [74] and new abstract-domain
Vertical view of a generic example of the ACAS Xu benchmark set.[41]
Reachable sets of N 41 network w.r.t property φ 4 with different methods
Neural network control system (NNCS)
Emergency Braking System Architecture
Illustration of Emergency Braking System
Obtaining the required number of reachability analysis steps from the full-braking charac-
Emergency Braking System Simulink Diagram
Validation of the Simulink model of AEBS. The Simulink model captures well the behav-
Safe initial conditions for full braking action
Set of initial conditions that needs to be verified for the AEBS with the RL controller
Reachable sets of the AEBS computed by the polyhedron and the interval approaches be-
An overview of NNV
Run-time verification example
