The existing CNN models are known to perform inference in tens of milliseconds on recent off-the-shelf smartphones [31]. Therefore, to apply NeuroValve to mobile devices in practice, the delay overhead from NeuroValve should be reasonably small. Table 4 summarizes the average time to perform an inference on the MobileNet V3-Large model with NeuroValve. When the model is loaded, NeuroValve runs its analyzer and the governor and runssysfsI/O function to adjust CPU frequency and memory bandwidth for each block. We measure the time from the model load to the actual adjustment and consider it as
Function Time Portion
Analyzer 0.01ms 0.02%
Governor 0.3ms 0.56%
sysfsI/O 0.99ms 1.56%
Inference 62.25ms 97.86%
Total 63.61 100%
Table 4: System overhead of NeuroValve in time.
overhead. However, note that NeuroValve does not delay any inference because the computation for NeuroValve is made parallel to the inference. This overhead only captures the time till the optimal adjustment is made for each block. As shown in Table 4, thanks to the light computation from our decision tree-based design, the total time overhead of NeuroValve is about 1.3 ms, which accounts for only 2.24% of the total time for a CNN inference8. This is acceptable given the improvement in EDP that NeuroValve provides. Moreover, considering thesysfs I/O takes the significant portion, we can further reduce the overhead by avoiding to adjust CPU clock frequency and memory bandwidth when the adjustment needs to be made to a neighboring level because such a small change would result in similar energy consumption and delay.
8
MobileNet V3-S MobileNet V3-L EfficientNet-B0 MobileNet V2 -25%
25%
50%
75%
100%
Delay(%)
Default
NV (w=0) NV (w=0.25) NV (w=0.5) NV (w=0.75) NV (w=1) SysScale
(a) Delay (the lower the better)
MobileNet V3-S MobileNet V3-L EfficientNet-B0 MobileNet V2 -50%
-40%
-30%
-20%
-10%
10%
Energy(%)
Default
NV (w=0) NV (w=0.25) NV (w=0.5) NV (w=0.75) NV (w=1) SysScale
(b) Energy consumption (the lower the better)
MobileNet V3-S MobileNet V3-L EfficientNet-B0 MobileNet V2 -30%
-20%
-10%
10%
20%
EDP(%)
Default
NV (w=0.25) NV (w=0.5) NV (w=0.75) NV (w=1) SysScale NV (w=0)
(c) EDP (notwEDP for comparison) (the lower the better)
Figure 11: Delay, energy consumption, and EDP comparison of NeuroValve (NV(w)) and SysScale over the Android default in four CNN models on Pixel 3.
Min Max NV (w=0)
NV (w=0.5) NV (w=1) Default SysScale
CPU Frequency Levels
Usage(%)
0%
20%
40%
60%
80%
100%
Min Max
NV (w=0) NV (w=0.5) NV (w=1) Default SysScale
Memory Bandwidth Levels
Usage(%)
0%
20%
40%
60%
80%
100%
(a) MobileNet V3-Small
Min Max
NV (w=0) NV (w=0.5) NV (w=1) Default SysScale
CPU Frequency Levels
Usage(%)
0%
20%
40%
60%
80%
100%
Min Max
NV (w=0) NV (w=0.5) NV (w=1) Default SysScale
Usage(%)
Memory Bandwidth Levels
0%
20%
40%
60%
80%
100%
(b) MobileNet V3-Large
Min Max
NV (w=0) NV (w=0.5) NV (w=1) Default SysScale
CPU Frequency Levels
Usage(%)
0%
20%
40%
60%
80%
100%
Min Max
NV (w=0) NV (w=0.5) NV (w=1) Default SysScale
Usage(%)
Memory Bandwidth Levels
0%
20%
40%
60%
80%
100%
(c) EfficientNet-B0
Min Max
NV (w=0) NV (w=0.5) NV (w=1) Default SysScale
CPU Frequency Levels
Usage(%)
0%
20%
40%
60%
80%
100%
Min Max
NV (w=0) NV (w=0.5) NV (w=1) Default SysScale
Usage(%)
Memory Bandwidth Levels
0%
20%
40%
60%
80%
100%
(d) MobileNet V2
Figure 12: Resource usage distribution of NeuroValve (NV(w)) and other governors during inferences with four CNN models on Pixel 3: CPU frequency (left) and memory bandwidth (right).
MobileNet V3-S MobileNet V3-L -10%
10%
20%
30%
40%
50%
Delay(%)
Default
NV (w=0) NV (w=0.25) NV (w=0.5) NV (w=0.75) NV (w=1) SysScale
(a) Delay (the lower the better)
MobileNet V3-S MobileNet V3-L
-30%
-20%
-10%
10%
Energy(%)
Default
NV (w=0) NV (w=0.25) NV (w=0.5) NV (w=0.75) NV (w=1) SysScale
(b) Energy consumption (the lower the better)
MobileNet V3-S MobileNet V3-L
-20%
-10%
10%
20%
EDP(%)
Default
NV (w=0) NV (w=0.25) NV (w=0.5) NV (w=0.75) NV (w=1) SysScale
(c) EDP (notwEDP for comparison) (the lower the better)
Figure 13: Delay, energy consumption, and EDP comparison of NeuroValve (NV(w)) and SysScale over the Android default in two CNN models on Pixel 3a.
0 20 40 60 80 100
NV (w=0) NV (w=0.5) NV (w=1) SysScale Default
Delay (ms)
Inference Others
(a) Average delay
0.0 0.5 1.0 1.5 2.0 2.5
AvgPower(W) NV (w=0)
NV (w=0.5) NV (w=1) SysScale Default
(b) Average power consumption
-25 -20 -15 -10 -5
EDP(%)
Default NV (w=0)
NV (w=0.5) NV (w=1) SysScale
(c) EDP improvement
Figure 14: Average delay and power consumption per frame in the object detection application and EDP improvement from NeuroValve andSysScalecompared to the Android default.
VII Discussion
CNN inference on mobile GPU. Mobile GPU still has limited capacity in its cache memory (e.g., 1MB for Adreno 630 GPU). Also, running inference on mobile GPU requires data memory copy from CPU to GPU, possibly leading to under-utilization of mobile GPU. As a result, main memory access can be the bottleneck for running inferences on mobile GPU. Since the existing governor for GPU clock frequency scaling is independent of memory access [20], understanding MAR can improve energy efficiency in running CNN over mobile GPU. In this sense, our CNN architecture-aware scaling approach would be beneficial to mobile GPU as well.
Multi-DNN. There may exist a particular need to run multiple DNN models simultaneously in an application. For instance, self-driving should continuously detect obstacles while deciding the direction or speed of the car. In this circumstance, focusing only on one model can degrade the overall energy efficiency. For instance, if one model runs a computation-intensive block and another model tries to run a memory-intensive block, it is challenging to decide. This difficulty can be alleviated when adopting the idea of per-core assignment of each DNN, in which NeuroValve is applied to each core differently.
How to jointly control them is left for future research.
Extension to other DNN models.We focus our analysis on CNN models. Another prevalent task carried out on mobile devices is sequential processing with recurrent neural network (RNN) models such as speech recognition [18, 19] and activity recognition from sensor data [59, 50]. Such recurrent models also have specific parameters that can affect computation and memory access costs. Therefore, we expect their configurations to exhibit a unique energy-delay trade-off in the frequency scaling, which is also left for future research.
VIII Related Work
Energy efficiency in mobile CPU. Recent research works [11, 38] found that the memory-intensive tasks and CPU-intensive tasks show different energy-performance trade-off relations. They also found metrics determining the memory-intensity of a task and proposed to utilize the metrics to improve energy efficiency. Su et al. [52] proposed a framework that predicts power, performance, and energy across all DVFS states by building a performance model and a per-core power model. Rao et al. [45] observed the energy inefficiency of existing general-pur
pose governors in various Android applications, and built an online controller to run applications energy- efficiently while meeting user-specified performance requirement. However, none of these works has studied or analyzed the processing of neural networks.
Coordinated DVFS.Using DVFS to optimize memory access for energy efficiency has been explored in [13] where the authors studied dynamic scaling of frequencies and voltages of the memory controller, memory channels, and DRAM devices. Deng et al. [12] made a similar proposal by pointing out the necessity to coordinate DVFS controllers of CPU and memory system. Extending both works, SysS- cale [21] optimizes multiple system components simultaneously by using DVFS to balance and relocate the preserved power according to each subsystem’s performance demands.
DVFS in deep learning. Deep learning workloads pose novel optimization challenges coming from their unique computational characteristics [1, 10]. Among others, recent works [3, 4] have investigated the use of DVFS on deep learning systems. PredJoule [4] explored specific performance and energy characteristics of different layers in a DNN and identified the best DVFS configuration for each layer using an embedded platform. NeuOS [3] attempted to simultaneously optimize multiple DNN workloads by balancing energy at the system level via DVFS and accuracy at the application level via configuration adjustments of DNNs. However, to the best of our knowledge, NeuroValve is the first to enable frequency scaling directly from analyzing a deep neural network’s block configuration parameters.
IX Conclusion
In this work, we present NeuroValve, a novel CPU frequency and memory bandwidth control framework that understands CNN blocks and exploits this understanding to improve the energy efficiency of CNN inferences. We investigated the impact of various CNN block configurations on energy consumption and delay performance. We found that adaptive CPU frequency scaling and memory bandwidth control per CNN block can significantly improve energy efficiency. We evaluated the efficacy of Neurovalve with various state-of-the-art CNN models and a deep learning application via implementation on the off-the- shelf smartphones. Our evaluations confirmed that NeuroValve could save energy consumption by up to 30.3% compared to the Android default setting and up to 12.8% compared an advanced memory-aware DVFS technique,SysScale.
References
[1] ADOLF, R., RAMA, S., REAGEN, B., WEI, G.,ANDBROOKS, D. Fathom: reference workloads for modern deep learning methods. In Proceedings of 2016 IEEE International Symposium on Workload Characterization (IISWC)(2016), pp. 1–10.
[2] BADRINARAYANAN, V., KENDALL, A., AND CIPOLLA, R. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence(2017).
[3] BATENI, S., AND LIU, C. NeuOS: A Latency-Predictable Multi-Dimensional Optimization Framework for DNN-driven Autonomous Systems. InProceedings of 2020 USENIX Annual Tech- nical Conference (ATC 20)(2020), pp. 371–385.
[4] BATENI, S., ZHOU, H., ZHU, Y., AND LIU, C. Predjoule: A timing-predictable energy opti- mization framework for deep neural networks. InProceedings of 2018 IEEE Real-Time Systems Symposium (RTSS)(2018), IEEE, pp. 107–118.
[5] BIANCO, S., CADENE, R., CELONA, L., ANDNAPOLETANO, P. Benchmark analysis of repre- sentative deep neural network architectures. IEEE Access 6(2018), 64270–64277.
[6] BRODOWSKI, D. Linux CPUFreq User Guide. https://www.kernel.org/doc/Documentation/
cpu-freq/user-guide.txt.
[7] BRODOWSKI, D., GOLDE, N., WYSOCKI, R. J., ANDKUMAR, V. Linux CPUFreq Governors.
https://www.kernel.org/doc/Documentation/cpu-freq/governors.txt.
[8] BROWN, N. Improvements in CPU frequency management. https://lwn.net/Articles/
682391/.
[9] CHEN, L.-C., PAPANDREOU, G., KOKKINOS, I., MURPHY, K., ANDYUILLE, A. L. Deeplab:
Semantic image segmentation with deep convolutional nets, atrous convolution, and fully con- nected crfs. IEEE transactions on pattern analysis and machine intelligence(2017).
[10] CHEN, T., MOREAU, T., JIANG, Z., ZHENG, L., YAN, E., SHEN, H., COWAN, M., WANG, L., HU, Y., CEZE, L., GUESTRIN, C.,ANDKRISHNAMURTHY, A. TVM: An automated end-to-end optimizing compiler for deep learning. InProceedings of 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18)(2018), pp. 578–594.
[11] CHOI, K., SOMA, R.,ANDPEDRAM, M. Dynamic voltage and frequency scaling based on work- load decomposition. In Proceedings of International symposium on Low power electronics and design(2004).
[12] DENG, Q., MEISNER, D., BHATTACHARJEE, A., WENISCH, T. F.,ANDBIANCHINI, R. Coscale:
Coordinating cpu and memory system dvfs in server systems. In 2012 45th annual IEEE/ACM international symposium on microarchitecture(2012), IEEE, pp. 143–154.
[13] DENG, Q., MEISNER, D., RAMOS, L., WENISCH, T. F.,ANDBIANCHINI, R. Memscale: active low-power modes for main memory. ACM SIGPLAN Notices 46, 3 (2011), 225–238.
[14] FRUMUSANU, A. The Apple iPhone 11, 11 Pro 11 Pro Max Review: Per- formance, Battery, Camera Elevated. https://www.anandtech.com/show/14892/
the-apple-iphone-11-pro-and-max-review/3, 2019.
[15] GOOGLE. Tensorflow Lite. https://www.tensorflow.org/lite/, 2019.
[16] GOOGLE. Tensorflow Lite object detection. https://www.tensorflow.org/lite/examples, 2019.
[17] GOOGLE. TensorFlow 1 Detection Model Zoo.https://github.com/tensorflow/models/blob/
master/research/object_detection/g3doc/tf1_detection_zoo.md, 2020.
[18] GRAVES, A.,ANDJAITLY, N. Towards end-to-end speech recognition with recurrent neural net- works. InProceedings of International conference on machine learning (ICML)(2014).
[19] GRAVES, A., MOHAMED, A.-R.,ANDHINTON, G. Speech recognition with deep recurrent neural networks. InProceedings of 2013 IEEE international conference on acoustics, speech and signal processing(2013).
[20] GUPTA, A. All about CPU Governors and GPU governors. http://ajgupta.github.io/
android/2015/01/28/CPU-and-GPU-governors/.
[21] HAJ-YAHYA, J., ALSER, M., KIM, J., YAGLIKÇI, A. G., VIJAYKUMAR, N., ROTEM, E., AND
MUTLU, O. SysScale: Exploiting Multi-domain Dynamic Voltage and Frequency Scaling for Energy Efficient Mobile Processors. InProceedings of the ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA)(2020).
[22] HAN, S., LIU, X., MAO, H., PU, J., PEDRAM, A., HOROWITZ, M. A., AND DALLY, W. J.
Eie: efficient inference engine on compressed deep neural network. ACM SIGARCH Computer
[23] HAN, S., MAO, H., ANDDALLY, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149(2015).
[24] HE, K., GKIOXARI, G., DOLLÁR, P.,AND GIRSHICK, R. Mask r-cnn. InProceedings of IEEE International Conference on Computer Vision (ICCV)(2017).
[25] HE, K., ZHANG, X., REN, S., AND SUN, J. Deep residual learning for image recognition. In Proceedings of IEEE conference on computer vision and pattern recognition(2016), pp. 770–778.
[26] HE, Y., ZHANG, X., ANDSUN, J. Channel pruning for accelerating very deep neural networks.
InProceedings of IEEE International Conference on Computer Vision (ICCV)(2017).
[27] HOWARD, A., SANDLER, M., CHU, G., CHEN, L.-C., CHEN, B., TAN, M., WANG, W., ZHU, Y., PANG, R., VASUDEVAN, V., LE, Q. V., AND ADAM, H. Searching for mobilenetv3. In Proceedings of IEEE International Conference on Computer Vision (ICCV)(2019).
[28] HOWARD, A. G., ZHU, M., CHEN, B., KALENICHENKO, D., WANG, W., WEYAND, T., AN-
DREETTO, M., AND ADAM, H. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861(2017).
[29] HU, J., SHEN, L., AND SUN, G. Squeeze-and-excitation networks. In Proceedings of IEEE conference on computer vision and pattern recognition(2018), pp. 7132–7141.
[30] IANDOLA, F. N., HAN, S., MOSKEWICZ, M. W., ASHRAF, K., DALLY, W. J., ANDKEUTZER, K. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360(2016).
[31] IGNATOV, A., TIMOFTE, R., CHOU, W., WANG, K., WU, M., HARTLEY, T.,ANDVANGOOL, L. Ai benchmark: Running deep neural networks on android smartphones. InProceedings of the European Conference on Computer Vision (ECCV) Workshops(September 2018).
[32] JACOB, B., KLIGYS, S., CHEN, B., ZHU, M., TANG, M., HOWARD, A., ADAM, H., AND
KALENICHENKO, D. Quantization and training of neural networks for efficient integer-arithmetic- only inference. InProceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)(2018).
[33] JOUPPI, N. P., YOUNG, C., PATIL, N., PATTERSON, D., AGRAWAL, G., BAJWA, R., BATES, S., BHATIA, S., BODEN, N., BORCHERS, A., ET AL. In-datacenter performance analysis of a tensor processing unit. InProceedings of the 44th Annual International Symposium on Computer Architecture(2017), pp. 1–12.
[34] KANG, Y., HAUSWALD, J., GAO, C., ROVINSKI, A., MUDGE, T., MARS, J., AND TANG, L.
Neurosurgeon: Collaborative Intelligence Between the Cloud and Mobile Edge. In Proceedings of the Twenty-Second International Conference on Architectural Support for Programming Lan- guages and Operating Systems (ASPLOS)(2017).
[35] KRAFTON. PUBG MOBILE. https://www.pubgmobile.com/en-US/.
[36] LASKARIDIS, S., VENIERIS, S. I., ALMEIDA, M., LEONTIADIS, I., ANDLANE, N. D. Spinn:
Synergistic progressive inference of neural networks over device and cloud. InProceedings of the 26th Annual International Conference on Mobile Computing and Networking(2020).
[37] LI, H., KADAV, A., DURDANOVIC, I., SAMET, H.,ANDGRAF, H. P. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710(2016).
[38] LIANG, W.-Y.,ANDLAI, P.-T. Design and implementation of a critical speed-based dvfs mecha- nism for the android operating system. InProceedings of 5th International Conference on Embed- ded and Multimedia Computing(2010).
[39] LIU, W., ANGUELOV, D., ERHAN, D., SZEGEDY, C., REED, S., FU, C.-Y.,ANDBERG, A. C.
Ssd: Single shot multibox detector. InProceedings of European conference on computer vision (2016), Springer, pp. 21–37.
[40] LONG, J., SHELHAMER, E.,AND DARRELL, T. Fully convolutional networks for semantic seg- mentation. InProceedings of IEEE conference on computer vision and pattern recognition (CVPR) (2015).
[41] MONSOON-SOLUTIONS. High Voltage Power Monitor. http://www.msoon.com/LabEquipment/
PowerMonitor/, 2019.
[42] PARASHAR, A., RHU, M., MUKKARA, A., PUGLIELLI, A., VENKATESAN, R., KHAILANY, B., EMER, J., KECKLER, S. W., AND DALLY, W. J. Scnn: An accelerator for compressed-sparse convolutional neural networks. ACM SIGARCH Computer Architecture News 45, 2 (2017), 27–40.
[43] QUALCOMM. Snapdragon 845 Mobile Platform. https://www.qualcomm.com/products/
snapdragon-845-mobile-platform, 2020.
[44] QUALCOMM. Snapdragon 855 Mobile Platform. https://www.qualcomm.com/products/
snapdragon-855-mobile-platform, 2020.
[45] RAO, K., WANG, J., YALAMANCHILI, S., WARDI, Y., ANDHANDONG, Y. Application-specific performance-aware energy optimization on android mobile devices. InProceedings of IEEE Inter- national Symposium on High Performance Computer Architecture (HPCA)(2017).
[46] REDMON, J., DIVVALA, S., GIRSHICK, R., AND FARHADI, A. You only look once: Unified, real-time object detection. In Proceedings of IEEE conference on computer vision and pattern recognition (CVPR)(2016).
[47] REN, S., HE, K., GIRSHICK, R.,AND SUN, J. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proceedings of Advances in neural information processing systems(2015).
[48] SANDLER, M., HOWARD, A., ZHU, M., ZHMOGINOV, A., AND CHEN, L.-C. Mobilenetv2:
Inverted residuals and linear bottlenecks. InProceedings of IEEE conference on computer vision and pattern recognition (CVPR)(2018).
[49] SIAMASHKA, S. TinyMemBench. https://github.com/ssvb/tinymembench.
[50] SINGH, D., MERDIVAN, E., PSYCHOULA, I., KROPF, J., HANKE, S., GEIST, M., AND
HOLZINGER, A. Human activity recognition using recurrent neural networks. InProceedings of International Cross-Domain Conference for Machine Learning and Knowledge Extraction(2017).
[51] SPILIOPOULOS, V., KAXIRAS, S., AND KERAMIDAS, G. Green governors: A framework for continuously adaptive dvfs. In Proceedings of International Green Computing Conference and Workshops(2011), IEEE, pp. 1–8.
[52] SU, B., GU, J., SHEN, L., HUANG, W., GREATHOUSE, J. L.,ANDWANG, Z. PPEP: Online per- formance, power, and energy prediction framework and DVFS space exploration. InProceedings of the 47th Annual IEEE/ACM International Symposium on Microarchitecture(2014), pp. 445–457.
[53] SZEGEDY, C., LIU, W., JIA, Y., SERMANET, P., REED, S., ANGUELOV, D., ERHAN, D., VAN-
HOUCKE, V., AND RABINOVICH, A. Going deeper with convolutions. InProceedings of IEEE conference on computer vision and pattern recognition (CVPR)(2015).
[54] SZEGEDY, C., VANHOUCKE, V., IOFFE, S., SHLENS, J.,ANDWOJNA, Z. Rethinking the incep- tion architecture for computer vision. InProceedings of IEEE conference on computer vision and pattern recognition (CVPR)(2016).