neural solvers for differential equations - SGU Repository

(1)

APPLIED IN PHYSICAL SYSTEMS

By

Fabio Milentiansen Sim 11601062

BACHELOR’S DEGREE in

MECHANICAL ENGINEERING - MECHATRONICS CONCENTRATION FACULTY OF ENGINEERING AND INFORMATION TECHNOLOGY

SWISS GERMAN UNIVERSITY The Prominence Tower

Jalan Jalur Sutera Barat No. 15, Alam Sutera Tangerang, Banten 15143 - Indonesia

Revision after Thesis Defence on 07 July 2020 July 2020

(2)

STATEMENT BY THE AUTHOR

I hereby declare that this submission is my own work and to the best of my knowledge, it contains no material previously published or written by another person, nor material which to a substantial extent has been accepted for the award of any other degree or diploma at any educational institution, except where due acknowledgement is made in the thesis.

Fabio Milentiansen Sim

Student Date

Approved by:

Dr Eka Budiarto, S.T., M.Sc.

Thesis Advisor Date

Dr Rusman Rusyadi

Thesis Co-Advisor Date

Dr Maulahikmah Galinium, S.Kom, M.Sc

Dean Date

(3)

ABSTRACT

NEURAL SOLVERS FOR DIFFERENTIAL EQUATIONS APPLIED IN PHYSICAL SYSTEMS

By

Fabio Milentiansen Sim

Dr Eka Budiarto, S.T., M.Sc., Advisor Dr Rusman Rusyadi, Co-Advisor

SWISS GERMAN UNIVERSITY

Differential equations are ubiquitous in many fields of study, yet analytical solutions to such equations, whether ordinary or partial, have long eluded discovery. With the advent of modern deep learning, neural networks have become a viable alternative to numerical methods. By reformulating the problem as an optimisation task, neural networks can be trained to approximate nonlinear solutions. In this paper, neural solvers are implemented in TensorFlow for a variety of equations. Their overall performance is analysed and even found to surpass traditional schemes in certain cases. Experimental data is also used to validate the neural solutions. Stiff and nonlinear systems of equations are attempted as well, and the stability of the method is investigated. A normalisation technique, akin to feature scaling for supervised learning, is proposed and shown to achieve superior convergence. Although neural solvers will not replace the computational speed offered by traditional schemes in the near future, they remain a feasible, easy-to-implement substitute when all else fails.

Keywords: Differential Equations, Deep Learning, Neural Networks, Numerical Analysis, Computer Simulation

(4)

(5)

DEDICATION

For the pursuit of knowledge in not only mathematics, but also every other aspect of the universe.

(6)

ACKNOWLEDGEMENTS

I would like to extend my gratitude to my advisors, Dr Eka Budiarto, for guiding me through the intricacies of applied mathematics, and Dr Rusman Rusyadi, for igniting my passion for deep learning.

I thank my family and friends for supporting me throughout my academic journey, as well as in my personal life.

(7)

TABLE OF CONTENTS

STATEMENT BY THE AUTHOR 2

ABSTRACT 3

DEDICATION 5

ACKNOWLEDGEMENTS 6

TABLE OF CONTENTS 7

LIST OF FIGURES 10

LIST OF TABLES 13

CHAPTER 1 - INTRODUCTION 14

1.1 Background 14

1.2 Research Problems 15

1.3 Research Objectives 15

1.4 Significance of the Study 16

1.5 Research Questions 16

1.6 Hypotheses 16

CHAPTER 2 - LITERATURE REVIEW 17

CHAPTER 3 - RESEARCH METHODS 20

3.1 Theoretical Approach 20

3.1.1 Differential Equations Preliminaries 21

3.1.2 Neural Network Preliminaries 22

3.1.3 Combination of Neural Networks and ODEs 25

3.2 Implementation Considerations 26

3.3 Model Validity 30

3.4 Model Performance 30

3.5 Model Robustness 30

3.6 Timeline 31

(8)

CHAPTER 4 - RESULTS AND ANALYSIS 32

4.1 Environment Setup 32

4.2 Ordinary Differential Equations (ODEs) 33

4.2.1 First Order Semilinear ODE 33

4.2.1.1 Ansatz Variant 38

4.2.1.2 Convolutional Variant 39

4.2.1.3 Recurrent Variant 40

4.2.2 First Order Linear ODE 41

4.2.3 First Order Nonlinear ODE 44

4.2.4 Second Order Linear ODE 46

4.2.5 Second Order Quasilinear ODE 50

4.2.6 Second Order Nonlinear ODE 52

4.3 Partial Differential Equations (PDEs) 53

4.3.1 Poisson’s Equation for the Dirichlet Problem 53 4.3.2 Heat Equation for the Dirichlet Problem 59 4.3.3 Heat Equation for the Neumann Problem 62

4.3.4 Wave Equation 64

4.3.5 Inviscid Burgers’ Equation 66

4.4 Systems of Differential Equations and Additional Examples 70

4.4.1 Linear System of ODEs 72

4.4.2 Stiff System of ODEs 74

4.4.3 Nonlinear System of ODEs 78

4.4.4 Proportional (P) Controller Simulation 81

4.4.5 Proportional-Integral (PI) Controller Simulation 83

4.5 Experimental Validation 85

4.5.1 Spring-Mass System 85

4.5.2 Gauss’s Law - Laplace’s Equation 92

4.6 Robustness Test 96

CHAPTER 5 - CONCLUSION 97

GLOSSARY 99

REFERENCES 100

APPENDIX A - ADDITIONAL MATHEMATICAL PROOFS 103 A1 Normalisation of Ordinary Differential Equations for Neural Methods 103

(9)

A2 Normalisation of Partial Differential Equations for Neural Methods 108 APPENDIX B - FULL PROGRAM CODE EXAMPLES 110

B1 First Order Semilinear ODE Ansatz Program 110

B2 Poisson’s Equation Program 113

B3 Lotka-Volterra System Ansatz Program 117

APPENDIX C - SUPPLEMENTAL FIGURES AND TABLES 121

C1 Experimental Data for Spring-Mass System 121

CURRICULUM VITAE 122

(10)

LIST OF FIGURES

Figure 3.0.1: Research Methodology Block Diagram 20

Figure 3.1.1: Illustration of a deep neural network 22

Figure 3.2.1: Main Algorithm Flowchart 28

Figure 3.2.2: Training Callback Flowchart 29

Figure 4.2.1.1: Training Loss Curve 35

Figure 4.2.1.2: True Loss Curve 36

Figure 4.2.1.3: Solution and Error Plots 36

Figure 4.2.1.4: Error Trends 37

Figure 4.2.1.5: Ansatz True Loss Curve 38

Figure 4.2.1.6: CNN True Loss Curve 39

Figure 4.2.1.7: RNN True Loss Curve 40

Figure 4.2.1.8: RNN Solution and Error Plots 41

Figure 4.2.2.1: True Error Characteristics 42

Figure 4.2.2.2: Solution Plot 42

Figure 4.2.6.1: Error Characteristics and Solution Plot 52

Figure 4.3.1.1: Error Characteristics 56

Figure 4.3.1.2: Solution Plots 56

Figure 4.3.1.3: Neural Error 57

(11)

Figure 4.3.5.2: Solution Plot and Characteristic Lines 67

Figure 4.3.5.3: Solution Time Frames 67

Figure 4.3.5.4: Solution Plot and Characteristic Lines 68

Figure 4.3.5.5: Solution Time Frames 68

Figure 4.3.5.6: Solution Plot, Characteristic Lines, Time Frames 69

Figure 4.4.1.2: Solution Plots and Errors 73

Figure 4.4.2.1: Error Characteristics (No Scaling) 75 Figure 4.4.2.2: Error Characteristics (With Scaling) 75

Figure 4.4.2.4: Forward Euler Comparison 76

Figure 4.4.2.5: Ansatz Error Characteristics and Solution Plot 77 Figure 4.4.3.1: Error Characteristics, Half Domain (Left), Whole Domain

(Right) 79

Figure 4.4.3.2: Solution Plots & Errors, Half Domain (Top), Whole

Domain (Bottom) 79

Figure 4.4.3.3: Error Characteristics, Before Transformation (Left), After

Transformation (Right) 80

Figure 4.4.3.4: Solution Plots and Errors, Before Transformation (Top),

After Transformation (Bottom) 80

Figure 4.4.4.1: P-Controller Block Diagram 81

Figure 4.4.4.2: Solution Plot and Error 82

(12)

Figure 4.4.5.2: Solution Plot and Error 84 Figure 4.5.1.1: Position Sensor (Left), Solenoid (Right) 86

Figure 4.5.1.2: Sensor-CASSY 86

Figure 4.5.1.3: Experiment Setup 87

Figure 4.5.2.1: Loss Characteristics (Left), Error Contour (Right) 93 Figure 4.5.2.2: Experimental Data (Moradi and Marvasti, 2016) (Left),

Neural Method (Right) 93

Figure 4.5.2.3: Boundary Condition Network Output 94

Figure 4.5.2.4: Loss Characteristics (Left), Error Contour (Right) 95 Figure 4.5.2.5: Experimental Data (Moradi and Marvasti, 2016) (Left),

Neural Method Ansatz Variant (Right) 95

(13)

LIST OF TABLES

Table 3.6: Timeline 31

Table 4.6: Robustness Test Results 96

Table C1: Experimental Data for Spring-Mass System 121