CHAPTER 3: SYSTEM METHODOLOGY

It is hereby certified that ______Desmond Cheong Yonghong_______ (ID No: __18ACB04824 ) completed this senior year project titled "___Stock Indicator Scanner Customization Tool Using Deep Reinforcement Learning _" under the supervision of ___ Ts Dr. Ku Chin Soon ___. Promoter) of the Department of ____ Informatics ___, Faculty of ___ Information and Communication Technology. I certify that this report entitled "STOCK INDICATOR SCANNER CUSTOMIZATION TOOL USING DEEP REINFORCEMENT LEARNING" is my own work, except as noted in the references. Aside from that, many current stock indicator scanners only allow the user to specify some simple conditions to scan the stock and do not take advantage of machine learning advancements.

The stock indicators selected by the user will be the input of the DQN algorithm and will act as states while the allowed actions for the DQN agent will be buy and sell. For simplicity, return on investment (ROI) will be used as the reward of the RL agent. Moreover, the ADAM optimization technique will be applied to adjust the network parameters in DQN and the ReLU activation function will be used since these techniques have shown promising achievements in several literature reviews.

The mean square error (MSE) loss function used in classical DQN will be replaced by Huber loss to improve the stability of the model training.

INTRODUCTION

Project Inspiration
Problem Statements
Project Objectives
Project Scope
Project Impact and Contribution
Chapter Summary

The purpose of using an indicator instead of raw market data is to reduce model training costs without compromising model performance. The user will be able to customize the input of the predictive model on the stock. The trend of the stock (buy or sell) will be the result of the model.

The results of the model evaluation enable the user to know and compare the performance of the created models. To limit the scope of the project, the model will only focus on the top 100 stocks in Malaysia. The user can select and create the indicators they prefer to use for training the model.

Investor can also use this system to examine the performance of the new indicator with which they come out with real money in stock market.

LITERATURE REVIEW

Overview of Malaysia Stock Market
Fundamental Analysis

Earnings Per Share (EPS)
Price-to-Earnings (P/E) Ratio
Price-to-Book (P/B) Ratio
Price/Earnings-to-Growth (PEG) Ratio

Technical Analysis

Moving Averages (MA)
Moving Average Convergence Divergence (MACD)
Relative Strength Index (RSI)
On-Balance Volume (OBV)

Overview of Reinforcement Learning: Q-Learning
Overview of Deep Reinforcement Learning: Deep Q-Learning
Reinforcement Learning Approach
Deep Reinforcement Learning Approach

This "cheat sheet" allows the agent to take the best action for each state and generate maximum total rewards. 11 environment by performing any action or "exploiting" the Q table by performing the action with the highest Q value for the given state st. As shown in Figure 2.5.1, in the proposed framework, the input data is pre-processed using wavelet transforms (WT) to eliminate noise, then stacked auto-encoders (SAEs) are used for feature extraction, and finally the closing price of stocks is predicted using Long Short-Term Memory (LSTM).

In the proposed framework, historical tick data containing the largest intraday data is used as input to the prediction model. The Gated Recurrent Unit (GRU) is used in the model to derive the financial characteristics of the stock. In the proposed method, agent state information of RL and open-high-low-close volume (OHLCV) are used as information.

Return on investment (ROI), which divides the dollar return by the dollar invested, is used as the reward of RL agent in the proposed multi-DQN.

Figure 2.5 Comparison between Q-Learning and Deep Q-Learning. [19]

SYSTEM METHODOLOGY

Project Development
Information Gathering
System Functionalities for Stock Indicator Scanner Customization Tool

Design of stock trend prediction procedure
Design of DQN
Design of DQN network structure

Hardware and Software Requirements

The characteristics, strengths and weaknesses of the tools and technologies will be studied to ensure they are suitable for this project. During the execution phase, programming is done to realize all designs in the design phase. On the other hand, the OHLCV of the stock will also be used in the visualization of the performance of the predictive model by plotting a graph.

As new stock market data becomes available, the new data will be used to update the forecast model and ensure forecast accuracy. The selected model will be used in the forecast and the investor can view the forecast result. For simplicity, the return on investment (ROI) will be used as the DQN agent reward.

Information generated after each step (st, at, rt, st+1) will be added to replay memory. For training the network, a minibatch of random samples of size 64 will be retrieved from replay memory. Prepare the Q values of the current state (current Qs list) will be retrieved by predicting the Q network, while the Q values of the new state (future Qs list) will be retrieved by predicting using the Target Q Network.

To predict using the model, the inventory data is loaded and used to initialize the environment. 17], therefore LSTM will be used in this project to replace the feedforward DNN used in classic DQN. At the dense layer of the network, the ReLU activation function will be used, as [24] and [27] have demonstrated the reliability of this activation function.

Since Théate & Ernst [27] suggested that using Huber loss instead of mean square error (MSE) loss in classical DQN could improve model training stability, therefore Huber loss will be used in the network as a loss function. Visual Studio Code will be used as the IDE in developing the system.

Table 3.2 List of indicators provide to user

SYSTEM DESIGN

Graphical User Interface Design

At the bottom there are 2 tabs, that is the "Select indicators" tab and the "Select model" tab. The user can go to the "Select Model" tab and select the model they created earlier to start making predictions. Other than that, the user can also directly choose the indicators that he wants to use to make the prediction without creating a model beforehand.

To view other shares, the user can use the pagination feature to navigate to another page. As shown in Figure 4.1.3, for forecasting, the user must first select a watchlist consisting of stocks of interest to the user. Next, the user has to select the indicators or select the model that the user would like to use in the forecast.

After a while, the prediction will be ready, the stock trend (strong buy, buy, hold, sell, strong sell) for each stock will be displayed in the "Recommendation" column of the table, and the accuracy of the prediction will be displayed in the "Accuracy" column. On the "Model" page, the models that the user has previously created will be displayed as illustrated in figure 4.1.4. 35 will occur for the user to enter a name for the model and select the indicators used in the model.

Additionally, for models that are not trained using the latest stock data (not updated), the user can click the "Start Training" button to train (update) the model using the latest stock data. As shown in Figure 4.1.6, a modal appears and the user can change the model name and the indicator that is composed of the model. The user can also remove the model by clicking the cross button next to the model. A modal will be displayed for the user to enter the watchlist name and select the stocks to add to the watchlist.

After the user selects a watchlist, selects indicators or model and clicks "Get Performance Report" button, a report will be generated for each stock. On the left side of the generated report, some graphs like cumulative returns, EOY return, monthly returns, daily return rolling volatility, etc. will be displayed.

Data Storage Design

System Architecture Design

SYSTEM TESTING

Project Verification Plans

Test Results

Strongly Sell” on symbol 1155.KL (Malayan Banking Bhd) which is the same as the Investing.com prediction made based on technical indicators as shown in Figure 5.2.3. For the symbol 5099.KL (Capital A Bhd), the proposed model recommended "Hold" while Investing.com suggested "Sell" as shown in Figure 5.2.1 and Figure 5.2.4. Although the prediction result for symbol 5099.KL is different between system 2, however, the difference between the result of both predictions is not big.

In Figure 5.2.4, we can see that 5 indicators suggest "Sell" and 3 indicators suggest "Buy". If you change the indicator proposal from "Sell" to "Buy", the number of both "Sell" and "Buy" will be the same and the final proposal will be the same. The forecast result for symbol 5099.KL differs between the proposed model and Investing.com, perhaps because Investing.com considers the bull/bear strength indicator, while the proposed model does not include this indicator, as shown in Figure 5.2.2.

Investing.com suggests the trading action by following the forecast made by majority indicators, while the proposed model uses DQN which considers all selected indicators as one. This may indicate that more training is needed for the model to achieve higher accuracy and better prediction.

Table 5.2.2 Performance metrics of strategies on symbol 5099.KL

DISCUSSION

System Limitation

Future Enhancement

53 Coverage for other financial instruments such as bonds, futures, and options is also an update that can be implemented in the future. Furthermore, the accuracy of predictions can be increased by adding support for unstructured data such as news and social media as input.

CONCLUSION

Rao, “A Deep Learning Framework for Financial Time Series Using Stacked Autoencoders and Long Short-Term Memory,” PLOS ONE , vol. Dai, “Deep Direct Reinforcement Learning for Financial Signal Representation and Trading,” IEEE Transactions on Neural Networks and Learning Systems, vol. 57 Approach for Stock Index Future’s Intraday Trading,” 10th International Symposium on Computational Intelligence and Design (ISCID) 2017, Dec.

Ernst, "An application of deep reinforcement learning to algorithmic trading," Expert Systems with Applications, vol. An Ensemble of Q Deep Learning Agents for Stock Market Forecasting," Expert Systems with Applications, Vol. Student Name and ID: DESMOND CHEONG YONGHONG 18ACB04824 Supervisor: TS DR. Project Title: FOG PERSONALIZATION OF STOCK INDICATORS SCENARIO USING DEEP REINFORCEMENT LEARNING.

Changing the number of agent actions and trading positions causes the necessary retraining of the DQN model. Note: Supervisor/candidate(s) are required to provide a soft copy of the full set of the originality report to the faculty/department. Based on the above findings, I hereby declare that I am satisfied with the originality of the final project report submitted by my students as mentioned above.

Form Title: Supervisor's Comments on Originality Report Generated by Turnitin for Final Year Project Report Submission (for undergraduate programs).

CHECKLIST