7.6 Experimental Results
7.6.2 Comparative Study Against Alternative Algorithms
Following the success against a random classifier in the preliminary study, we conduct a comparative study against the alternative algorithms, namely TGS, TGS+, ARTIVA, TVDBN-0, TVDBN-bino-hard and TVDBN-bino-soft.
Naming Conventions: ForTGS, TGS+, TGS-Lite and TGS-Lite+, some exten- sions are used with their names to imply different settings of these algorithms. If no extensions are used, then it implies a serial execution of that algorithm withMf set to 14. On the other hand, if an extension of ‘mf[X]’ is used, then it implies thatMf is set to ‘X’. For example, TGS.mf24 implies a serial execution of TGS with Mf set to 24.
Lastly, an extension of ‘p[X]’ implies a multicore-parallelised execution with the number of computing cores set to ‘X’. An example would be TGS-Lite.p10, which implies a parallel execution of TGS-Lite with 10 cores.
Lighter Memory Footprint: To compare memory footprints ofTGS and TGS- Lite, an experiment is performed with dataset Ds100n for Mf = 24. From Figure 7.5 ,
it is observed that TGS occupies 54.7% (column ‘%MEM’) of the memory (∼32 GB) within first 1 min 35.25 seconds (column ‘TIME+’) of execution. Its memory require- ment rises further as the time goes on until it reaches a ‘thrashing’ state (Silberschatz et al., 1991). At that point, we terminate the process. On the other hand, TGS-Lite occupies only 0.7% of the memory during the same time point. Moreover, it completes execution without any issues. This experiment demonstrates the advantage of TGS- Lite’s memory-efficiency over that of TGS. Additionally, it opens up the possibility of parallel execution withTGS-Lite. Since the serial execution of TGS-Lite only requires 0.7% of the memory, we can execute it in parallel over at most b(100/0.7)c '142 cores who share the same memory. The parallelised execution is expected to reduce the run- time by at most a factor of 142. TGS, on the other hand, can not take advantage of multicore-parallelisation schemes, since its serial execution alone has such a heavy memory footprint.
Percentage of Memory Usage with dataset Ds100n
TGS.mf24 TGS Lite.mf24
Figure 7.5: Comparative Memory Requirements of theTGS and TGS-Lite Algorithms.
The percentage of memory usage (‘%MEM’) byTGS and TGS-Lite when the max fan- in parameter is set to 24. The shown figure is a screenshot of the ‘top’ command in the Ubuntu OS.
No Loss in Correctness: The memory-efficiency of TGS-Lite does not come at the cost of its correctness. It provides the same recall, precision and F1-scores as that of TGS (Figure 7.6) 2. TGS-Lite+ also meets its expectation by achieving the same recall, precision and F1-scores as that ofTGS+ (Figure 7.6).
Multicore Parallelisation: Although,TGS-Lite shares the same time complexity as that ofTGS, it takes longer runtime than that of TGS (Table 7.1). Same is true for TGS-Lite+, which takes longer runtime than that ofTGS+ even though they have the same time complexity. The potential cause lies in the difference between their imple- mentations. The strategy which replaces the Bene step inTGS-Lite and TGS-Lite+ is implemented inRprogramming language, except the function for BIC score calculations which is implemented in C programming language. On the other hand, the Bene step in TGS and TGS+ is implemented in C, which is expected to be significantly faster than the R implementation of the replacement strategy. Hence, a C implementation of the replacement strategy is planned for future.
2Please note that algorithmsTVDBN-exp-hard andTVDBN-exp-softresult in error for all datasets.
B
Ds10n Ds50n Ds100n
0231 0231
075 075
0 0 0125 0182
0042 0042 0057 0057 0086 0034 0026 00340034 0034 0095 0095 0081 0013 002 0014
TGS-Lite
TGS
TGS-Lite
TGS AR
TIV A
TVDBN-bino-hardTVDBN-bino-soft
Precision
TGS-Lite
TGS
TGS-Lite
TGS AR
TIV A
TVDBN-0
TVDBN-bino-hardTVDBN-bino-soft
03 03 03 03 0 0 01
0195 0195 0078 0078 0078 0091 0143 02
0169 0169 0114 0114 0084 0054 0157 0180108
Ds10n Ds50n Ds100n
Recall
TGS-Lite
TGS
TGS-Lite
TGS AR
TIV A
TVDBN-0
TVDBN-bino-hardTVDBN-bino-soft
03 03 03 03 0 0 01
0195 0195 0078 0078 0078 0091 0143 02
0169 0169 0114 0114 0084 0054 0157 0180108
Ds10n Ds50n Ds100n
TGS-Lite
TGS
TGS-Lite
TGS AR
TIV A
TVDBN-0
TVDBN-bino-hardTVDBN-bino-soft
C
Ds10n Ds50n Ds100n
TGS-Lite
TGS
TGS-Lite
TGS AR
TIV A
TVDBN-0
TVDBN-bino-hardTVDBN-bino-soft
F1score
0261 02610069 0057
0429 0429
0 0 0111 019
0069 0066 0066 0082 0049 0044 00580057 0104 0083 0021 0035 0024
0104
Figure 7.6: Comparative Performance of the Selected Algorithms on the Benchmark Datasets. A) Recall, B) precision and C) F1-scores of the selected algorithms are shown.
At this time, we take advantage of the memory-efficiency of TGS-Lite and TGS- Lite+. We do that by executing them in parallel over 10 cores. As expected, the runtime reduce drastically compared to that of the serial executions (Table 7.1)3 .
Table 7.1: Runtime of the Selected Algorithms on the Benchmark Datasets.
Algorithm Ds10n Ds50n Ds100n
TGS-Lite 8.9s 3h 58m 2s 8h 41m 17s
TGS-Lite.p10 6.673s 24m 45s 1h 2m 3s
TGS 5.789s 7m 36s 17m 49s
TGS-Lite+ 2.986s 17.383s 12m 12s
TGS-Lite+.p10 6.028s 9.025s 1m 25s
TGS+ 5.515s 22.034s 1m 4s
ARTIVA 10m 20s 4h 30m 15s 31h 52m 54s
TVDBN-0 2m 24s 11m 59s 52m 17s
TVDBN-bino-hard 2m 15.2s 9m 38s 2h 53m 32s TVDBN-bino-soft 2m 14.6s 8m 8s 17m 20s
Effects of Multicore Parallelisation: In this paragraph, we study the effects of multicore parallelisation on the runtime ofTGS-LiteandTGS-Lite+. We use Ds100n for this study as this is the largest benchmark dataset. From Figure 7.7 , it is observed that the runtime reduces strictly as the number of cores increases. The single core represents the serial execution. The 3-core executions are presented to offer a reference point to the users with quad-core processors, where one core can be left out for monitoring purposes.
Similarly, the 7-core executions are presented to offer a reference point to the users with octa-core processors.
B
2m 2s
4m 8s
m 53s
m 25s
Number of Cores
Runtimesec)
Number of Cores
TGS-Lite .mf14 TGS-Lite.mf14
Runtimesec)
8h 4m 7s
3h 2m 6s
h 9m 7s
h 2m 3s
Figure 7.7: The Effects of the Multicore Parallelisation on the Runtime of TGS-Lite and TGS-Lite+. Dataset Ds100n is used and the max fan-in parameter is set to 14.
Effects of Max Fan-in: Finally, we conclude this section by studying the effects of the max fan-in parameter on the correctness and runtime of the proposed algorithms.
The maximum value of the concerned parameter with which TGS and TGS+ are suc- cessfully demonstrated is 14. Since TGS-Lite and TGS-Lite+ are significantly more memory-efficient than the aforementioned algorithms, they should be able to run suc- cessfully with max fan-in values higher than 14. To verify that, we need to vary the
3Please note that algorithmsTVDBN-exp-hard andTVDBN-exp-softresult in error for all datasets.
parameter value from 14 to higher values. These variation can have an effect on the prediction only when at least one gene has more than 14 candidate regulators and the max fan-in restriction is not in place. We find that only dataset Ds100n fulfils this criterion. For this dataset, at least one gene has more than 14 candidate regulators in case of both TGS-Lite and TGS-Lite+ (Table 7.2). Therefore, we study the effects on these algorithms with Ds100n when the parameter value is varied from 14 to 18 (eigh- teen being the minimum of the maximum numbers of candidate regulators forTGS-Lite and TGS-Lite+). It is observed that the recall of TGS-Lite increases monotonically as the max fan-in increases; at the same time, its precision decreases monotonically;
as a result, an upheaval is caused in the F1-scores (Figure 7.8 A). On the other hand, TGS-Lite+ demonstrates a robust performance as its recall, precision and F1-score re- main unchanged for the given range of max fan-in values (Figure 7.8 B). The runtime of both the algorithms increase proportionally to the max fan-in values (Figures 7.8 C and 7.8 D). It is as expected since the higher the max fan-in value, the higher the maximum number of candidate regulators for a gene; therefore, the higher the runtime required for selecting the final set of regulators.
Table 7.2: Maximum Number of Candidate Regulators of a Gene in the TGS-Lite and TGS-Lite+ Algorithms for a given Dataset. These statistics are recorded when the max fan-in restriction is not applied. If the max fan-in restriction is applied, the numbers will be upper bounded by the value assigned to the max fan-in parameter.
Dataset TGS-Lite TGS-LIte+
Ds10n 7 4
Ds50n 33 8
Ds100n 84 18