Proof. By definition ofdkand Equation (15)
dk= −2F(xk) +F(xk) dk−1dk−1
≤2F(xk)+F(xk) dk−1dk−1
≤3F(xk)
≤3κ.
(20)
LettingM=3κ, we have the desired result.
Theorem 1. Suppose that assumptions (G1)–(G3) hold and let the sequence{xk}be generated by Algorithm1, then
lim inf
k→∞ F(xk)=0, (21)
Proof. To prove the Theorem, we consider two cases;
Case 1
a very small number. Therefore, we choseβ=0.7 andσ=0.0001 for the implementation of the proposed algorithm.
We test 9 different problems with dimensions ranging fromn=1000, 5000, 10, 000, 50, 000, 100, 000 and 6 initial points:x1= (0.1, 0.1,· · ·, 1)T,x2= (0.2, 0.2,· · ·, 0.2)T,x3= (0.5, 0.5,· · ·, 0.5)T,x4= (1.2, 1.2,· · ·, 1.2)T,x5= (1.5, 1.5,· · ·, 1.5)T,x6= (2, 2,· · ·, 2)T. In Tables1–9, the number of iterations (ITER), number of function evaluations (FVAL), CPU time in seconds (TIME) and the norm at the approximate solution (NORM) were reported. The symbol ‘−’ is used when the number of iterations exceeds 1000 and/or the number of function evaluations exceeds 2000.
The test problems are listed below, where the function F is taken as F(x) = (f1(x),f2(x), . . . ,fn(x))T.
Problem 1([26]). Exponential Function.
f1(x) =ex1−1,
fi(x) =exi+xi−1,for i=2, 3, ...,n, andΨ=Rn+.
Problem 2([26]). Modified Logarithmic Function.
fi(x) =ln(xi+1)−xi
n,for i=2, 3, ...,n, andΨ={x∈Rn:
∑
n i=1xi≤n,xi>−1,i=1, 2, . . . ,n}.
Problem 3([13]). Nonsmooth Function.
fi(x) =2xi−sin|xi|,i=1, 2, 3, ...,n, andΨ={x∈Rn:
∑
n i=1xi≤n,xi≥0,i=1, 2, . . . ,n}.
It is clear that Problem 3 is nonsmooth at x=0.
Problem 4([26]). Strictly Convex Function I.
fi(x) =exi−1,for i=1, 2, ...,n, andΨ=Rn+.
Problem 5([26]). Strictly Convex Function II.
fi(x) = i
nexi−1,for i=1, 2, ...,n, andΨ=Rn+.
Problem 6([27]). Tridiagonal Exponential Function f1(x) =x1−ecos(h(x1+x2)),
fi(x) =xi−ecos(h(xi−1+xi+xi+1)),for i=2, ...,n−1, fn(x) =xn−ecos(h(xn−1+xn)),
h= 1
n+1andΨ=Rn+.
Problem 7([28]). Nonsmooth Function
fi(x) =xi−sin|xi−1|,i=1, 2, 3, ...,n.
andΨ={x∈Rn:
∑
n i=1xi≤n,xi≥ −1,i=1, 2, . . . ,n}.
Problem 8([23]). Penalty 1
ti=
∑
ni=1
x2i, c=10−5
fi(x) =2c(xi−1) +4(ti−0.25)xi,i=1, 2, 3, ...,n.
andΨ=Rn+. Problem 9([29]). Semismooth Function
f1(x) =x1+x13−10, f2(x) =x2−x3+x32+1, f3(x) =x2+x3+2x33−3, f4(x) =2x34,
andΨ={x∈R4:
∑
4 i=1xi≤3,xi≥0,i=1, 2, 3, 4}.
In addition, we employ the performance profile developed in Reference [30] to obtain Figures1–3, which is a helpful process of standardizing the comparison of methods. The measure of the performance profile considered are; number of iterations, CPU time (in seconds) and number of function evaluations. Figure1reveals that Algorithm1most performs better in terms of number of iterations, as it solves and wins 90 percent of the problems with less number of iterations, while PCG and PDY solves and wins less than 10 percent. In Figure2, Algorithm1performed a little less by solving and winning over 80 percent of the problems with less CPU time as against PCG and PDY with similar performance of less than 10 percent of the problems considered. The translation of Figure3is identical to Figure1. Figure4is the plot of the decrease in residual norm against number of iterations on problem 9 withx4as initial point. It shows the speed of the convergence of each algorithm using the convergence tolerance 10−5, it can be observed that Algorithm1converges faster than PCG and PDY.
Table1.NumericalResultsforAlgorithm1(DCG),PCGandPDYforProblem1withgiveninitialpointsanddimensions. Algorithm1PCGPDY DIMENSIONINITIALPOINTITERFVALTIMENORMITERFVALTIMENORMITERFVALTIMENORM 1000
x111490.0255578.88×10−618730.0192955.72×10−612490.162489.18×10−6 x212530.0141644.78×10−618730.0116489.82×10−613530.037806.35×10−6 x312530.0085248.75×10−619770.0111977.1×10−614570.015505.59×10−6 x413570.0113336.68×10−618730.0221978.27×10−615610.017464.07×10−6 x513570.0142026.09×10−6632540.0460729.58×10−614570.021939.91×10−6 x613570.0110458.14×10−6612460.0316089.15×10−6401620.034729.70×10−6 5000
x112530.0243115.82×10−618730.114317.42×10−613530.031586.87×10−6 x213570.0273613.13×10−619770.039976.53×10−614570.042704.62×10−6 x313570.025415.73×10−620810.0561595.2×10−615610.054334.18×10−6 x414610.0320384.38×10−619770.0383818.1×10−615610.043579.08×10−6 x514610.0390443.98×10−6622500.158369.53×10−615610.089607.30×10−6 x614610.0272315.33×10−6602420.132769.1×10−6391580.112849.86×10−6 10,000
x112530.054348.23×10−618730.0732079.5×10−613530.063719.70×10−6 x213570.0456644.43×10−619770.0907718.15×10−614570.063366.53×10−6 x313570.0419228.09×10−620810.0708596.74×10−615610.064145.90×10−6 x414610.0476416.2×10−620810.0873575.11×10−616650.079204.28×10−6 x514610.0457345.62×10−6622500.246468.87×10−6391580.221017.97×10−6 x614610.0571047.54×10−6592380.199499.96×10−6873510.362379.93×10−6 50,000
x113570.163845.41×10−619770.254878.8×10−614570.276077.12×10−6 x213570.186339.9×10−620810.326897.39×10−615610.262204.91×10−6 x314610.208015.32×10−621850.336496.31×10−616650.282604.37×10−6 x415650.19464.08×10−621850.327795.1×10−6381540.606507.54×10−6 x515650.197993.69×10−6612460.826158.85×10−61777122.523309.44×10−6 x615650.224184.95×10−6592380.799928.5×10−636114495.979509.74×10−6 100,000 x113570.322917.65×10−620810.538465.52×10−615610.393423.39×10−6 x214610.333294.12×10−621850.615334.62×10−615610.421546.94×10−6 x314610.370487.52×10−621850.536388.78×10−616650.458516.18×10−6 x415650.360585.76×10−621850.620027.21×10−61757044.361009.47×10−6 x515650.349755.22×10−6602421.45649.73×10−61767084.291809.91×10−6 x615650.36217.01×10−6582341.41559.42×10−636014459.711909.99×10−6
Table2.NumericalResultsforAlgorithm1(DCG),PCGandPDYforProblem2withgiveninitialpointsanddimensions. Algorithm1PCGPDY DIMENSIONINITIALPOINTITERFVALTIMENORMITERFVALTIMENORMITERFVALTIMENORM 1000
x19383.17445.84×10−615590.0498998.59×10−610390.010536.96×10−6 x210420.0146336.25×10−611420.0150899.07×10−611430.009379.23×10−6 x39380.0170677.4×10−617660.0169356.44×10−613510.011116.26×10−6 x47300.0063926.53×10−618690.014366×10−614550.021549.46×10−6 x511460.0119543.47×10−613480.009077.58×10−615590.018504.60×10−6 x612500.686666.74×10−618680.013525.4×10−615590.019387.71×10−6 5000
x110420.112413.53×10−616630.0411519.35×10−611430.035284.86×10−6 x211460.0287233.81×10−612460.0287068.8×10−612470.040326.89×10−6 x310420.0293674.3×10−618700.0475326.98×10−614550.048894.61×10−6 x413540.0362313.67×10−619730.0521646.45×10−615590.048266.96×10−6 x511460.049637.21×10−614520.0405296.71×10−616630.059693.37×10−6 x613540.0549714.05×10−619720.123035.71×10−616630.062535.64×10−6 10,000
x110420.0496144.98×10−617670.0747796.6×10−611430.067326.85×10−6 x211460.0615955.36×10−613500.083086.11×10−612470.122329.72×10−6 x310420.0545876.02×10−618700.0855549.83×10−614550.082886.51×10−6 x413540.0733335.16×10−619730.105799.07×10−615590.084139.82×10−6 x512500.063062.83×10−614520.0749829.18×10−616630.095894.75×10−6 x613540.0622595.69×10−619720.0991678.02×10−616640.114998.55×10−6 50,000
x111460.207033.1×10−618710.394737.37×10−612470.278265.23×10−6 x212500.232513.35×10−614540.273466.74×10−613510.296427.11×10−6 x311460.213383.73×10−620780.372495.5×10−615590.356024.82×10−6 x414580.32323.22×10−621810.375915.07×10−6351410.694706.69×10−6 x512500.227036.27×10−616600.263395.02×10−6351410.684889.12×10−6 x614580.259793.54×10−620760.338148.93×10−6351410.709739.91×10−6 100,000 x111460.555114.38×10−619750.654945.22×10−612470.445417.39×10−6 x212500.546944.73×10−614540.49449.52×10−614550.532993.39×10−6 x311460.409225.27×10−620780.783197.78×10−615600.586038.71×10−6 x414580.620494.55×10−621810.760517.17×10−6722902.706308.31×10−6 x512500.470398.86×10−616600.585457.07×10−6722902.722208.68×10−6 x614580.711745.01×10−621800.770516.32×10−6722902.758508.96×10−6
Table3.NumericalResultsforAlgorithm1(DCG),PCGandPDYforProblem3withgiveninitialpointsanddimensions. Algorithm1(DCG)PCGPDY DIMENSIONINITIALPOINTITERFVALTIMENORMITERFVALTIMENORMITERFVALTIMENORM 1000
x110430.753229.9×10−619760.557525.62×10−612480.012554.45×10−6 x211470.0069335.46×10−620800.0109365.58×10−612480.013119.02×10−6 x312510.006763.48×10−621840.0110486.58×10−613520.014868.34×10−6 x412510.0096644.41×10−622880.0110585.67×10−614560.016988.04×10−6 x511470.0104879.06×10−622880.0121985.64×10−614560.015519.72×10−6 x613550.0127023.15×10−621840.0182318.36×10−614560.015349.42×10−6 5000
x111470.0194586.19×10−620800.0408086.29×10−612480.036609.94×10−6 x212510.0215623.42×10−621840.066886.25×10−613520.036166.85×10−6 x312510.0242747.79×10−622880.041447.37×10−614560.045946.14×10−6 x412510.0267719.86×10−623920.0522146.35×10−615600.043426.01×10−6 x512510.0268145.67×10−623920.0414446.31×10−615600.042967.25×10−6 x613550.0239037.03×10−622880.0401359.37×10−6321290.100818.85×10−6 10,000
x111470.0441348.75×10−620800.0643128.9×10−613520.061924.77×10−6 x212510.0519474.83×10−621840.0881028.84×10−613520.064429.68×10−6 x313550.0572913.08×10−623920.072965.22×10−614560.094998.69×10−6 x413550.0551343.9×10−623920.0752658.99×10−615600.076968.5×10−6 x512510.0475518.02×10−623920.0739378.93×10−6331330.186256.45×10−6 x613550.0550699.95×10−623920.0998886.64×10−6331330.155487.51×10−6 50,000
x112510.199385.47×10−621840.270319.97×10−614560.236423.51×10−6 x213550.224993.02×10−622880.26579.9×10−614560.248137.12×10−6 x313550.193966.89×10−624960.32465.85×10−615600.270496.53×10−6 x413550.202598.72×10−6251000.323735.04×10−6341370.545457.13×10−6 x513550.194525.01×10−6251000.337645.01×10−6682741.023309.99×10−6 x614590.220156.22×10−624960.336877.44×10−6692781.038108.05×10−6 100,000 x112510.399837.74×10−622880.638097.06×10−614560.454754.96×10−6 x213550.327654.28×10−623920.634587.02×10−615600.490183.39×10−6 x313550.301339.75×10−624960.714228.27×10−615600.490169.24×10−6 x414590.428653.45×10−6251000.735247.13×10−61395594.031109.01×10−6 x513550.345127.09×10−6251000.706257.09×10−6702822.071008.54×10−6 x614590.403878.8×10−6251000.767775.27×10−61395594.024409.38×10−6
Table4.NumericalResultsforAlgorithm1(DCG),PCGandPDYforProblem4withgiveninitialpointsanddimensions. Algorithm1PCGPDY DIMENSIONINITIALPOINTITERFVALTIMENORMITERFVALTIMENORMITERFVALTIMENORM 1000
x110430.154618.33×10−618720.118539.93×10−612480.009894.60×10−6 x211470.0062763.84×10−619760.0143188.75×10−612480.009669.57×10−6 x311470.0098593.91×10−620800.00937767.15×10−613520.008878.49×10−6 x411470.0079765.21×10−6471890.0233217.83×10−612480.012075.83×10−6 x512510.0083824.09×10−6461850.0471059.76×10−6291170.053719.43×10−6 x612510.0086453.32×10−6411650.0277198.77×10−6291170.023966.65×10−6 5000
x111470.0220245.21×10−620800.0294455.57×10−613520.025033.49×10−6 x211470.0205878.59×10−620800.0331159.8×10−613520.026267.24×10−6 x311470.0237148.75×10−621840.0333188.01×10−614560.033496.29×10−6 x412510.0247283.26×10−6491970.0717159.46×10−613520.022584.25×10−6 x512510.0310159.14×10−6491970.0685658.68×10−6311250.054717.59×10−6 x612510.0300127.43×10−6441770.0708627.79×10−6632540.100648.54×10−6 10,000
x111470.0414767.37×10−620800.0430137.88×10−613520.037614.93×10−6 x212510.0478663.4×10−621840.0516856.94×10−614560.041003.37×10−6 x312510.0426073.46×10−622880.0504225.67×10−614560.039198.90×10−6 x412510.0364064.61×10−6502010.175639.84×10−6321290.096136.02×10−6 x513550.0413743.61×10−6502010.200359.03×10−6321290.091776.44×10−6 x613550.0398472.94×10−6451810.122148.11×10−6642580.207919.39×10−6 50,000
x112510.139284.61×10−621840.271458.83×10−614560.171933.63×10−6 x212510.180317.6×10−622880.231497.78×10−614560.152377.54×10−6 x312510.125267.74×10−623920.287896.36×10−615600.165496.66×10−6 x413550.143222.88×10−6532130.616248.75×10−6672700.762837.81×10−6 x513550.179048.08×10−6532130.71198.02×10−6672700.761578.80×10−6 x613550.136356.57×10−6471890.481929.8×10−626910802.925109.41×10−6 100,000 x112510.242936.52×10−622880.608226.25×10−614560.302295.13×10−6 x213550.274333.01×10−623920.529655.51×10−615600.316483.59×10−6 x313550.27143.06×10−623920.570648.99×10−6321290.728389.99×10−6 x413550.268194.08×10−6542171.18059.1×10−61355432.867809.73×10−6 x514590.316963.2×10−6542171.1078.34×10−627210925.741409.91×10−6 x613550.26989.29×10−6491971.06177.49×10−6548219711.441309.87×10−6
Table5.NumericalResultsforAlgorithm1(DCG),PCGandPDYforProblem5withgiveninitialpointsanddimensions. Algorithm1PCGPDY DIMENSIONINITIALPOINTITERFVALTIMENORMITERFVALTIMENORMITERFVALTIMENORM 1000
x119780.717098.63×10−622830.0993387.48×10−616630.075756.03×10−6 x221860.0171277.65×10−623880.0160147.31×10−616630.014705.42×10−6 x323950.0139097.23×10−623900.0163289.31×10−6331320.022086.75×10−6 x422920.01658.64×10−6491970.0301248.45×10−6301210.018358.39×10−6 x5351450.0247028.26×10−6532130.0393218.38×10−6321290.027008.47×10−6 x6431820.0274718.7×10−6461850.0336278.8×10−6301210.017126.95×10−6 5000
x11465920.238039.45×10−624910.0601586.36×10−617670.043945.64×10−6 x221860.043379.46×10−625950.0603856.24×10−617670.046355.07×10−6 x324990.0546198.27×10−625980.0400155.86×10−6351400.083119.74×10−6 x4241000.0664246.66×10−6532130.0980979.11×10−6331330.080756.02×10−6 x5381570.0712229.28×10−6582330.109588.56×10−6351410.100917.51×10−6 x6451900.0902767.14×10−6502010.215217.65×10−6321290.080548.55×10−6 10,000
x12118530.603579.65×10−625950.0764275.4×10−617670.068168.81×10−6 x222900.080124.98×10−625950.0984618.9×10−617670.088337.80×10−6 x3251030.0892695.89×10−625980.074958.64×10−6371480.147326.36×10−6 x4251040.117815.54×10−6552210.190489.11×10−6371490.142938.25×10−6 x5401650.158597.43×10−6602410.197519.01×10−6361450.147198.23×10−6 x6461940.17288.62×10−6512050.288829.62×10−6742980.264567.79×10−6 50,000
x12259092.13739.93×10−626990.345756.75×10−6421690.581137.78×10−6 x223940.310984.48×10−6271030.438065.16×10−6421690.584567.13×10−6 x3261070.362936.83×10−6271060.48155.28×10−6411650.587178.87×10−6 x4261080.324279.72×10−6602410.908688.66×10−6401610.564317.17×10−6 x5431770.489389.47×10−6652610.79249.05×10−6823301.089208.44×10−6 x6502100.691178.12×10−6562250.723348.19×10−6803221.066707.82×10−6 100,000 x12319334.25889.85×10−626990.712429.73×10−6431731.096208.47×10−6 x21395642.72669.96×10−6271030.627467.39×10−6431731.100407.77×10−6 x3261070.575059.92×10−6271060.829897.77×10−6421691.083309.66×10−6 x4271120.622278.52×10−6622491.54749×10−6853422.118809.22×10−6 x5451850.89927.79×10−6672691.66929.5×10−6843382.106409.78×10−6 x6522181.43187.37×10−6582331.43338.32×10−61676714.062009.90×10−6
Table6.NumericalResultsforAlgorithm1(DCG),PCGandPDYforProblem6withgiveninitialpointsanddimensions. Algorithm1PCGPDY DIMENSIONINITIALPOINTITERFVALTIMENORMITERFVALTIMENORMITERFVALTIMENORM 1000
x113551.385.68×10−623920.40389.28×10−615600.016714.35×10−6 x213550.0133395.47×10−623920.0163258.92×10−615600.013464.18×10−6 x313550.0661424.81×10−623920.0230457.86×10−615600.016303.68×10−6 x413550.0268383.3×10−623920.0161725.38×10−614560.013397.48×10−6 x512510.0098649.45×10−622880.037858.62×10−614560.012676.01×10−6 x112510.0098815.57×10−622880.0150135.08×10−614560.016853.54×10−6 5000
x114590.0425333.56×10−6251000.0616425.22×10−615600.050389.73×10−6 x214590.0366483.43×10−6251000.0929525.02×10−615600.047759.36×10−6 x314590.0434523.02×10−624960.0681418.82×10−615600.049238.25×10−6 x413550.0325797.38×10−624960.0846256.04×10−615600.057935.64×10−6 x513550.032955.92×10−623920.0861229.67×10−615600.045974.53×10−6 x613550.0330623.49×10−623920.0933185.7×10−614560.050707.93×10−6 10,000
x114590.0649175.04×10−6251000.214247.38×10−6682740.407249.06×10−6 x214590.0699134.84×10−6251000.139787.09×10−6682740.418188.72×10−6 x314590.084734.27×10−6251000.17316.25×10−6341370.219056.22×10−6 x414590.0758472.92×10−624960.147448.54×10−615600.100767.98×10−6 x513550.079748.38×10−624960.141696.85×10−615600.126806.40×10−6 x613550.0631294.94×10−623920.152948.06×10−615600.119843.78×10−6 50,000
x115630.253293.15×10−6261040.646698.26×10−61435753.091209.42×10−6 x215630.363943.03×10−6261040.677177.95×10−61435753.062009.06×10−6 x314590.24139.54×10−6261040.55627×10−61425713.049509.04×10−6 x414590.275026.53×10−6251000.561719.56×10−6692781.539209.14×10−6 x514590.364045.24×10−6251000.579827.67×10−6682741.494909.43×10−6 x614590.25063.09×10−624960.586459.03×10−615600.381778.44×10−6 100,000 x115630.847814.45×10−6271081.32155.86×10−6292117213.595309.53×10−6 x215630.666634.28×10−6271081.50625.63×10−6290116413.309309.75×10−6 x315630.666833.77×10−6261041.1669.9×10−61445796.681509.96×10−6 x414590.626979.24×10−6261041.39616.78×10−61415676.508009.92×10−6 x514590.628917.41×10−6261041.27115.44×10−6702823.305108.07×10−6 x614590.624224.37×10−6251001.16856.4×10−6341371.645106.37×10−6
Table7.NumericalResultsforAlgorithm1(DCG),PCGandPDYforProblem7withgiveninitialpointsanddimensions. Algorithm1PCGPDY DIMENSIONINITIALPOINTITERFVALTIMENORMITERFVALTIMENORMITERFVALTIMENORM 1000
x16280.256892×10−617691.22756.98×10−614570.009535.28×10−6 x26280.0084691.26×10−615610.233969.89×10−613530.008969.05×10−6 x34200.0036199.25×10−616650.0080955.79×10−63120.004268.47×10−6 x45240.0043455.7×10−616650.0100775.21×10−615610.011696.73×10−6 x56280.0071464.42×10−619770.053544.95×10−6311260.036469.03×10−6 x66270.0042994.43×10−618720.0256778.93×10−615600.010823.99×10−6 5000
x16280.0129154.47×10−618730.177227.6×10−615610.032154.25×10−6 x26280.0122722.81×10−617690.0277295.25×10−614570.029427.40×10−6 x35240.0146691.16×10−617690.029856.31×10−64160.011071.01×10−7 x46280.0127657.14×10−717690.0281765.68×10−616650.043315.43×10−6 x56280.013319.89×10−620810.0322135.39×10−6331340.093797.78×10−6 x66270.0158289.91×10−619760.0443289.73×10−615600.040778.92×10−6 10,000
x16280.0223466.32×10−619770.178635.23×10−615610.064846.01×10−6 x26280.0226693.97×10−617690.0492427.42×10−615610.077343.77×10−6 x35240.0393421.64×10−617690.0482388.92×10−64160.027071.42×10−7 x46280.0210171.01×10−617690.048078.03×10−616650.079417.69×10−6 x57320.0316547.83×10−720810.0631567.62×10−6341380.149426.83×10−6 x67310.0234567.85×10−720800.0594386.7×10−6341380.152248.81×10−6 50,000
x17320.0924527.91×10−720811.08085.7×10−616650.259954.89×10−6 x26280.10688.88×10−618730.328048.08×10−615610.246748.42×10−6 x35240.0656843.66×10−618730.21899.71×10−64160.094053.18×10−7 x46280.101932.26×10−618730.34978.75×10−6361460.552076.39×10−6 x57320.0956761.75×10−621850.225958.3×10−6351420.546799.05×10−6 x67310.0928551.76×10−621840.223747.3×10−6361460.557647.59×10−6 100,000 x17320.175971.12×10−620812.16758.06×10−617690.525955.68×10−6 x27320.17417.03×10−719770.455535.57×10−616650.521024.34×10−6 x35240.175225.18×10−619770.432196.69×10−64160.148644.50×10−7 x46280.207853.19×10−619770.522596.03×10−6361461.053609.04×10−6 x57320.239792.48×10−622890.61715.72×10−6742992.107308.55×10−6 x67310.231282.48×10−622880.573845.03×10−6371501.082406.66×10−6
Table8.NumericalResultsforAlgorithm1(DCG),PCGandPDYforProblem8withgiveninitialpointsanddimensions. Algorithm1PCGPDY DIMENSIONINITIALPOINTITERFVALTIMENORMITERFVALTIMENORMITERFVALTIMENORM 1000
x17280.114953.03×10−69320.857977.6×10−6692790.055388.95×10−6 x27280.0050343.03×10−69320.0346757.6×10−627010850.187989.72×10−6 x37280.0067433.03×10−69320.0059857.6×10−624520.024396.57×10−6 x47280.0058563.03×10−69320.0048087.6×10−627580.015207.59×10−6 x57280.0046353.03×10−69320.0150267.6×10−628610.043309.21×10−6 x67280.0064873.03×10−69320.157787.6×10−640850.021168.45×10−6 5000
x15220.0090684.52×10−67260.672391.3×10−665826391.130309.98×10−6 x25220.0093694.52×10−67260.0106511.3×10−627580.051017.59×10−6 x35220.0108954.52×10−67260.0157581.3×10−6491040.080358.11×10−6 x45220.0149584.52×10−67260.0149351.3×10−640850.079798.45×10−6 x55220.015074.52×10−67260.015241.3×10−618400.091289.14×10−6 x65220.0087164.52×10−67260.19991.3×10−617380.185288.98×10−6 10,000
x16270.0311983.81×10−65190.0443875.06×10−6491040.204437.62×10−6 x26270.020983.81×10−65190.02235.06×10−640850.158018.45×10−6 x36270.019913.81×10−65190.0182095.06×10−619420.378807.66×10−6 x46270.0254023.81×10−65190.0216545.06×10−6901871.258029.7×10−6 x56270.0258163.81×10−65190.0173535.06×10−6988198812.682599.93×10−6 x66270.0250653.81×10−65190.0197635.06×10−627580.328597.59×10−6 50,000
x14210.0836412.34×10−78330.429025.15×10−619420.522916.42×10−6 x24210.0741562.34×10−78330.115255.15×10−61483043.930639.92×10−6 x34210.0785962.34×10−78330.144325.15×10−6937188622.970979.87×10−6 x44210.0782892.34×10−78330.115625.15×10−627580.684677.59×10−6 x54210.0735352.34×10−78330.116745.15×10−63467028.450439.79×10−6 x64210.0819092.34×10−78330.104865.15×10−640850.992308.45×10−6 100,000 x14220.16636.25×10−66251.29226.81×10−7---- x24220.151476.25×10−66250.188396.81×10−7---- x34220.155826.25×10−66250.161536.81×10−7---- x44220.154656.25×10−66250.173976.81×10−7---- x54220.167446.25×10−66250.185866.81×10−7---- x64220.16876.25×10−66250.179386.81×10−7----
Table9.NumericalResultsforAlgorithm1(DCG),PCGandPDYforProblem9withgiveninitialpointsanddimensions. Algorithm1PCGPDY DIMENSIONINITIALPOINTITERFVALTIMENORMITERFVALTIMENORMITERFVALTIMENORM 4
x1512150.236659.01×10−6793210.59789.76×10−6592410.712689.36×10−6 x2512150.049689.99×10−6773130.0163269.85×10−6582370.0454419.73×10−6 x3532230.0172119.46×10−6803250.165299.38×10−6592410.0195529.9×10−6 x4532230.0190049.68×10−6833370.0417139.57×10−6622530.0220078.07×10−6 x5572390.0234478.87×10−6813290.119729.04×10−6612490.0401178.36×10−6 x6542270.0208329.31×10−6823330.0161279.3×10−6612490.0173749.18×10−6 012345678 t
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
P(t)
PCG
PDY Algorithm 2.3 Figure1.Performanceprofilesforthenumberofiterations.
0 1 2 3 4 5 6 7 8 9 10 t
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
P(t)
PCG
PDY
Algorithm 2.3
Figure 2.Performance profiles for the CPU time (in seconds).
0 1 2 3 4 5 6 7
t 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
P(t)
PDY
PCG
Algorithm 2.3
Figure 3.Performance profiles for the number of function evaluations.
10 15 20 25 30 35 40 45 50 55 60 65
ITERATION 0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
RESIDUALS NORM
Algorithm2.2 PCG PDY
Figure 4.Convergence histories ofAlgorithm1,PCGandPDYon Problem 9.
Applications in Compressive Sensing
There are many problems in signal processing and statistical inference involving finding sparse solutions to ill-conditioned linear systems of equations. Among popular approach is minimizing an objective function which contains quadratic (2) error term and a sparse1−regularization term, that is,
minx 1
2y−Bx22+ηx1, (24)
wherex∈Rn,y∈Rkis an observation,B∈Rk×n(k<<n) is a linear operator,ηis a non-negative parameter,x2denotes the Euclidean norm ofxandx1=∑ni=1|xi|is the1−norm ofx. It is easy to see that problem (24) is a convex unconstrained minimization problem. Due to the fact that if the original signal is sparse or approximately sparse in some orthogonal basis, problem (24) frequently appears in compressive sensing and hence an exact restoration can be produced by solving (24).
Iterative methods for solving (24) have been presented in many papers (see References [5,31–35]).
The most popular method among these methods is the gradient based method and the earliest gradient projection method for sparse reconstruction (GPRS) was proposed by Figueiredo et al. [5]. The first step of the GPRS method is to express (24) as a quadratic problem using the following process. Letx∈Rn and splitting it into its positive and negative parts. Thenxcan be formulated as
x=u−v, u≥0, v≥0,
whereui= (xi)+,vi= (−xi)+for alli=1, 2, ...,nand(.)+=max{0, .}. By definition of1-norm, we havex1=eTnu+eTnv, whereen= (1, 1, ..., 1)T∈Rn. Now (24) can be written as
minu,v
1
2y−B(u−v)22+ηeTnu+ηeTnv, u≥0, v≥0, (25) which is a bound-constrained quadratic program. However, from Reference [5], Equation (25) can be written in standard form as
minz
1
2zTDz+cTz, such that z≥0, (26)
wherez= u
v
, c=ωe2n+ −b
b
, b=BTy, D=
BTB −BTB
−BTB BTB
.
Clearly, D is a positive semi-definite matrix, which implies that Equation (26) is a convex quadratic problem.
Xiao et al. [19] translated (26) into a linear variable inequality problem which is equivalent to a linear complementarity problem. Furthermore, it was noted thatzis a solution of the linear complementarity problem if and only if it is a solution of the nonlinear equation:
F(z) =min{z,Dz+c}=0. (27)
The functionFis a vector-valued function and the “min” is interpreted as component-wise minimum.
It was proved in References [36,37] thatF(z)is continuous and monotone. Therefore problem (24) can be translated into problem (1) and thus Algorithm1(DCG) can be applied to solve it.
In this experiment, we consider a simple compressive sensing possible situation, where our goal is to restore a blurred image. We use the following well-known gray test images; (P1) Cameraman, (P2) Lena, (P3) House and (P4) Peppers for the experiments. We use 4 different Gaussian blur kernals with standard deviationσto compare the robustness of DCG method with CGD method proposed in Reference [19]. CGD method is an extension of the well-known conjugate gradient method for unconstrained optimization CG-DESCENT [20] to solve the1-norm regularized problems.
To access the performance of each algorithm tested with respect to metrics that indicate a better quality of restoration, in Table10we reported the number of iterations, the objective function (ObjFun) value at the approximate solution, the mean of squared error (MSE) to the original image ˜x,
MSE= 1
nx˜−x∗2,
wherex∗is the reconstructed image and the signal-to-noise-ratio (SNR) which is defined as SNR=20×log10 x¯
x−x¯ .
We also reported the structural similarity (SSIM) index that measure the similarity between the original image and the restored image [38]. The MATLAB implementation of the SSIM index can be obtained athttp://www.cns.nyu.edu/~lcv/ssim/.
Table 10.Efficiency comparison based on the value of the number of iterations (Iter), objective function (ObjFun) value, mean-square-error (MSE) and signal-to-noise-ratio (SNR) under different Pi (σ).
Image Iter ObjFun MSE SNR
DCG CGD DCG CGD DCG CGD DCG CGD
P1(1E-8) 8 9 4.397×103 4.398×103 3.136×10−2 3.157×10−2 9.42 9.39 P1(1E-1) 8 9 4.399×103 4.401×103 3.147×10−2 3.163×10−2 9.40 9.38 P1(0.11) 11 8 4.428×103 4.432×103 3.229×10−2 3.232×10−2 9.29 9.29 P1(0.25) 12 8 4.468×103 4.473×103 3.365×10−2 3.289×10−2 9.11 9.21 P1(1E-8) 9 9 4.555×103 4.556×103 3.287×10−2 3.3412×10−2 9.14 9.07 P1(1E-1) 9 9 4.558×103 4.559×103 3.298×10−2 3.348×10−2 9.12 9.06 P1(0.11) 12 12 4.588×103 4.591×103 3.416×10−2 3.446×10−2 8.97 8.93 P1(0.25) 7 8 4.628×103 4.630×103 3.621×10−2 3.500×10−2 8.72 8.86 P1(1E-8) 9 9 5.179×103 5.179×103 3.209×10−2 3.3259×10−2 10.03 9.96 P1(1E-1) 9 9 5.182×103 5.182×103 3.231×10−2 3.267×10−2 10.00 9.95 P1(0.11) 7 9 5.209×103 5.209×103 3.436×10−2 3.344×10−2 9.73 9.85 P1(0.25) 10 8 5.250×103 5.254×103 3.557×10−2 3.438×10−2 9.58 9.73 P1(1E-8) 9 9 4.388×103 4.389×103 3.299×10−2 3.335×10−2 9.03 8.99 P1(1E-1) 9 9 4.391×103 4.393×103 3.308×10−2 3.340×10−2 9.02 8.98 P1(0.11) 12 8 4.421×103 4.424×103 3.425×10−2 3.411×10−2 8.87 8.89 P1(0.25) 7 8 4.461×103 4.463×103 3.621×10−2 3.483×10−2 8.63 8.80
The original, blurred and restored images by each of the algorithm are given in Figures5–8.
The figures demonstrate that both the two tested algorithm can restored the blurred images. It can be observed from Table10and Figures5–8that Algorithm1(DCG) compete with the CGD algorithm, therefore, it can be used as an alternative to CGD for restoring blurred image.
Original Blurred
Recovered by CGD Recovered by DCG
Figure 5. The original image (top left), the blurred image (top right), the restored image by CGD (bottom left) with SNR = 20.05, SSIM = 0.83 and by DCG (bottom right) with SNR = 20.12, SSIM = 0.83.
Original Blurred
Recovered by CGD Recovered by DCG
Figure 6. The original image (top left), the blurred image (top right), the restored image by CGD (bottom left) with SNR = 22.93, SSIM = 0.87 and by DCG (bottom right) with SNR = 24.36, SSIM = 0.90.
Original Blurred
Recovered by CGD Recovered by DCG
Figure 7. The original image (top left), the blurred image (top right), the restored image by CGD (bottom left) with SNR = 25.65, SSIM = 0.86 and by DCG (bottom right) with SNR = 26.37, SSIM = 0.87.
Original Blurred
Recovered by CGD Recovered by DCG
Figure 8. The original image (top left), the blurred image (top right), the restored image by CGD (bottom left) with SNR = 21.50, SSIM = 0.84 and by DCG (bottom right) with SNR = 21.81, SSIM = 0.85.
4. Conclusions
In this research article, we present a CG method which possesses the sufficient descent property for solving constrained nonlinear monotone equations. The proposed method has the ability to solve non-smooth equations as it does not require matrix storage and Jacobian information of the nonlinear equation under consideration. The sequence of iterates generated converge the solution under appropriate assumptions. Finally, we give some numerical examples to display the efficiency of the proposed method in terms of number of iterations, CPU time and number of function evaluations compared with some related methods for solving convex constrained nonlinear monotone equations and its application in image restoration problems.
Author Contributions:conceptualization, A.B.A.; methodology, A.B.A.; software, H.M.; validation, P.K. and A.M.A.; formal analysis, P.K. and H.M.; investigation, P.K. and A.M.A.; resources, P.K.; data curation, A.B.A. and H.M.; writing—original draft preparation, A.B.A.; writing—review and editing, H.M.; visualization, A.M.A.;
supervision, P.K.; project administration, P.K.; funding acquisition, P.K.
Funding: Petchra Pra Jom Klao Doctoral Scholarship for Ph.D. program of King Mongkut’s University of Technology Thonburi (KMUTT) and Theoretical and Computational Science (TaCS) Center. Moreover, this project was partially supported by the Thailand Research Fund (TRF) and the King Mongkut’s University of Technology Thonburi (KMUTT) under the TRF Research Scholar Award (Grant No. RSA6080047).
Acknowledgments:We thank Associate Professor Jin Kiu Liu for providing us with the access of the CGD-CS MATLAB codes. The authors acknowledge the financial support provided by King Mongkut’s University of Technology Thonburi through the “KMUTT 55th Anniversary Commemorative Fund”. This project is supported by the theoretical and computational science (TaCS) center under computational and applied science for smart research innovation (CLASSIC), Faculty of Science, KMUTT. The first author was supported by the “Petchra Pra Jom Klao Ph.D. Research Scholarship from King Mongkut’s University of Technology Thonburi”.
Conflicts of Interest:The authors declare no conflict of interest.
References
1. Gu, B.; Sheng, V.S.; Tay, K.Y.; Romano, W.; Li, S. Incremental support vector learning for ordinal regression.
IEEE Trans. Neural Netw. Learn. Syst.2015,26, 1403–1416. [CrossRef] [PubMed]
2. Li, J.; Li, X.; Yang, B.; Sun, X. Segmentation-based image copy-move forgery detection scheme.IEEE Trans.
Inf. Forensics Secur.2015,10, 507–518.
3. Wen, X.; Shao, L.; Xue, Y.; Fang, W. A rapid learning algorithm for vehicle classification. Inf. Sci.2015,295, 395–406.
[CrossRef]
4. Michael, S.V.; Alfredo, I.N. Newton-type methods with generalized distances for constrained optimization.
Optimization1997,41, 257–278.
5. Figueiredo, M.A.T.; Nowak, R.D.; Wright, S.J. Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems.IEEE J. Sel. Top. Signal Process.2007,1, 586–597. [CrossRef]
6. Magnanti, T.L.; Perakis, G. Solving variational inequality and fixed point problems by line searches and potential optimization.Math. Program.2004,101, 435–461. [CrossRef]
7. Pan, Z.; Zhang, Y.; Kwong, S. Efficient motion and disparity estimation optimization for low complexity multiview video coding.IEEE Trans. Broadcast.2015,61, 166–176.
8. Xia, Z.; Wang, X.; Sun, X.; Wang, Q. A secure and dynamic multi-keyword ranked search scheme over encrypted cloud data.IEEE Trans. Parallel Distrib. Syst.2016,27, 340–352. [CrossRef]
9. Zheng, Y.; Jeon, B.; Xu, D.; Wu, Q.M.; Zhang, H. Image segmentation by generalized hierarchical fuzzy c-means algorithm.J. Intell. Fuzzy Syst.2015,28, 961–973.
10. Solodov, M.V.; Svaiter, B.F. A globally convergent inexact newton method for systems of monotone equations.
InReformulation: Nonsmooth, Piecewise Smooth, Semismooth and Smoothing Methods; Springer: Dordrecht, The Netherlands, 1998; pp. 355–369.
11. Mohammad, H.; Abubakar, A.B. A positive spectral gradient-like method for nonlinear monotone equations.
Bull. Comput. Appl. Math.2017,5, 99–115.
12. Zhang, L.; Zhou, W. Spectral gradient projection method for solving nonlinear monotone equations.
J. Comput. Appl. Math.2006,196, 478–484. [CrossRef]
13. Zhou, W.J.; Li, D.H. A globally convergent BFGS method for nonlinear monotone equations without any merit functions.Math. Comput.2008,77, 2231–2240. [CrossRef]
14. Zhou, W.; Li, D. Limited memory BFGS method for nonlinear monotone equations.J. Comput. Math.2007, 25, 89–96.
15. Abubakar, A.B.; Waziria, M.Y. A matrix-free approach for solving systems of nonlinear equations.J. Mod.
Methods Numer. Math.2016,7, 1–9. [CrossRef]
16. Abubakar, A.B.; Kumam, P. An improved three-term derivative-free method for solving nonlinear equations.
Comput. Appl. Math.2018,37, 6760–6773. [CrossRef]
17. Abubakar, A.B.; Kumam, P.; Awwal, A.M. A descent dai-liao projection method for convex constrained nonlinear monotone equations with applications.Thai J. Math.2018,17, 128–152.
18. Wang, C.; Wang, Y.; Xu, C. A projection method for a system of nonlinear monotone equations with convex constraints.Math. Methods Oper. Res.2007,66, 33–46. [CrossRef]
19. Xiao, Y.; Zhu, H. A conjugate gradient method to solve convex constrained monotone equations with applications in compressive sensing.J. Math. Anal. Appl.2013,405, 310–319. [CrossRef]
20. Hager, W.; Zhang, H. A new conjugate gradient method with guaranteed descent and an efficient line search.
SIAM J. Optim.2005,16, 170–192. [CrossRef]
21. Liu, S.-Y.; Huang, Y.-Y.; Jiao, H.-W. Sufficient descent conjugate gradient methods for solving convex constrained nonlinear monotone equations.Abstr. Appl. Anal.2014,2014, 305643. [CrossRef]
22. Liu, J.K.; Li, S.J. A projection method for convex constrained monotone nonlinear equations with applications.
Comput. Math. Appl.2015,70, 2442–2453. [CrossRef]
23. Ding, Y.; Xiao, Y.; Li, J. A class of conjugate gradient methods for convex constrained monotone equations.
Optimization2017,66, 2309–2328. [CrossRef]
24. Liu, J.; Feng, Y. A derivative-free iterative method for nonlinear monotone equations with convex constraints.
Numer. Algorithms2018, 1–18. [CrossRef]
25. Muhammed, A.A.; Kumam, P.; Abubakar, A.B.; Wakili, A.; Pakkaranang, N. A new hybrid spectral gradient projection method for monotone system of nonlinear equations with convex constraints.Thai J. Math.2018, 16, 125–147.
26. La Cruz, W.; Martínez, J.; Raydan, M. Spectral residual method without gradient information for solving large-scale nonlinear systems of equations.Math. Comput.2006,75, 1429–1448. [CrossRef]
27. Bing, Y.; Lin, G. An efficient implementation of Merrill’s method for sparse or partially separable systems of nonlinear equations.SIAM J. Optim.1991,1, 206–221. [CrossRef]
28. Yu, Z.; Lin, J.; Sun, J.; Xiao, Y.H.; Liu, L.Y.; Li, Z.H. Spectral gradient projection method for monotone nonlinear equations with convex constraints.Appl. Numer. Math.2009,59, 2416–2423. [CrossRef]
29. Yamashita, N.; Fukushima, M. Modified Newton methods for solving a semismooth reformulation of monotone complementarity problems.Math. Program.1997,76, 469–491. [CrossRef]
30. Dolan, E.D.; Moré, J.J. Benchmarking optimization software with performance profiles.Math. Program.2002, 91, 201–213. [CrossRef]
31. Figueiredo, M.A.T.; Nowak, R.D. An EM algorithm for wavelet-based image restoration. IEEE Trans.
Image Process.2003,12, 906–916. [CrossRef]
32. Hale, E.T.; Yin, W.; Zhang, Y. A Fixed-Point Continuation Method for1-Regularized Minimization with Applications to Compressed Sensing; CAAM TR07-07; Rice University: Houston, TX, USA, 2007; pp. 43–44.
33. Beck, A.; Teboulle, M. A fast iterative shrinkage-thresholding algorithm for linear inverse problems.SIAM J.
Imaging Sci.2009,2, 183–202. [CrossRef]
34. Van Den Berg, E.; Friedlander, M.P. Probing the pareto frontier for basis pursuit solutions. SIAM J.
Sci. Comput.2008,31, 890–912. [CrossRef]
35. Birgin, E.G.; Martínez, J.M.; Raydan, M. Nonmonotone spectral projected gradient methods on convex sets.
SIAM J. Optim.2000,10, 1196–1211. [CrossRef]
36. Xiao, Y.; Wang, Q.; Hu, Q. Non-smooth equations based method for1-norm problems with applications to compressed sensing.Nonlinear Anal. Theory Methods Appl.2011,74, 3570–3577. [CrossRef]
37. Pang, J.-S. Inexact Newton methods for the nonlinear complementarity problem. Math. Program.1986,36, 54–71.
[CrossRef]
38. Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity.IEEE Trans. Image Process.2004,13, 600–612. [CrossRef]
c2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
mathematics
Article