A New Heuristic for the Convex Quadratic Programming Problem

This paper presents a new heuristic to linearise the convex quadratic programming problem. The usual Karush-Kuhn-Tucker conditions are used but in this case a linear objective function is also formulated from the set of linear equations and complementarity slackness conditions. An unboundedness challenge arises in the proposed formulation and this challenge is alleviated by construction of an additional constraint. The formulated linear programming problem can be solved efficiently by the available simplex or interior point algorithms. There is no restricted base entry in this new formulation. Some computational experiments were carried out and results are provided.

• Convex Quadratic Programming
• Linear Programming
• Karush-Kuhn-Tucker Conditions
• Simplex Method
• Interior Point Method

1. Introduction

There are so many real life applications for the convex quadratic programming (QP) problem. The applications include portfolio analysis, structural analysis, discrete-time stabilisation, optimal control, economic dispatch and finite impulse design; see [1] -[3] . Some of the methods for solving the convex quadratic problem are active set, interior point, branch and bound, gradient projection, and Lagrangian methods, see [4] -[9] for more information on these methods.

In this paper we present a new heuristic to linearise the convex quadratic programming problem. The usual Karush-Kuhn-Tucker conditions are still used but in this case a linear objective function is also formulated from the set of linear objective function equations and the complementary slackness conditions. There is an unboundedness challenge that is associated with the proposed linear formulation. To alleviate this challenge, an additional constraint is constructed and added to the linear formulation. The new linear formulation can be solved efficiently by the available simplex and interior point algorithms. There is no restricted base entry in the proposed approach. The time consuming complementarity pivoting is no longer necessary. Some computational experiments have been carried out and the objective of the computational experiments was to determine CPU times of the:

1) Proposed heuristic;

2) Regularised Active Set Method Mae and Saunders [10] ;

3) Primal-Dual Interior Point Algorithm.

It may be noted that the proposed method is suitable only if the quadratic programming problem satisfies conditions (1) to (5) mentioned in Section 2.1.

2 Mathematical Background

2.1 The Quadratic Programming Problem

Let a quadratic programming (QP) problem be represented by (1).

Minimize

Subject to:

where

It is assumed that:

1) Matrix is a symmetric and positive definite,

2) Function is strictly convex,

3) The conditions and hold. Here and are dual and primal slack variables, respectively.

4) Since constraints are linear then the solution space is convex, and

5) Any maximization quadratic problem can be changed into a minimization and vice versa.

When the function is strictly convex for all points in the convex region then the quadratic problem has a unique local minimum which is also the global minimum [11] .

2.2. Karush-Kuhn-Tucker Conditions

The convex quadratic programming problem has special features that we can capitalize on when solving. All constraints are linear and the only nonlinear expression is the objective function. Let the Lagrangian function for the QP problem be and in this case

where and. In this case we exclude the non-negativity conditions. If and then the Karush-Kuhn-Tucker conditions as given in [11] for a local minimum are:

Complementary slackness conditions are given in (5) and are only satisfied at the optimal point. These conditions are:

Note and are and dimensional vectors representing the slack variables. At this stage, we are unable to apply the simplex algorithm due to restricted base entry and this makes the simplex method approximately 8 times slower than its full speed compared to its unrestricted basis version.

2.3. Some Matrix Operations

Suppose and are single row matrices of the same dimension and is an dimensional matrix, the following must hold.

Equations (6) and (7) can be easily verified. These simple results are used to eliminate the complementary slackness conditions.

3. Elimination of Complementary Slackness Conditions

3.1. Elimination of

Pre-multiply (3) by X, we have:

From (6) and from (5), then

By rearranging, we have

3.2. Elimination of

Pre-multiply (4) by, we have

Since from (4), then

3.3. Elimination of or

From (7), we have: , hence we can replace by in relations (11) to get t(15):

Subtracting (14) from (15), we obtain (16):

3.4. Linear Objective Function for the Quadratic Programming Problem

Note that the expression in relation (13) is nonlinear but it can be rearranged so that the original quadratic programming objective function becomes a linear quantity. This can be achieved as follows:

Divide relation (16) by two, one obtains:

Rearranging (17), we obtain (18):

From (1), then, (18) becomes (19) or equivalently (20):

Thus the nonlinear objective function of the QP problem is now linearised but it creates a new challenge. We will discuss this in the next section.

3.5. LP Equivalent to the Given QP

From (1), (3) and (20), we have the following LP problem:

Minimize

Subject to:

The minimisation problem (21) will have an unbounded solution due to negative coefficient of in the objective function and negative coefficients of the slack variable in the constraints. These are the only source

of unboundedness in the LP (21). Here, we let: and where a row vec-

tor of dimension. The objective function is now modified as :

Minimize, where and are very large constants relative to all other objective

coefficients. Both of these constants do not have to assume the same large values. A large number of experiments were done on a large number of quadratic programming problems and and it was observed that seems to work well. These experiments have been recorded later in this paper. In these experiments, it was noted that values of and on higher side can be as much as and.

3.6. Existence of a Linear Objective Function and Verification of Optimality

The optimal solution of a convex quadratic programming model is unique and it satisfies the complementary conditions and. The unique optimal solution to the convex quadratic programming is a corner point. Since the KKT conditions can be expressed as a linear objective function that can make exist.

4. Numerical Illustrations

4.1. Example 1

Minimize

Subject to:

This example was taken from Jensen and Bard (2012) without any modifications.

Linear formulation of the above QP

In this case we took and which are very large compared to coefficients 4; 8; 2.5; and 1.5. The LP problem is given by:

Maximize

Subject to

The solution of (23) by the simplex method is given by:

From the original QP objective function, we have the objective value given in (25).

Verification of optimality

The solution is optimal because complementary slackness conditions are satisfied as given in (26).

4.2. Two More Examples

Two more examples are solved to illustrate how the large constants are selected. Example 2 is taken from [12] and example 3 is from [13] .

Example 2 from [12]

Minimize

Subject to:

The linear formulation of (27) becomes as given in (28).

Maximize

Such that:

,

,

,

,

,

The solution of (28) is as given in (29) and once again it is optimal as all complementary slackness conditions are satisfied.

Example 3 from [13]

Minimize

Subject to:

The linear formulation of the above example is given by (30).

Maximize

Subject to:

;

The solution is given by: This solution is once again optimal as all complementary slackness conditions are satisfied.

5. Computational Experiments

A set of convex quadratic programming test problems are given in [14] . All these test problems were used in testing the proposed approach. The objective of the computational experiments was:

1) To determine that the LP optimal solution is also optimal to the given QP.

2) Compare CPU times of the proposed heuristics with Regularized Active Set Method and Primal-Dual Interior Point Method

The results are tabulated in Table 1. MATLAB R2013 (version 8.2) running on an Intel Pentium Dual desktop

Table 1

Computational experiments on the set of QP test problems

Exp. No.	Prob. Name	No. of constraints (m)	No. of Variables (n)	CPU secs Proposed Heuristic	CPU secs Active Set	CPU secs Interior Point
1	AUG2D	10,000	20,200	29.34	0.55	15.12
2	AUG2DC	10,000	20,200	34.39	0.57	14.25
3	AUG2DCQP	10,000	20,200	21.89	240.73	14.63
4	AUG2DQP	10,000	20,200	37.19	228.72	14.76
5	AUG3D	1000	3873	0.29	0.07	1.65
6	AUG3DC	1000	3873	0.45	0.06	1.69
7	AUG3DCQP	1000	3873	0.82	3.84	1.39
8	AUG3DQP	1000	3873	0.53	5.02	1.56
9	BOYD1	18	93,261	89.67	214.24	107.10
10	BOYD2	0	93,263	*	4168.93	2245.64

11	CONT-050	2401	2597	0.31	0.84	3.37
12	CONT-100	9801	10197	1.88	26.37	19.12
13	CONT-101	10,098	10197	112.7	35855.97	20.66
14	CONT-200	39,601	40,397	76.16	277.68	136.22
15	CONT-201	40,198	40,397	51.12	285.50	143.56
16	CONT-300	90,298	90,597	219.47	2449.75	721.76
17	CVXQP1-L	5000	10,000	413.18	4516.19	2488.97
18	CVXQP1-M	500	1000	0.65	5.94	1.85
19	CVXQP1-S	50	100	0.01	0.04	0.21
20	CVXQP2-L	2500	10,000	218.82	670.50	443.34
21	CVXQP2-M	250	1000	0.63	4.24	1.52
22	CVXQP2-S	25	100	0.02	0.04	0.26
23	CVXQP3-L	7500	10,000	76.22	14069.08	736.74
24	CVXQP3-M	750	1000	0.65	17.10	2.63
25	CVXQP3-S	75	100	0.02	0.40	0.23
26	DPKLO1	77	133	0.02	0.01	0.17
27	DTOC3	9998	1499	74.1	0.32	107.10
28	DUAL1	1	85	~0.00	0.03	0.46
29	DUAL2	1	96	~0.00	0.01	0.43
30	DUAL3	1	111	~0.00	0.03	0.58
31	DUAL4	1	75	~0.00	0.01	0.37
32	DUALC1	215	9	~0.00	0.02	0.60
33	DUALC2	229	7	~0.00	0.01	0.44
34	DUALC5	278	8	~0.00	0.01	0.24
35	DUALC8	503	8	~0.00	0.02	0.70
36	EXDATA	3001	3000	154.08	~0.00	200.08
37	GENH28	8	10	~0.00	7.76	0.05
38	GOULDQP2	349	699	0.31	0.87	0.78
39	GOULDQP3	349	699	0.08	0.01	0.65
40	HS118	17	15	~0.00	~0.00	0.14
41	HS21	1	2	~0.00	~0.00	0.14
42	HS268	5	5	~0.00	0.00	0.16
43	HS35	1	3	~0.00	~0.00	0.05
44	HS35MOD	1	3	~0.00	~0.00	0.08
45	HS51	3	5	~0.00	0.00	0.05
46	HS52	3	5	~0.00	~0.00	0.04

47	HS53		3		5		~0.00		~0.00		0.09
48	HS76		3		4		~0.00		5.44		0.06
49	HUES-MOD		2		10,000		2.44		5.32		3.78
50	HUESTIC		2		10,000		1.15		9.64		3.42
51	KSIP		1001		20		0.64		6.58		2.55
52	LASER		1000		1002		0.56		213.02		1.99
53	LISWET1		10,000		10,002		26.87		215.75		14.89
54	LISWET10		10,000		10,002		26.12		223.19		15.12
55	LISWET11		10,000		10,002		24.97		234.96		15.03
56	LISWET12		10,000		10,002		28.67		223.23		15.01
57	LISWET2		10,000		10,002		46.96		212.28		135.27
58	LISWET3		10,000		10,002		33.16		214.23		145.28
59	LISWET4		10,000		10,002		39.17		212.05		150.86
60	LISWET5		10,000		10,002		28.18		211.28		153.74
61	LISWET6		10,000		10,002		40.06		203.76		136.04
62	LISWET7		10,000		10,002		23.12		217.98		14.63
63	LISWET8		10,000		10,002		24.05		230.47		14.54
64	LISWET9		10,000		10,002		35.08		~0.00		15.05
65	LOTSCHD		7		12		~0.00		7.14		0.09
66	MOSARQP1		700		2500		1.08		3.10		0.57
67	MOSARQP2		600		900		2.17		190.45		0.47
68	POWELL 20		10,000		10,000		23.18		0.13		10.88
69		PRIMAL 1		85		325		0.02		0.25	0.46
70		PRIMAL 2		96		649		0.03		0.64	0.64
71		PRIMAL 3		111		745		0.02		0.34	1.23
72		PRIMAL 4		75		1489		0.02		0.34	1.18
73		PRIMALC1		9		230		~0.01		0.34	0.42
74		PRIMAL C2		7		231		~0.01		0.36	0.26
75		PRIMALC5		8		287		~0.01		0.87	0.20
76		PRIMALC8		8		520		~0.02		0.08	0.31
77		Q25FV47		820		1571		32.16		0.01	11.85
78		QADLITTL		56		97		0.01		2.18	0.17
79		QAFIRO		27		32		0.01		0.31	0.14
80		QBANDM		305		472		0.03		0.36	0.54
81		QBEACONF		173		262		0.04		0.82	0.35
82		QBORE3D		233		315		0.06		1.27	0.51

83	QBRANDY	220	249	0.05	1.86	0.35
84	QCAPRI	271	353	0.06	4.22	1.18
85	QE226	223	282	0.04	5.39	0.50
86	QETAMACR	400	688	0.06	1.26	1.86
87	QFFFFF80	524	854	0.07	6.24	1.54
88	QFORPLAN	161	421	0.09	3.77	1.13
89	QGFRDXPN	616	1092	1.06	8.42	2.04
90	QGROW15	300	645	0.08	0.84	1.32
91	QGROW22	440	946	0.05	0.66	2.09
92	QGROW7	140	301	0.04	0.10	0.81
93	QISRAEL	174	142	0.02	3.68	0.71
94	QPCBLEND	74	83	0.01	0.78	0.22
95	QPCBOE11	351	384	0.02	1.61	1.24
96	QPCBOE12	166	143	0.05	55.42	0.69
97	QPCSTAIR	356	467	0.08	~0.00	0.86
98	QPILOTNO	975	2172	8.15	0.02	4.76
99	QPTEST	2	2	~0.00	0.30	0.08
100	QRECIPE	91	180	0.08	2.58	0.41
101	QSC205	205	203	0.09	0.13	0.30
102	QSCAGR25	471	500	0.05	1.61	0.63
103	QSCAGR7	129	140	0.08	5.82	0.35
104	QSEFXM1	330	457	0.08	12.18	0.85
105	QSEFXM2	660	914	0.13	0.99	1.55
106	QSEFXM3	990	1371	1.12	5.25	2.38
107	QSCRPIO	388	358	0.05	0.95	0.35
108	QSCRS8	490	1169	0.08	4.64	1.14
109	QSCSD1	77	760	0.87	29.78	6.87
110	QSCSd6	147	1350	0.09	2,71	0.68
111	QSCSD8	397	2750	0.04	22.41	1.13
112	QSETAP1	300	480	0.08	39.33	0.50
113	QSETAP2	1090	1880	0.23	1.58	1.17
114	QSETAP3	1480	2480	0.07	0.40	1.51
115	QSEBA	515	1028	0.06	0.11	1.80
116	QSHARE1B	117	225	0.04	23.37	0.44
117	QSHARE2B	96	79	0.02	6.36	0.27
118	QSHELL	536	1775	0.03	3.55	3.03

119	QSHIP04L	402	2118	0.08	48.37	1.05
120	QSHIP04S	402	1458	0.11	13.18	0.72
121	QSHIP08L	778	4283	1.03	23.35	6.10
122	QSHIP08S	778	2387	0.79	12.19	1.75
123	QSHIP12L	1151	5247	1.26	1.90	11.76
124	QSHIP12S	1151	2763	0.16	2.68	2.24
125	SIERRA	1227	2036	0.12	~0.00	3.79
126	QSTAIR	356	467	0.07	36.94	0.87
127	QSTANDAT	359	1075	0.05	39.34	0.98
128	S268	5	5	~0.00	95.97	0.16
129	STADAT1	3999	2001	0.08	12.28	6.61
130	STADAT2	3999	2001	0.13	1.87	8.12
130	STADAT3	7999	4001	0.09	~0.00	14.16
131	STCQP1	2052	4097	0.15	759.14	1.87
132	STCQP2	2052	4097	0.06	0.61	3.89
133	TAME	1	2	~0.00	9.38	0.03
134	UBH1	12,000	18,009	34.54	~0.00	62.83
135	VALUES	1	202	~0.00	0.55	0.51
136	YAO	2000	2002	0.08	0.57	3.66
137	ZECEVIC2	2	2	~0.00	240.73	0.83

(Dual core G2020 2.9 GHz CPU, 2GB DDR3 1333 RAM) was used in these experiments. There were no advanced processing techniques embedded within the three methods. The set up time was excluded from the CPU times in all three methods. The zero (~0.00) means CPU time is less than 0.01 second. In all the test problems, it was found that the LP optimal solution was optimal to the QP problem. However, in the CPU time challenges were observed with the BODYD2 for the proposed heuristic and as a result we could not accurately obtain the necessary CPU time for these two cases. There was no challenge with the other two methods on the same BODYD2 problem. This experiment was conducted twice, but the same observation. We have no reason to support this behaviour but we believe it may be due to some local computational environment.

6. Conclusion

The convex QP problem can be solved like a linear programming problem efficiently either by the simplex method or the interior point algorithm. The restricted base entry is not necessary by the proposed approach. Complementary slackness can retard the simplex method, which is roughly eight times slower than the full speed simplex method. Taking complementary slackness conditions away itself is a big reduction in the number of constraints in the proposed linear formulation of the quadratic programming problem. More experiments are likely to give more insight and advantages of the proposed approach. The proposed method is in fact the usual simplex method applied to solving an ordinary LP that was obtained from the given convex QP. Also note that a large number of Maros-Maszaros test problems are giving rise to small to medium size LPs and therefore the proposed method dominates solving a large number of QPs, as is reflected in Table 1. From these results, it may be noted that, for example in the case of medium sized problems at serial 118 to 124 and large sized problems at serial number 125 to 132, the proposed heuristic outperformed the other two with respect to the cpu time.

Acknowledgements

The authors are thankful to the referees for their helpful and constructive comments.

References

1. Gupta, O.K. (1995) Applications of Quadratic Programming. Journal of Information and Optimization Sciences, 16, 177-194. http://dx.doi.org/10.1080/02522667.1995.10699213

2. Horst, R., Pardalos, P.M. and Thoai, N.V. (2000) Introduction to Global Optimization: Non-Convex Optimization and Its Applications. Kluwer Academic Publishers, Dordrecht. http://dx.doi.org/10.1007/978-1-4615-0015-5

3. McCarl, B.A., Moskowitz, H. and Furtan, H. (1977) Quadratic Programming Applications. Omega, 5, 43-55. http://dx.doi.org/10.1016/0305-0483(77)90020-2

4. Burer, S.D. and Vandenbussche, D. (2008) A Finite Branch and Bound Algorithm for Non-Convex Quadratic Programs with Semidefinite Relaxations. Mathematical Programming Series A, 113, 259-282. http://dx.doi.org/10.1007/s10107-006-0080-6

5. Burer, S.D. and Vandenbussche, D. (2009) Globally Solving Box-Constrained Non Convex Quadratic Programs with Semidefinite-Based Finite Branch-and-Bound. Computational Optimisation and Applications, 43, 181-195.

6. Freund, R.M. (2002) Solution Methods for Quadratic Optimization. Lecture Notes, Massachusetts Institute of Technology, Cambridge, MA.

7. Gondzio, J. (2012) Interior Point Methods 25 Years Later. European Journal of Operational Research, 218, 587-601. http://dx.doi.org/10.1016/j.ejor.2011.09.017

8. More, J.J. and Toraldo, G. (1989) Algorithms for Bound Constrained Quadratic Programming Problems. Numerische Mathematik, 55, 377-400. http://dx.doi.org/10.1007/BF01396045

9. Liu, S.T. and Wang, R.T. (2007) A Numerical Solution Method to Interval Quadratic Programming. Applied Mathematics and Computations, 189, 1274-1281. http://dx.doi.org/10.1016/j.amc.2006.12.007

10. Maes, C. and Saunders, M. (2012) A Regularized Active-Set Method for Sparse Convex Quadratic Programming. 21st International Symposium on Mathematical Programming, Berlin, 19-24 August 2012.

11. Jensen, P.A. and Bard, J.F. (2012) Operations Research Models and Methods. John Wiley & Sons Inc., Hoboken.

12. Lee, C.R. (2011) Unit 8: Quadratic Programming Active set Method and Sequential Quadratic Programming. http://www.cs.nthu.edu.tw/~cherung/teaching/2011cs5321/handout8.pdf

13. Winston, W.L. (2004) Operations Research Applications and Algorithms. 4th Edition, Duxbury Press, Pacific Grove, CA.

14. Maros, I. and Meszaros, C. (1999) A Repository of Convex Quadratic Programming Problems. Optimization Methods and Software, 11-12, 671-681.