No longer updated!

This page presents benchmarks on solvers with GAMS interface.

MIP

Date	Solvers	Models	Comment
18.05.2012	CBC 2.7, CPLEX 12.4.0.0, GUROBI 4.6.1, SCIP 2.1.1	MIPLIB 2010
28.02.2011	CBC 2.4, CPLEX 12.2.0.2, GUROBI 4.0.1, SCIP 2.0.1	LINLib
09.02.2010	CBC 2.4, CPLEX 12.1, GUROBI 2.0, SCIP 1.2	LINLib
18.01.2009	CPLEX 11.2, CBC 2.2, SCIP 1.1	LINLib
04.10.2008	CPLEX 11.1, CBC 2.2, SCIP 1.1, SCIP 1.0, GLPK 4.30	LINLib
14.09.2008	CPLEX 11.1, CBC 2.2, SCIP 1.0, GLPK 4.30	LINLib
25.08.2008	CBC 1.1, 1.2, 2.0, 2.1, 2.2	LINLib
19.07.2008	CBC 2.1	LINLib	one vs. two threads

NLP

Date	Solvers	Models	Comment
04.03.2011	CONOPT 3.14V, IPOPT 3.8, KNITRO 7.0.0, MINOS 5.51, PATH-NLP 4.7.02, SNOPT 7.2-4	GlobalLib
20.11.2009	IPOPT 3.7, CONOPT 3.14T, MINOS 5.51, SNOPT 7.2-4	GlobalLib
20.11.2009	IPOPT 3.7	GlobalLib	linear solver MA27 vs. MUMPS 4.8.3; GAMS version vs. locally compiled version
19.10.2008	IPOPT 3.5, KNITRO 5.1.2, CONOPT 3.14S, MINOS 5.51, SNOPT 7.2-4	GlobalLib
21.09.2008	IPOPT 3.5	GlobalLib	linear solver MA27 vs. MUMPS 4.8.3

LP

Date	Solvers	Models	Comment
21.01.2009	CPLEX 11.2, CLP 1.8, GLPK 4.35	LINLib
07.03.2011	CPLEX 12.2.0.2, CLP 1.11, GLPK 4.43, GUROBI 4.0.1, MOSEK 6.0.0.96	LINLib

MINLP

Date	Solvers	Models	Comment
12.11.2012	Baron 11.5.2, Couenne 0.4, Lindo API 7.0.1.497, SCIP 3.0.0	MINLPLib
13.09.2012	Baron 11.3.0, Couenne 0.4, Lindo API 7.0.1.497, SCIP 3.0.0	MINLPLib
14.09.2012	AlphaECP 2.09.02, Bonmin 1.6, DICOPT, Knitro 8.0.0, SBB, SCIP 3.0.0	convex MINLPs from IBM-CMU MINLP, MINLPLib, Günlück and Linderoth
25.02.2011	AlphaECP 1.75.04, Bonmin 1.4, DICOPT 2x-C, Knitro 7.0.0, SBB	convex MINLPs from Bonami, Kilinç, Linderoth
25.02.2011	Bonmin 1.4	convex MINLPs from Bonami, Kilinç, Linderoth	B-BB vs. B-ECP vs. B-Hyb vs. B-OA vs. B-QG
29.09.2008	Baron 8.1, LindoGlobal 5.0	MINLPLib
29.09.2008	Bonmin 0.99	MINLPLib	B-BB vs. B-Hyb vs. B-OA, only convex models
29.09.2008	Baron 8.1, LindoGlobal 5.0, AlphaECP 1.63, Bonmin 0.99, DICOPT 2x, SBB, OQNLP	MINLPLib	only convex models
29.09.2008	Baron 8.1, LindoGlobal 5.0, AlphaECP 1.63, Bonmin 0.99, DICOPT 2x, SBB, OQNLP	MINLPLib

Older benchmarks:

This page contains details about benchmarks of COIN-OR solvers with GAMS interface vs. other GAMS solvers.
Benchmarks were made by creating trace-files using the GAMS trace option and processing them with the PAVER server.

NEW:

25.08.2008: benchmarking CBC from various GAMS versions and current 2.2
19.07.2008: benchmarking CBC single threaded vs. multithreaded version (running 2 threads).
06.07.2008: updated benchmarks of MIP solvers and of GAMS BCH using a newer CBC version.
01.06.2008: benchmarking MIP solvers CPLEX, CBC, GLPK, SCIP on LINLib.
19.05.2008: benchmarking GAMS BCH-facility on the FCNetLib (CPLEX, SCIP, CBC).
28.01.2008: LP solvers 4 years ago and today (CLP, GLPK, IPOPT(today)).
07.01.2008: Comparision with updated MIP solvers (CPLEX 11, GLPK 4.25).
07.01.2008: Comparision with updated LP solvers (CPLEX 11, GLPK 4.24).
27.09.2007: Comparision of DICOPT stop rules 0 to 3
28.09.2007: Comparision of SBB with CONOPT and default and high memnodes
28.09.2007: Updated comparision of SBB, AlphaECP, DICOPT, BONMIN.
01.10.2007: Updated comparision of CPLEX, CLP, GLPK, IPOPT on LPs.

Content:

General Notes

Solvers are always run on each model with a timelimit of 1 hour and without iteration or domain violation limit.
The PerformanceTools scripts behind the PAVER server currently contain a bug that make PAVER omit the "locally optimal" row/column in the solver square summary for discrete models (MIP, MINLP). This also leads to wrong sum-ups in the "total"-row/column.
Resource time summary and performance profiles should be correct.
LINLib results were produced on a machine with Intel Core2 2.40 GHz, 2 GB RAM, and running under Linux 2.6.18.
MINLPLib results were produced on a machine with Intel Pentium IV 3 GHz, 1 GB RAM, and running under Linux 2.6.11.

Solvers of benchmarks from 02.09.2007

We used commercial solvers as they are included in GAMS 22.5 and open-source solvers as their were available on 21.8.2007. For results after 02.09.2007, the solver version is indicated together with the results, if different. In detail, the following solvers were used:

CPLEX 10.2.0

CBC and CLP

The following trunk revisions were used: CBC rev 770, CLP rev 1092, CGL rev 520, CoinUtils rev 837, OSI rev 1073

GAMS/CBC uses the OSI/CLP interface and CbcSolver routines which are also used by the CBC-standalone version. Additionally the following options are changed:

Dual and primal optimality tolerances are set to 1e-7.

GLPK 4.20

GAMS/GLPK uses the OSI/GLPK interface. MIPs are solved with GLPKs advanced Branch and Cut solver (lpx_intopt).
Additionally the following options are set or changed:

Dual and primal optimality tolerances are set to 1e-7.
Scaling is set to "geometric mean scaling, then equilibrium scaling".
The primal simplex algorithm is used.
LP presolve is turned on.
Textbook pricing is used.
Backtracking by best projection heuristic is used.
The integer tolerance is 1e-5.
All cutgenerators are turned on.

CONOPT 3.14r

KNITRO 5.1

IPOPT 3.3

The stable branch revision 1067 was used. Linear solver is MA27.
The GAMS/IPOPT interface changes the defaults of the following IPOPT options:

mu_strategy is set to adaptive
bound_relax_factor is set to 0.0

SBB = Simple Branch and Bound

Since we also deal with nonconvex models, we have set acceptnonopt to 1.

DICOPT = Discrete and Continuous Optimizer

DICOPT was run with standard options, which means that DICOPT stops if the NLP solution is worsening or the major iteration limit of 20 is reached. It will be interesting to see how things will change with a change in these options.

AlphaECP 1.30 = Extended Cutting Plane

BARON 7.8.1 = Branch And Reduce Optimization Navigator

BONMIN

The trunk revision 681 was used.

LaGO

The trunk revision 127 was used.

LP Benchmarks

We run on all LPs from the LINLib which have at least 50000 nonzeros (categories medium, large, and huge). This are 118 models.

02.09.2007:

results of the following comparision:
- CPLEX-0: CPLEX with default parameters (simplex algorithm).
- CPLEX-1: CPLEX with "lpmethod 4" (barrier algorithm).
- CBC-0: CLP with default parameters.
- GLPK-0: GLPK with default parameters (see above).
- IPOPT-0: IPOPT with default parameters (see above).
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.
results of the following comparision (as above, but less solvers):
- CPLEX-0: CPLEX with default parameters (simplex algorithm).
- CBC-0: CLP with default parameters.
- GLPK-0: GLPK with default parameters (see above).
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.
results of the following comparision:
- IPOPT-0: IPOPT with default parameters (see above).
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.

01.10.2007:

results of the following comparision:
- CPLEX-0: CPLEX with default parameters (simplex algorithm).
- CBC-0: CLP with default parameters.
- GLPK-0: GLPK 4.22 with default parameters (see above), but steepest depth pivot choice algorithm.
- GLPK-4.20: GLPK 4.20 with default parameters (see above).
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.

07.01.2008:

results of the following comparision:
- 1-CPLEX-0: CPLEX 11 with default parameters (simplex algorithm).
- 2-CPLEX-1: CPLEX 11 with "lpmethod 4" (barrier algorithm).
- 3-CPLEX-0: CPLEX 10.2 with default parameters (simplex algorithm).
- CBC-0: CLP with default parameters.
- GLPK-0: GLPK 4.22 with default parameters (see above), but steepest depth pivot choice algorithm.
- GLPK-4.20: GLPK 4.20 with default parameters (see above).
- IPOPT-0: IPOPT with default parameters.
- IPOPT-1: IPOPT with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.

28.01.2008:

results of the following comparision:
- CBC2004: CLP from GAMS 21.4 (2004) with default parameters
- CBC2008: CLP 2.0 (2008) with default parameters
- GLPK2004: GLPK 4.5 from GAMS 21.4 (2004) with default parameters
- GLPK2008: GLPK 4.25 (2008) with default parameters
- IPOPT2008: IPOPT (2008) with mehrotra_algorithm, jac_c_constant, jac_d_constant, hessian_constant turned on.

MIP Benchmarks

We run on all MIPs from the LINLib. This are 125 models.

25.08.2008:

results of the following comparision on 110 mip models from the LINLib running
- 22.4.trc: CBC 1.1 from GAMS 22.4
- 22.5.trc: CBC 1.2pre from GAMS 22.5
- 22.7.trc: CBC 2.0 from GAMS 22.7
- 22.8.trc: CBC 2.1 from GAMS 22.8
- 22.9.trc: CBC 2.2 from 21.08.2008 (CBC stable/2.2 rev. 1050; CLP stable/1.8 rev. 1262; Cgl stable/0.53 rev. 676; Osi stable/0.99 rev. 1288; CoinUtils stable/2.3 rev. 1070; compiled with flags "-DDISALLOW_PRINTING -DCLP_FAST_CODE -DCOIN_FAST_CODE -DCOIN_USE_RESTRICT -DNEW_STYLE_SOLVER=0")
time limit = 1800s
gap tolerance = 0%
default solver parameters
machine: Core2 Duo T7500, 2 GB RAM, Linux 2.6.22, GCC 4.2.1

19.07.2008:

results of the following comparision on the 125 mip models from the LINLib running
- CBC-1: Cbc 2.1 (stable version from approx. 24.06.08) compiled in singlethreading mode
- CBC-2: Cbc 2.1 (stable version rev. 1004 from approx. 15.07.08) compiled in multithreading mode and run with 2 threads
gap tolerance = 0%
default solver parameters
machine: Core2 Duo T7500, 2 GB RAM, Linux 2.6.22, GCC 4.2.1

06.07.2008:

results of the following comparision on the 125 mip models from the LINLib:
- CPLEX 11.0 (as in GAMS 22.6)
- SCIP 1.0
- CBC 2.1 (stable version from approx. 24.06.08)
- GLPK 4.26
gap tolerance = 0%
default solver parameters
machine: Core2 Duo T7500, 2 GB RAM, Linux 2.4.16, GCC 4.2.1
results of benchmarking MIP solvers on the GAMS FCNetLib (Fixed Cost Network Flow Library) analysing the effect of dicuts computed via the GAMS BCH facility. The following solvers were used:
- CPLEX 11.0 (as in GAMS 22.6)
- SCIP 1.0
- CBC 2.1 (stable/2.1 from approx. 24.06.08)

01.06.2008:

results of the following comparision:
- CPLEX 11.0 (as in GAMS 22.6)
- SCIP 1.0
- CBC 2.1 (stable version from approx. 25.05.08)
- GLPK 4.26
gap tolerance = 0%
default solver parameters
machine: Core2 Duo T7500, 2 GB RAM, Linux 2.4.16, GCC 4.2.1

19.05.2008:

results of benchmarking MIP solvers on the GAMS FCNetLib (Fixed Cost Network Flow Library) analysing the effect of dicuts computed via the GAMS BCH facility. The following solvers were used:
- CPLEX 11.0 (as in GAMS 22.6)
- SCIP 1.0
- CBC 2.1 (stable/2.1 revision 946)

07.01.2008:

results of the following comparision (we run with a gap tolerance of 0.01%):
- 1-CPLEX-1: CPLEX 11 with default parameters.
- 2-CPLEX-1: CPLEX 10.2 with default parameters.
- CBC-1: CBC 2.0 with default parameters.
- GLPK-1: GLPK 4.25 with default parameters (see above).

02.09.2007:

results of the following comparision (we run with a gap tolerance of 0.01%, except for GLPK which does not have a MIP gap tolerance yet):
- CPLEX-1: CPLEX with default parameters.
- CBC-1: CBC with default parameters.
- GLPK-1: GLPK with default parameters (see above).

NLP Benchmarks

We run on all NLPs from the GlobalLib. This are 379 models.

02.09.2007:

results of the following comparision:
- CONOPT-1: CONOPT with parameters rtredg=1e-6, rtnwmi=1e-6, and rtnwma=1.01e-6.
- IPOPT-1: IPOPT with parameters tol=1e-6 and acceptable_tol=1e-6.
- KNITRO-1: KNITRO with parameters feastol=1e-6 and opttol=1e-6.

Next we have run all QQPs from the GlobalLib with atmost 1000 variables. This are 162 models.
The gap tolerance is set to 1%, the feasibility tolerance to 1e-4.

02.09.2007:

results of the following comparision:
- BARON-1: BARON.
- LaGO-1: LaGO with violation subdivision.

MINLP Benchmarks

We run on a selected set of MINLPs from the MINLPLib. The selection aims to not overrepresent a type of model that one solver might be very good at (e.g., AlphaECP is very good on the models fo{7,8,9}*, m7*, no7*, and o{7,8,9}*, so we took only some of them). Also we limited to models with atmost 1000 variables. This gives a set of 167 models.
We run with a gap tolerance of 1%.

02.09.2007:

Solver A vs. Solver B:

results of the following comparision on all 161 models without semicontinuous variables and SOS constraints (since these are not supported by GAMS/Bonmin yet):
- SBB-1: SBB with CONOPT as subsolver.
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- DICOPT-1: DICOPT with CONOPT and CPLEX as subsolver.
- BONMIN-1: BONMIN.

results of the following comparision on all 40 convex models:
- SBB-1: SBB with CONOPT as subsolver.
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- DICOPT-1: DICOPT with CONOPT and CPLEX as subsolver.
- BONMIN-1: BONMIN.

results of the following comparision on all 149 models which BARON can handle (not SOS, no semicontinuous variables, no functions like sin, cos, ...):
- BARON
- LaGO-1: LaGO with violation subdivision and "unfixed discrete" as node selection type.
- BONMIN-1: BONMIN.

results of the following comparision on all 40 convex models (no SOS, semicontinuous variables, or functions like sin, cos, ...):
- BARON
- LaGO-1: LaGO with violation subdivision and "unfixed discrete" as node selection type.
- BONMIN-1: BONMIN.

results of the following comparision on all 149 models which BARON can handle (not SOS, no semicontinuous variables, no functions like sin, cos, ...):
- BARON
- LaGO-1: LaGO with violation subdivision and "unfixed discrete" as node selection type.

results of the following comparision on all 40 convex models (no SOS, semicontinuous variables, or functions like sin, cos, ...):
- BARON
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- BONMIN-1: BONMIN.
- DICOPT-1: DICOPT with CONOPT and CPLEX as subsolver.
- LaGO-1: LaGO with violation subdivision and "unfixed discrete" as node selection type.
- SBB-1: SBB with CONOPT as subsolver.

27.09.2007:

results of the following comparision on all 167 models:
- DICOPT-4: DICOPT with CONOPT and CPLEX as subsolvers and options maxcycles 10000 and stop 0
- DICOPT-5: DICOPT with CONOPT and CPLEX as subsolvers and options maxcycles 10000 and stop 1
- DICOPT-6: DICOPT with CONOPT and CPLEX as subsolvers and options maxcycles 10000 and stop 2
- DICOPT-7: DICOPT with CONOPT and CPLEX as subsolvers and options maxcycles 10000 and stop 3

28.09.2007:

results of the following comparision on all 167 models:
- SBB-1: SBB with CONOPT
- SBB-7: SBB with CONOPT and memnodes 100000

results of the following comparision on all 161 models without semicontinuous variables and SOS constraints (since these are not supported by GAMS/Bonmin yet):
- SBB-7: SBB with CONOPT as subsolver and memnodes 10000.
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- DICOPT-5: DICOPT with CONOPT and CPLEX as subsolver and options maxcycles 10000 and stop 1 (stop on crossover)
- BONMIN-1: BONMIN.

results of the following comparision on all 40 convex models:
- SBB-7: SBB with CONOPT as subsolver and memnodes 10000.
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- DICOPT-5: DICOPT with CONOPT and CPLEX as subsolver and options maxcycles 10000 and stop 1 (stop on crossover)
- BONMIN-1: BONMIN.

02.09.2007:

Influence of NLP subsolver on SBB performance:

results of the following comparision:
- SBB-1: SBB with CONOPT as subsolver. CONOPT option Rtmaxv is set to 1e+20.
- SBB-2: SBB with IPOPT as subsolver and IPOPT option expect_infeasible_problem set to yes.
- SBB-3: SBB with IPOPT as subsolver and IPOPT options expect_infeasible_problem and warm_start_init_point set to yes.
- SBB-4: SBB with CONOPT and IPOPT in a infeasseq as subsolver: If CONOPT reports a subproblem as infeasible and we are at atmost depth 50 of the Branch-and-Bound tree, SBB calls IPOPT. IPOPT option expect_infeasible_problem is set to yes.
- SBB-5: SBB with IPOPT as subsolver. IPOPT option expect_infeasible_problem set to yes and print_level set to 0.

results of the following comparision on all 40 convex models:
- SBB-1: SBB with CONOPT as subsolver. CONOPT option Rtmaxv is set to 1e+20.
- SBB-2: SBB with IPOPT as subsolver and IPOPT option expect_infeasible_problem set to yes.
- SBB-3: SBB with IPOPT as subsolver and IPOPT options expect_infeasible_problem and warm_start_init_point set to yes.
- SBB-4: SBB with CONOPT and IPOPT in a infeasseq as subsolver: If CONOPT reports a subproblem as infeasible and we are at atmost depth 50 of the Branch-and-Bound tree, SBB calls IPOPT. IPOPT option expect_infeasible_problem is set to yes.

results of the following comparision:
- SBB-1: SBB with CONOPT as subsolver. CONOPT option Rtmaxv is set to 1e+20.
- SBB-5: SBB with IPOPT as subsolver. IPOPT option expect_infeasible_problem set to yes and print_level set to 0.
- SBB-4: SBB with CONOPT and IPOPT in a infeasseq as subsolver: If CONOPT reports a subproblem as infeasible and we are at atmost depth 50 of the Branch-and-Bound tree, SBB calls IPOPT. IPOPT option expect_infeasible_problem is set to yes.

results of the following comparision on all 40 convex models::
- SBB-1: SBB with CONOPT as subsolver. CONOPT option Rtmaxv is set to 1e+20.
- SBB-5: SBB with IPOPT as subsolver. IPOPT option expect_infeasible_problem set to yes and print_level set to 0.
- SBB-4: SBB with CONOPT and IPOPT in a infeasseq as subsolver: If CONOPT reports a subproblem as infeasible and we are at atmost depth 50 of the Branch-and-Bound tree, SBB calls IPOPT. IPOPT option expect_infeasible_problem is set to yes.

Influence of NLP and MIP subsolver on DICOPT performance:

results of the following comparision:
- DICOPT-1: DICOPT with CPLEX and CONOPT as subsolver.
- DICOPT-2: DICOPT with CPLEX and IPOPT as subsolver.

results of the following comparision:
- DICOPT-1: DICOPT with CPLEX and CONOPT as subsolver.
- DICOPT-3: DICOPT with CBC and CONOPT as subsolver.

Influence of MIP subsolver on AlphaECP performance:

results of the following comparision:
- AlphaECP-1: AlphaECP with CPLEX as subsolver.
- AlphaECP-2: AlphaECP with CBC as subsolver.