The Traveling Salesman Problem in Theory & Practice

The Traveling Salesman Problem in Theory & Practice

Lecture 11: Branch & Cut & Concorde8 April 2014

David S. [email protected]

http://davidsjohnson.net

Seeley Mudd 523, Tuesdays and Fridays

mailto:[email protected]

http://davidsjohnson.net/

http://davidsjohnson.net/

Outline1. Cutting Planes2. Branch and Bound3. Performance of Concorde4. Student Presentation by Chun Ye:“Improving Christofides’ Algorithm for the Metric s-t

Path TSP”

Presentation strongly reliant on [ABCC06]

Branch & CutCombine Branch-&-Bound with Cutting Planes

• The term “branch & cut” was coined by Manfred Padberg and Giovanni Rinaldi in their paper [Padberg & Rinaldi, “Optimization of a 532-city traveling salesman problem by branch-and-cut,” Operations Res. Lett. 6 (1987), 1-7.

• First known use of the approach was in Saman Hong’s 1972 PhD thesis A Linear Programming Approach for the Traveling Salesman Problem at Johns Hopkins University.– Although, handicapped by his LP solver, he could only solve 20-city

problems. This was not all that impressive, since Dantzig et al. had already done 42 cities in 1954, and Held & Karp did 64 in 1971.

– Even worse: Our permutation based B&B approach, using NN + iteration, solves such instances in 0.01 seconds with just 165 lines of unsophisticated code.

– Machines were a bit slower in 1971 than our 3.5Ghz processor, but– A 1968 era PDP 10 was probably about a 0.5Mhz machine (1 μs cycle time,

2.1 μs per addition), so roughly 7000 time slower.– Our code would thus still have taken only 70 seconds on such a machine.– Moral : Ideas can be much more important than (initial) performance.

Refresher: The Cutting Plane ApproachSolve the edge-based integer programming formulation for the TSP, as follows:1. Start by solving a weak linear programming

relaxation.2. While the LP solution is not a tour,

a. Identify a valid inequality that holds for all tours but not the current solution (a “cutting plane” or “cut” for short) .

b. Add it to the formulation and re-solve.

Assume edge lengths are integers, and we have an algorithm ALB that computes a lower bound on the TSP length when certain constraints are satisfied, such as a set of edges being “fixed” (forced to be in the tour, or forced not to be in the tour), and which, for some subproblems, may produce a tour as well.• Start with an initial heuristic-created “champion” tour TUB, an upper bound UB = length(TUB)

on the optimal tour length, and a single “live” subproblem in which no edge is fixed. • While there is a live subproblem, pick one, say subproblem P, and apply algorithm ALB to it.

– If LB > UB-1, delete subproblem P and all its ancestors that no longer have live children. No improved tour is possible in this case (since tour length is an integer).

– Otherwise, we have LB ≤ UB-1.1. Pick an edge e that is unfixed in P and create two new subproblems as its children, one with e

forced to be in the tour, and one with e forced not to be in the tour.2. If algorithm ALB produced a tour T, and length(T) < UB

1. Set UB = length(T) and TUB = T.2. Delete all subproblems with current LB > UB-1, as well as their children and their

ancestors that no longer have any live children.• Halt. Our current champion is an optimal tour.

Refresher: Branch & Bound for the TSP

Some Key Implementation Issues for Branch & Cut

• How do we find the initial tour (upper bound)?• How do we find violated inequalities (cuts)?• What are the best inequalities to add?

– Based on speed of generating them– Based on effectiveness in improving lower bounds

• How do we decide when to split a case?• How do we choose the variable on which to split a case?• How do we pick the next subcase to work on?---------------------------------------------------------------------• How do we manage the inequalities?

– Solving the LP may take too long if there are too many inequalities– Some inequalities may lose their effectiveness in later subcases

• How do we cope with repeatedly solving LPs with millions of variables?

• What LP code do we use and how do we apply it?

Chapter 15, 64 pages

Chapters 5-11, 216 pages

1 page, somewhere:

Split when we (1) Can’t find any more cuts, or (2) reach point of diminishing returns.


Chapter 14, 1 paragraph: Subproblem with smallest lower bound.



Coverage by [ABCC06]

Finding Cuts: The Template Approach

Look for cuts (preferably facet-inducing) with structures from a (prioritized) predefined list.• Subtour Cuts• Combs• Clique-Tree Inequalities• Path Inequalities• …

For each cut class, can have an ordered sequence of stronger and stronger (possibly slower and slower) cut-finding heuristics, up to exact algorithms, should they exist.(These heuristics typically assume that our current LP solution satisfies all the degree-2 constraints.)

Heuristics for finding violatedsubtour constraints

Connected Component Test• Construct a graph G, where e is an edge if variable xe > 0.• Compute the connected components of G (running time linear in the number of e

with xe > 0).• If the graph is not connected, we get a subtour cut for each connected component.• Solve the resulting LP, and repeat the above if the resulting graph is still not

connected.

The cuts found in this way already get us very close to the Held-Karp bound.For random Euclidean instances with coordinates in [0,106], ABCC06 got• Within 0.409% of the HK bound for an instance with N = 1,000, and• Within 0.394% of the HK bound for an instance with N = 100,000.But note that the connected graph may only be connected because of edges with very low values for xe. To deal with this, one can use a “parametric connectivity” test.


Parametric Connectivity Test• Start with a graph G with no edges, and with every city c being a connected component of size 1

with weight(c) = 0.• For all edges e = {u,v} of our TSP instance, in non-increasing order by the value of xe.

– Find the the connected components Su and Sv containing u and v (using the “union-find” data structure).

– If Su = Sv, increase weight(Su) by xe.– Otherwise,

• Set Su = Su U Sv, and increase weight(Su) by weight(Sv) + xe.• If now δx(Su) = 2|Su| - 2weight(Su) < 2, add the corresponding subtour cut and continue.

– If there are now just two connected components, quit the loop (which took time O(mlogN), where m is the number of e with xe > 0).

• Solve the resulting LP.Repeat the above until the process yields no new cuts.

1,000 Cities 100,000 CitiesConnectivity - 0.409% - 0.394%

+ Parametric Connectivity - 0.005% - 0.029%


Interval Test• Let (c0, c1, …, cN-1) be the current champion tour in order. Note that every set

consisting of a subinterval (ci,ci+1,…,ck) of this order induces a subtour constraint with S = {ci,ci+1,…,ck}.

• For each i, 0 ≤ i < N-1, consider the set {ci,…ct} for the t that minimizes δx({ci,…ct}). If this minimum is less than 2, we have a subtour cut, for a total of as many as N-2 cuts. Add them all.

• With the right algorithms and data structures, this can be done in O(mlogN) time, where m is the number of edges e with xe > 0.

1,000 Cities 100,000 Cities

Connectivity - 0.409% - 0.394%

+ Parametric Connectivity - 0.005% - 0.029%

+ Interval Test = HK - 0.0008%

General Speedup Trick: Safe Shrinking

Exploiting the “shrinking” of a set of cities S, given a current LP solution x:• Replace S by a single city σ and x by a new function x’, where for all

distinct c,c’ in C-S, x’(u,v) = x(u,v), and for all c in C-S,x’(c,σ) = ∑v∈Sx(c,v).

• Find cut in the shrunken graph, then unshrink it back to a cut in the original graph.

• A set S is “safe” for shrinking if, whenever there is a violated TSP cut for x, then there is also one for x’.

• It is “template safe” for a given type of cut (subtour, comb, etc.) if, whenever there is a violated cut in the given template for x, then there is also one for x’.

• A natural candidate: Edges e with xe = 1.• Unfortunately, not always safe…

Unsafe Edges1.0

1.0

1.0

0.5

0.5

0.50.5

0.5

0.5

0.5

1.0

1.0

0.5

0.50.50.5

0.5

Violated Comb

0.5

0.5

0.5

0.5

0.5 0.5

0.5

0.5

0.50.5

Convex Combination of Two Tours

Edge Safety Theorem[Padberg & Rinaldi, “Facet identification for the symmetric traveling

salesman problem,” Math. Programming 47 (1990), 219-257]

Theorem: If x(u,v) = 1 and there exists a vertex w withx(w,u) + x(w,v) = 1,

in the solution to the current LP, then it is safe to shrink edge e = {u,v}.Proof: Suppose that x’ has no violated TSP cuts. Then it must be a convex combination of tours through the shrunken graph.Lemma: If x is a convex combination of tours in a shrunken (or unshrunken) graph, and we have x(e) = 1, then every tour in the convex combination contains edge e.Proof of Lemma: The tours containing e must have coefficients summing to 1 in the combination, meaning no other tours can have coefficients greater than 0, and hence none are part of the combination. QEDLet σ be the result of the merging of u and v. By our hypothesis, we have x’(w,σ) = 1, so every tour T in our convex combination of optimal tours uses edge {w,σ}. For each such T, let αT be its multiplier in the convex combination.

Edge Safety TheoremProof Continued

If we restrict attention to the edges e of these tours that do not involve σ, and hence have xe = x’e, then we have that the same convex combination precisely sums to xe for all such e.Consider a particular tour T in the combination, and let z1 be the city (other than w) that is adjacent to σ in the tour, as in the figure below (where at least one of the two edges {z1,u} and {z1,v} must be present in the original graph).

Because of the tour multiplier, we must have x’{z1,σ} ≥ αT and x’{z1,σ} ≥ αT. Thus in the original graph, the min cut between w and z1 in the graph induced by those two cities and {u,v} must be at least αT and there must be a flow of size αT between w and z1.This flow must be partitioned among the four paths (w,u,z1), (w,v,z1), (w,u,v,z1), and (w,v,u,z1). So the tour T in the shrunken graph can be replaced by 4 (or fewer) tours in the original graph, with multipliers summing to αT.This holds for all tours in the convex combination for the shrunken graph, perhaps involving other tour neighbors zi of σ. We thus get a convex combination of tours. I claim that the union of all these tours in the original graph is a convex combination of optimal tours with edge weights matching those under x.

σ

w

αT

…z1 zk

u

w

αT vβ 1-β

…zkz1

Original Graph

ShrunkenGraph

Edge Safety TheoremProof Continued

By the previous argument, we have a convex combination of tours in the original graph whose value ye for any edge e satisfies ye ≤ xe.I claim that in fact we must have ye = xe for edges e. For note that, since x is the solution to a valid LP, we have

∑exed(e) ≤ OPT.So if for any e we have ye < xe, we would have a convex combination y of tours with

∑eyed(e) < ∑exed(e) ≤ OPT,and so at least one tour of length less than OPT, a contradiction.Thus x must be a convex combination of tours, as desired. QED

Note that this theorem can be applied sequentially, leading to the merging of many edges, and in particular long paths. This can greatly reduce the size of the graph and consequently speed up our cut finding heuristics and algorithms.

Untangling Convex CombinationsThe previous discussion reminds us of the problem of degeneracy, where the final LP (no more TSP cuts possible) still contains fractional values, and so represents a convex combination of tours rather than a single tour.This is not a problem if our champion tour already has length equal to the LP solution value, but what if not?[ABCC06] doesn’t seem to address this issue, so it probably is a rare occurrence.The likely approach is to use heuristics, and exploit the fact that the graph is probably now very sparse and can be made much smaller.• Only e such that xe > 0 can be in a tour.

• All edges e with xe = 1 must be in an optimal tour, and so maximal paths of such edges can be collapsed into a single forced edge representing that path.

Such an approach can also be used even before the we have an unimprovable LP, as a way to potentially find new champion tours.

Managing and Solving the LP’s:Core Sets

Problem: Our LP’s potentially involve billions of variables.Solution: “Core sets”• We observe that only a relatively small number of edges typically get non-

zero values when solving our LP’s.• If we knew which ones they would be, we could simply eliminate them

from the formulation by fixing their values at 0.• Since we do not know them in advance, Concorde uses a simple heuristic

to define a “core set” of edges that are allowed non-zero values.• Standard possibilities:

– The edges in some collection of good tours.– The edges to the k-nearest neighbors for each city for some k.– A combination of the two.

• Concorde uses the union of the edges occurring in the tours resulting from 10 runs of Chained Lin-Kernighan for its initial core set.

• Thereafter, it will delete an edge e from the core set if for some constant L1 consecutive LP solves its value remains below a tolerance e1. Typical values are L1 = 200 and e1 = 0.0001.

Managing and Solving the LP’s:Adding edges to the Core Set

• In solving the LPs, Concorde computes both a primal and a dual solution.

• Consequently, it has access to reduced costs cj - yTAj for all the edges, including those not in the core.

• Edges with negative reduced costs are candidates for addition to the core (just as non-basic core variables with negative reduced costs are candidates for the entering variable in a step of the primal simplex algorithm).

• These are added to a queue, and every so often, the 100 with the most negative reduced costs are added to the LP. It is then re-solved and the new reduced costs for the remaining edges in the queue are computed. Any edge with reduced cost greater than some e2 < 0 is removed from the queue. Concorde uses e2 = - 0.0001.

• One difficulty: For very large instances, computing the reduced costs for all non-core edges can be very expensive because there are so many of them ( ~500 billion for N = 106). Concorde’s solutions:– Heuristics for approximating the reduced costs quickly (followed by exact pricing of the good

candidates).– Only price a fraction of the non-core edges each time, cycling through all of them over many iterations.– Alternate this with cycling through just those non-core edges that are within the 50 nearest neighbors of

some city.– For geometric instances, the list of edges to consider can be substantially pruned based on the city

coordinates and values from the current dual solution.

Managing Cuts• Start with just the degree-2 constraints.

– Note that we do not need an integer solution (as I described last time, using b-matching).– A fractional solution, with edge values in {0,½,1} will suffice for getting us started.– This can be accomplished without solving the LP, via a primal-dual algorithm.

• Subsequently, when a cut is found by a separation routine, it is appended to the end of a queue of cuts waiting to be added to the core LP. If the queue is small, we may call these routines many times, thus adding many cuts to the queue.

• When the cut-finding process stops, we repeated the following process until the queue is empty or we have added 250 cuts to the core LP:– Take the first cut from the queue– Check to confirm that it is still violated by the current x by at least some small tolerance

(say 0.002). This is needed since the cut may have been found for some earlier value of x.– If the cut is still valid in this sense, add it to the core LP. Otherwise, discard it.

• Cuts are deleted if the dual variables for them are less than some fixed tolerance (say 0.001) for 10 consecutive LP solves.

Storing Cuts• Problem: It is not necessarily efficient to store cuts as an actual

inequality on variables (or, worse yet, a vector of length |C|), since this can be a very inefficient use of space.

• 81 Gbytes for TSPLIB instance pla7397 in vector form.• This can be reduced to 96 Mbytes by more efficient representations.

– Lists of sets– Variable-length codes to point to sets (based on their frequency of occurrence)– Intervals represented by their endpoints, or better yet, their first endpoint and the

interval length– Etc.

• We need to do some computation to decode the representations (and their effect on the core set of variables) but this is a worthwhile tradeoff.

• Also, it pays to choose, among the many equivalent representations, the one that leads to the fewest non-zeros in the core-set LP formulation.– For instance, one can choose to represent a subtour inequality by either S or C-S, and the

smaller of those two sets is likely to be better in this regard.

Solving the LP’s• Use the Dual Steepest Edge variant of the Simplex algorithm.• Used the CPLEX package when [ABCC06] was being written.• Current Concorde package includes its own LP solving package

(QSopt), tailored for solving the kinds of LP’s encountered in Concorde.

• I am omitting loads of details.• (as I have in all the other Concorde-related issues I have

discussed).

• One detail that should be discussed: Round-off error and valid bounds.

Coping with Round-Off ErrorCommercial Linear Programming codes use floating point arithmetic.Their arithmetic routines presumably meet the IEEE standard, but this still means that one can only report results to within some fixed tolerance.Given this, our LP solutions only generate imprecise lower bounds.And may be in error since the solution may violate some of the LP cut constraints if it is very close to the bound of the cut, and merely setting tolerances may not suffice.One frequently used approach: Use exact-arithmetic LP codes (or exact-arithmetic hand computations in the case of [DFJ54]). Unfortunately, this is very slow.Concorde’s approach:• Start with the solution to the dual (which it already computes).• In the exact arithmetic world, this equals the primal opt, and any feasible solution

to the dual is a lower bound on the primal opt.• Find a fixed-precision feasible solution to the dual by exploiting the fact that all

the dual constraints have unique slack variables. (In Concorde’s case that precision is 32 bits each to the right and left of the decimal point).

• This remains quite close to the floating point optimal.

BranchingActually there are two types of branching in Concorde.1. Edge Branching (xe = 1 or xe = 0).2. Subtour Branching [Clochard and Naddef, “Using path inequalities in a branch and

cut code for the symmetric traveling salesman problem,” Third IPCO Conference (1993), 291-311].

For some subset S of cities that does not yield a violated subtour inequality, break into cases depending on whether δx(S) ≤ 2 or δx(S) ≥ 4.

This covers all possibilities, since in a tour we cannot have an odd value for δx(S).

Before we split the root subproblem, we first fix as many of the edge values at 0 or 1 as possible, using the reduced costs λe of the edges in the final LP and the “integrality gap” Δ between the length Length(T*) of our current best tour and the LP solution:

xe = 1 if λe < -(Δ-1) and xe = 0 if λe > (Δ-1).(In each case, fixing the variable to the other choice from {0,1} would cause the LB to grow larger than Length(T*) – 1, allowing us to prune the subproblem.)

Choosing the SplitEdge Candidates:For each fractional variable xe, estimate the change (z0 or z1) in LP objective value that would result from setting xe to either 0 or 1 in our current LP and making a single dual Simplex pivot.Rank the choices by the formula

p(z0, z1, γ) = (γ∙minimum(z0, z1) + maximum(z0, z1))/(γ+1),with γ = 10, saving the top 5 as candidates.(It is more important to improve the smaller bound than the larger.)Subtour Candidates:For each of the 3500 sets S involved in our cuts that have δx(S) closest to 3, rank each choice in an analogous way and save the top 25 candidates.Get another 25 candidates from a separate, more combinatorial approach.

Ranking the Candidates: Strong Branching

Each candidate consists of a pair of constraint sets which produce a disjoint partition of the possibilities, either• (xe = 1, xe = 0), or • (δx(S) ≤ 2, δx(S) ≥ 4)Denote these pairs by (P0

1, P11), (P0

2, P12), …, (P0

k, P1k).

For each Pij, add the corresponding constraints to the LP, and starting with the current

basic solution, perform a fixed (limited) number of dual-steepest-edge Simplex pivots and let zi

j be the resulting dual objective value. (If the problem is infeasible, set zij =

length(T*)).Pick the candidate that maximizes p(z0

j, z1j, 100).

For the most difficult instances, we can take this one step further (“tentative branching”), whose added cost may be justified by the need for fewer subproblems.• Take the top h candidates according to the above ranking.• For each use the full cutting plane approach to get a lower bound żi

j.

• Then rank these by p(ż0j, ż1

j, 10) and take the best.

Branching PerformanceStart with a root LP with value 0.00009% below the optimal tour length. (Tour found by Held Kelsgaun using a variant of his “LKH” algorithm. More to come about how the root LP was built).

Computations performed on a network of 250 2.6 Ghz AMD Opteron processors.

The CPU times are the sum of the times used on the individual processors. pla85900

# of Subproblems CPU Time

Strong Branching >> 3,000 >> 4 years

Tentative Branching, h = 4 1,149 1.8 years

Tentative Branching, h = 32 243 0.8 years

More on pla85900’s Solution

More on pla85900’s SolutionNot solved with vanilla Concorde. Instead:• Perform an initial run (without branching) to drive up the root solution.

Result: bound 0.0802% below optimal, 22.6 CPU days.• Perform a run with an upper bound of 142,320,000 (below optimal) and

starting with cuts from previous run.Result: bound 0.0456% below optimal, 144.0 CPU days.

• Perform a run with an upper bound of 142,340,000 (still below opt), and starting with the cuts from previous runs.Result: bound 0.0395% below optimal, 227.3 CPU days.

• Perform a run with the true upper bound, starting with the cuts from all the previous runs and running until there were 1000 active search nodes.Result: bound 0.0324% below optimal, 492.8 CPU days.

• Repeat previous step, but including the new cuts it generated. Result: bound 0.0304% below optimal, 740.0 CPU days.

• Over the course of a year, repeatedly apply all of Concorde’s separation routines to further drive up the root bound.

Result: bound 0.00087% below optimal, lots of CPU days…

More on pla85900’s SolutionType NumberSubtours 1,030Combs 2,787Paths and Stars 4,048Bipartition 41Domino Parity 164Others (Local Cuts) 809

Cutting Planes in the pla85900 root LP

With this last root LP, Concorde finally solved pla85900 in just 2,719.5 CPU days

Adding the additional cuts found to the root LP led to the even better root gap of 0.00009%

and the results quoted 3 slides back.

Vanilla ConcordeRandom Euclidean Instances

NSamples ABCC06

2d

Mean ABCC06

2d

Samples iMac14

2d

MeaniMac14

2d

Samples iMac14

3d

Mean iMac14

3d

Samples iMac14

7d

Mean iMac14

7d

100 10,000 0.7

200 10,000 3.6

300 10,000 10.6

400 10,000 25.9

500 10,000 50.2 1,000 20.9 1,000 6.6 1,000 2.6

600 10,000 92.0 1,000 35.2 1,000 9.8 1,000 4.1

700 10,000 154.5 1,000 52.3 1,000 13.2 1,000 5.6

800 10,000 250.4 1,000 81.7 1,000 17.4 1,000 6.6

900 10,000 384.7 1,000 116.0 1,000 21.0 1,000 8.1

1000 10,000 601.6 277 158.6 1,000 29.4 1,000 9.1

1500 1,000 3,290.9 1,000 61.3 1,000 10.6

2000 1,000 14,065.6 1,000 27.1

2500 1,000 53,737.9 100 45.7

7d requires a version of Concorde modified for higher fixed-precision.(Modified by Jeffrey Stoltz with hints from Bill Cook.)

Next TimeMore properties of Random Euclidean Instances

(as revealed by Concorde)

Documents

The Traveling Salesman Problem in Theory & Practice