Competitor Ordering
The following illustrates how the 'Finalists' for each track will be chosen.
Let m be the total number of early and late instances and k be the number of participants that produce a solution for all m instances.
Let X_ij be the result supplied (and verified) by participant i for instance j. Each X_ij is a pair (d,s) composed by the distance to feasibility d and the score of the objective function s (remember that results are compared based on d first, and then on s in the case where competitors have the same value for d).
The matrix X of results is transformed into a matrix of ranks assigning
to each X_ij a value R_ij from 1 to k. That is, for instance j the
supplied X_1j, X_2j, ...,X_kj are compared with each other and the
rank 1 is assigned to the smallest observed value, the rank 2 to the
second smallest, and so on to the rank k, which is assigned to the
largest value for instance i. Ranks are assigned in all of the
instances. We use average ranks in case of ties.
Consider the following example with m=6 instances and k=7 participants.
Instance 1 2 3 4 5 6
---------------------------------------------------------
Solver 1 (0,34) (0,35) (1,42) (1,32) (0,10) (0,12)
Solver 2 (3,32) (1,24) (1,44) (1,33) (0,13) (0,15)
Solver 3 (0,33) (0,36) (2,30) (5,12) (1,11) (0,17)
Solver 4 (0,36) (0,32) (1,46) (1,32) (0,12) (0,13)
Solver 5 (0,37) (0,30) (1,43) (1,29) (0,9) (0,4)
Solver 6 (2,68) (0,29) (1,41) (0,55) (0,10) (0,5)
Solver 7 (0,36) (0,30) (2,43) (0,58) (0,10) (0,4)
The ranks are the following:
Instance 1 2 3 4 5 6
---------------------------------------------------------
Solver 1 2 5 2 4.5 3 4
Solver 2 7 7 4 6 6 6
Solver 3 1 6 6 7 7 7
Solver 4 3.5 4 5 4.5 5 5
Solver 5 5 2.5 3 3 1 1.5
Solver 6 6 1 1 1 3 3
Solver 7 3.5 2.5 7 2 3 1.5
We define for each solver the mean of the ranks. The finalists of
the competition will be the 5 solvers with the lowest mean ranks. In
case of a tie for entering the last positions, all the last equal-mean
solvers are included in the final (in this case the finalists will be
more than 5). In the example, the mean ranks are:
Mean rank
--------------------
Solver1 3.42
Solver2 6
Solver3 5.66
Solver4 4.5
Solver5 2.66
Solver6 2.5
Solver7 3.25
In this case the finalists would be solvers 1,4,5,6, and 7.
The organisers will check the runs of the candidate finalist with the submitted seed to make sure that the submitted runs are repeatable. If they are not then another entrant will be chosen for the top five.
For the final, the same evaluation process is repeated for the finalists with the following differences:
1. All instances, including hidden ones, will be used.
2. The solvers will be run by the organisers, thus the finalist should give support to the organisers in the process of compiling and running the solvers.
3. For each instance, the organisers will run 10 independent trails with seeds chosen at random. For each trial, we will compute the ranks and average them on all trials on all instances.
The winner is the one with the lowest mean rank. In case of a tie, 1 trial is added for all instances until a single winner is found
Last Updated: Monday, July 30, 2007 10:16 PM