A methodology for explaining the performance of approximation algorithms for the Bin Packing Problem: a causal approach

Joaqun Prez1, Laura Cruz2, Rodolfo Pazos1, Vanesa Landero1, Veronica Perez2, Gilberto Rivera Zrate2,
Centro Nacional De Investigacin Y Desarrollo Tecnolgico (CENIDET) Departamento de Ciencias Computacionales, AP 5-164, Cuernavaca, 62490, Mxico 2 Instituto Tecnolgico De Ciudad Madero (ITCM) Divisin de Estudios de Posgrado e Investigacin y Departamento de Sistemas y Computacin {jperez, pazos, greyes}@cenidet.edu.mx { landerov76, lauracruzreyes}@yahoo.com
1

Abstract. The majority of works related to analysis and characterization of approximation algorithms performance, in an experimental way, build: descriptive and predictive models; the information used to build their models is: problem descriptive features or search space features. Nevertheless, there is not a complete and formal model to analyze relations between them and other kind of features such as problem space, and algorithm behavior and trajectory, that impact on algorithm performance. Also there is not a work that presents explicit guidelines on the formulation of complete and formal models to explain why an algorithm is better on instances set and understand its performance. This paper presents a methodology as a guide for analysis and characterization of algorithm performance in order to find explanations through causal modeling of earlier mentioned features. In particular, we show a study case for several versions of Tabu Search algorithm when they solve the bin packing problem.

1 Introduction
In many public or private entities there are a great number of difficult optimization problems. These problems are known in the computation scientific community as combinatorial optimization problems and they are classified as NP-hard. Heuristic algorithms have been proposed as a good alternative to solve very large instances [1, 2]. However, in the real life situations, there is not an algorithm that outperforms others algorithms in all circumstances [3]. The analysis and characterization of algorithms performance requires experimental research to produce precise, reliable, robust and generalizable results [4]. The majority of works related to analysis and characterization of optimization algorithms performance, in an experimental way, build: descriptive models, through comparative experimental results that show which is the best algorithm on a set of instances and predictive models to select the appropriate algorithm for a given instance; The information used by these works to build their models is: problem features or simplified measures of algorithm behavior or search space features. Nevertheless, there is not a complete and formal model to analyze and understand all kind of features (problem and algorithm) and relations between them that impact on algorithm performance, with the finally of explaining why an algorithm is the best on an instances set. The acquired understanding may lead to better predictions of algorithms performance in new situations and the discovering of improved algorithms. Nowadays, there are some works which present general guides and ideas to analyze the algorithm performance in the field of statistics because it provides the appropriate basis for supporting the research on metaheuristics. Nevertheless, there is not a work that presents explicit guidelines on the formulation of complete and formal models to explain and understand the algorithm performance that connects statistical analysis and causal analysis. We present in this paper a complete methodology that supports the two above mentioned situations and involves several ideas from [4, 5, 6, 7]. The principal parts of the methodology are: Identification of instances and possible features, Experimentation, Data Preparation, Identification of significant features and relations, Selection of a minimal set of features, Causal Analysis and Model Validation.

2 Related Works
The analysis of an observed behavior is made through a transition between three stages: description (D), prediction (P) and causality (C) [6]. The first stage gives a descriptive response without understanding exploratory questions. The second stage permits to extend the response to unseen instances, and requires a better understanding of the conditions under which the behavior occurs. In the third stage the responses become complex and imply to know the causes of the phenomenon in order to explain the observed behavior. In the case of analysis for approximation algorithms behavior, the transition between the mentioned stages has been done mainly through the next approaches. At the beginning the works reported by the scientific community were focused on the study of performance by the demonstration of the algorithm superiority on a set of standard instances [8]. The following works incorporated an algorithm characterization and some observations about the performance [9]. Subsequent works added the prediction possibility based on the previous algorithm characterization [10]. Recently, some works are trying to create models that explain which characteristics affect the performance and how they are related [11]. Table 1 presents some related works. The columns 2 and 3 indicate if the model includes problem features. The columns 4 to 6 indicate if the model incorporates algorithm features; these detail if elements such as algorithm behavior, search space or structure design, are considered to analyze the performance. The column 7 shows if the model presents explanations of the algorithm performance. Also the works are grouped by model type Descriptive (D), Predictive (P) and Causal (C).

Table 1. Related work on algorithm performance characterization

Work

Problem Features Descriptive Space

Algorithm Features Behavior Search observation Trajectory

Algorithm Structure Design

Give formal explanations

D [12] (D) [13] (P) [14] (P) [15] (P) [16] (P) [16] (C) [17] (C) C [18] (C) [19] (C) This work As it is observed in Table 1, the works does not include all aspects that influence to algorithm performance. This is a promising area to build models that learn patterns of relations between problem and algorithm features from algorithm domain regions; this models will have as main purposes to predict with a high accuracy the most adequate algorithm for given instance and to explain why an algorithm is the best on an instances set. The generation of strong theories in the experimental algorithmic area, demands the development of causal models which give a formal explanation of the observed behavior. Causal modeling is a generalized form to represent knowledge. Nowadays there are some works that propose general guides and ideas to analyze the algorithm performance. McGeogh presented methodological issues and guidelines for selecting research problems, input classes, and program metrics; for applying appropriate data analysis techniques and presenting conclusions [4]. Chiarandini formalized different scenarios for the analysis and comparison of metaheuristics by experimentation. For each scenario they give pointers to the existing statistical methodology for carrying out a deep analysis [5]. Nevertheless, also there is not a work that proposes a complete methodology that connects the statistical analysis and causal analysis with the finally to create a formal model that permits understand and give explanations about the algorithm performance. In this paper we present a complete framework that includes the ideas presented in [4, 5, 6, 7], also it deals the showed situations in Table 1 and connects the statistical and causal analysis.

3 Methodology to analyze the performance of the algorithms

The Figure 1 shows the proposed methodology to create a formal model that permits to understand and to explain the algorithm performance. The parts of this framework are explained in the next sections.

Identification of instances and possible features Data Preparation

Identification of significant features and relations

Experimentation

General Graphical Analysis and hypothesis formulation

Graphical and Statistical Analysis of each feature

Selection of a minimal set of features Multivariate Analysis

Graphical and Statistical Analysis of relations between features

Feature Selection

Causal Analysis

Model validation

Fig 1 A proposed methodology

3.1 Identification of instances and possible features The generation of observation questions leads to identification of possible features (dependent variables) and performance measures (response variable), in some cases a previous graphical analysis can help in this step. The problem instances to be studied must be carefully selected according the investigation goals. A good practical scheme is first to build a straightforward implementation of the algorithm to be studied, and to conduct a pilot study to help plan later a larger experimental project. Information from pilot study can be used to identify interesting input classes and to get an idea of appropriate problem sizes and project scope. Computational experiments can produce extraordinarily huge data sets, but a reasonable balance must be reached between too much and too little information [4, 20]. 3.2 Experimentation A well experimental design can lead to obtain precise, reliable, robust and generalizable results. It is recommendable take care with the randomness of the experiment one alternative is to use a new random seed for each instance to solve by the algorithm. Several runs of the algorithm for each instance help us to explore a significant sample of the problem space. The correct number of runs can be obtained verifying the variance of the best solutions of these runs for several instances of the problem. If the variance is very small, the number of runs is correct; else there are alternative techniques to reduce the variability. One of them is the variance reduction method which is described in [4]. 3.3 Data Preparation The data can be captured manually, but we strongly recommend collect it with a computational procedure in order to eliminate human errors. If the data are in different magnitudes, it is recommendable to normalize to same interval, for example values between [0, 1]. There are situations when the data presents incongruent values which can produce noise in the posterior analysis, it is important that the researcher revise carefully the data to

identify the instances which can have some noise and eliminate them. After that, the dependent variables can be discretized depending on type of analysis to perform. If a MANOVA analysis is performed, it is recommendable to discretize the dependent variables, but if a Principal Component Analysis (PCA) is performed, it is recommendable to manage the continuous data. The analysis process is amenable to the generation of valid scientific and engineering conclusions if the data follows four assumptions of statistics field: randomness, variation and location fixed, and normality. If the assumptions are not valid, then the process is drifting (with respect to location, variation, or distribution), unpredictable, and out of control, it leads to engineering conclusions that are not valid, are not supportable (scientifically or legally), and which are not repeatable in the laboratory. An alternative way to board these assumptions is to transform the data; some techniques are commonly used in this case: Box-Cox method [21], logarithm and square root. 3.4 Identification of significant features It is necessary to perform a previous general graphical analysis to the data, to identify features that could be discriminatory related to performance. Tables and graphics of relative frequencies can be used to analyze the distribution of the features. The formulation of hypothesis can be done once the discriminatory features are identified. After that, a graphical and statistical analysis of each feature and relations between them can be performed to verify the hypothesis formulated. The techniques commonly used are: Box Plot, Effect Mean Plot, Interaction Mean Plot and MANOVA analysis. In this case the model residuals must be analyzed to validate model results. The residuals must accomplish: normality, constant variation, independence, non derivation. Finally, the results of those analyses can be used to identify the significant features that affect the algorithm performance. 3.5 Selection of a minimal set of features The set of selected features in the previous step can have redundant features with the same significance, therefore, it is necessary to eliminate that features to identify a minimal features set. We recommended two common alternatives: one is to perform the multivariate analysis and another is to select the main features by some method. Both alternatives could be complementary. The first alternative can help to discover existing relations between features and to verify the founded results in the previous step. It is necessary the data must accomplish with a multivariate normal distribution, if not, it must be transformed. PCA can be used to verify if the data after transformation follow a multivariate normal distribution. We recommended a Factor Analysis (FA) to discover the minimal set of significant features. In the second alternative, a supervised method of features selection could be used. It can help to verify the founded results in the previous step. 3.6 Causal Analysis A causal model is a generalized representation of knowledge that is obtained by finding dependences through the data that implies cause-effect relations [11]. The process of generating causal models is not trivial; Chickering proved that this problem is NP-hard [22]. Causal Analysis generally has four stages: specification, estimation, interpretation. The first indicates which variables are causes and which are effects, one of the most common representations for these relations is using directed acyclic graphs. We proposed to use, like alternative, PC algorithm [7] for the creation of the causal model using the significant features previously selected and the algorithm performance. The second determines the intensity of founded causal relations, conditional probabilities of these relations and their support (indicates how many observations accomplish a given dependence) are calculated; causal relations which showed greater values of conditional probability and support are interpreted. The third find explanations inferred from these relations. This scenario confirms the suppositions made in the previous analyses.

3.7 Model validation There are two important alternatives to validate the model which are mentioned in this framework. One is the theoretical validation, which must accomplish Markov, Minimality and Faithfulness conditions. The PC algorithm gives guarantee of these conditions. Second, the model is proved to discover its accuracy, the features of the model can be test to know if they are sufficient to predict unknown instances.

4 A test case: Analyzing the performance of Tabu Search algorithm

The proposed methodology is used to build a causal model for one version of Tabu Search algorithm [23], applied in the solution of the bin packing problem. The design of the algorithm structure of this version (V1) is characterized by these situations: it uses a dynamic Tabu list, a heuristic initial solution and several methods to create neighbour solutions. We modified each one of these situations from V1 to create other versions V2, V3 and V4 (see Table 2) to identify if there is an impact on performance of the V1 algorithm. One example of each step of the methodology is presented in the next sections. 4.1 Identification of instances and possible features We identify several observation questions to identify possible features (dependent variables) and performance measures (response variable), which some are showed to continuation. 1. 2. How is the distribution of the weights in won instances and lost instances? Is there a significant difference? How is the problem space (a sample of solutions of the problem) and search space (during the execution) of the won instances and the lost instances? Is there a significant difference in the number and variability of the feasible solutions? Are there more inflexion points in the trajectory of the algorithm during execution in the lost instances than in the won instances? Are there more valleys in the trajectory of the algorithm during execution in the lost instances than in the won instances? How are the sizes of these valleys?

3.

The identified features (dependent variables) are classified as: problem description (PD), problem space (PS), and during the execution the features of the algorithm behavior (AB) and search trajectory (ST). The performance measure is the ratio (rcal) of the best found solution by the algorithm and the theoretical solution. A pilot study was performed to identify interesting input instances, we select a subset of the Bin-Packing Problem defined by Scholl-Klein-Juergens [24] and these are recognized and utilized by the scientific community. 4.2 Experimentation Table 2 shows the versions of the Tabu Search Algorithm. The first and second column indicates if the algorithm uses a tabu list static or dynamic. The third and fourth column indicates if the algorithm starts with a random or heuristic initial solution. The fifth and sixth column indicates if the algorithm uses one method or several methods to create neighbour solutions. Table 2. Versions of the Tabu Search Algorithm Tabu List Initial Solution Creation of Neighbourhood Static Dynamic Random Heuristic One method Several

Version V1 V2 V3 V4

The versions were executed 15 times (in the pilot study we observed a very small variance of these 15 runs) for each instance on a server with characteristics: DELL Power Edge 2600, CPU Xeon 3.06 GHz, RAM 3.87 GB, Microsoft Windows Server 2003. We initialize the random seed for each run to guaranty the randomness.

To establish the region of domination of each version a comparison between them was carried out. It was made by means of two criterions, first the obtained rcal in the instance solution, second the fitness value when the algorithms obtain the same rcal. The won instances of the comparisons between versions are: V1=192 and V2=132; V1=229 and V3=95; V1=166 and V4=158. V1 was superior over version V3 with a high difference. Therefore we analyze the version V1 as an example to apply the methodology and explain why it was superior on V3. 4.3 Data Preparation The versions generate three output files: one contains the performance results for all instances, second contains characteristics on the algorithm behavior during execution for all instances, third is for each solved instance and contains the fitness values for each run. Also we created external programs: one program to read this last file and generate statistics from the search trajectory of the algorithm; the second program to characterize the weights for each instance; the third program to obtain the problem space features for each instance. The data were normalized by min-max method, all values are in a range of [0, 1]. Also we eliminated found noise in the data, for example there were some instances when the number of feasible solutions was zero, but it does not mean that the search trajectory was difficult, only the algorithm found the optimal solution when the initial solution was created. After that, we recollected 324 instances and the dependent variables were discretized by the MDL method [25] (weka project libraries, www.cs.waikato.ac.nz/ml/weka/). Finally, we used the BoxCox method to transform the response variable (rcal) to follow the assumptions randomness, variation and location fixed and normality. Table 3 shows the intervals obtained for the final selected features in this analysis to have a major understanding in the final conclusions.

Table 3. Example of Discretization of variables

Features vr evfac vec prt nc nv tm

Type PD PS AB AB ST ST ST

Interval 1 0,0 0,0.1366 0,0.0924 0,0.003 0,0.0324 0,0.0296 0,0.0878

Interval 2 0,1 0.1378,1 0.0930,0.2770 0.0003,0.0016 0.0328,0.1501 0.0298,1 0.0878,1

Interval 3

0.2791,1 0.0016,0.0134 0.1541,1

Brief description Variability of the weighs distribution Variability of the feasible solutions Size of the neighborhood Percentage of Tabu rejected solutions Percentage of inflexion points Percentage of valleys Size of valleys

4.4 Identification of significant features General Graphical Analysis and hypothesis formulation

We created tables and graphics of relative frequencies to identify discriminatory features that have high differences in their values between won instances and lost instances. Figure 2 shows an example of one selected feature nc, the version V1 won when the number of inflexion points in the search trajectory nc is big and lost when nc is small. We formulated hypothesis from this analysis, which one of them is showed. H0: The number of inflexion points nc in the search trajectory of the version V1 does not have relation with the algorithm performance. H1: The number of inflexion points nc in the search trajectory of the version V1 is a factor that has an influence on algorithm performance.

Fig 2 Percentage of frequencies of the values of nc for won and lost instances by version V1.

Graphical and Quantitative Analysis of each feature

Figure 3 shows a Box Plot of the feature vec and a Mean Plot of the features evfac, prt, tm as example. These features have an influence on algorithm performance in variation and location. The response variable rcal is affected when the size of neighborhood vec is small and the version V1 obtains a better rcal when the values of neighborhood vec are increased.

Fig 3 Box Plot of vec and Mean Plot of evfac, prt, tm.

Graphical and Quantitative Analysis of relations between features

Relations between the variables efac, evfac, nv, pa, pn, lcandi, vec and prt with the variable performance as a response variable were analyzed. Figure 4 shows as an example the interactions of some features with their respective levels; there apparently exists evidence of relations between values, some of them are: The percentage of feasible solutions efac and their variability evfac, the negative pendant of the valleys pn and the number of valleys nv, the size of candidates list lcandi and the size of the neighborhood vec to create it, are related to performance. To confirm the suppositions based on the graphic analyses, a multivariate analysis of variance (MANOVA) was carried out. The analysis was done with MINITAB software, it showed that some features are related with the response variable rcal such as: vr, evfac, vec, prt, nc, nv, tm. The residuals were analyzed and they gradually adjust to a normality distribution and the constant variation test is rejected. However the ANOVA test is very robust to deviations of the assumption of normality [5] and the F test in ANOVA is affected a little in a model for balanced cases [26,6], which it is the case. Finally, we accepted the formulated hypothesis and identified the features and relations that impact on algorithm performance.

Fig 4 Effect Mean Plot of several features

4.5 Selection of a minimal set of features Multivariate Analysis The set of identified features in the previous step were vr, sfac, vfac, t, efac, evfac, nc, nv, tm, pn, pa, prt, vec, and lcandi. We transformed the features, only the features vr, sfac, vfac and pn can not to transform and we excluded them in this analysis. After that we performed the PCA analysis to confirm that they follow a multivariate normal distribution, as it is shown in Figure 5, each principal component follows a normality plot. We performed FA analysis with 3 factors, suggested by PCA analysis and the results are shown in Table 4.

Table 4. Results of FA analysis 1

pa prt nc nv vec lcandi evfac t efac Tm X X X X X X X

Fig 5. Normality plots of principal components in the PCA analysis

The features evfac, t (occupied capacity by an average object), efac and tm are not grouped by the analysis. The percentage of accepted solutions pa and percentage of Tabu rejected solutions prt could give same information. The size of the neighborhood vec and the size of candidate list lcandi could give same information. But we can not affirm the number inflexion points nc, and the number of valleys nv give same information. These results were validated with two analyses: we performed FA with only the grouped features, the result was the same number of factors; we performed FA with qualifications of the previous result and non correlated features, the result was the same, there were not common factors. Feature selection Also, we performed a selection of the identified features with the supervised method described in [27] (weka project libraries, www.cs.waikato.ac.nz/ml/weka/) to complement the founded results in the PCA and FA analyses. There were not selected features, but considering the results of PCA and FA analyses we eliminated the features pa and lcandi because we consider that they give same information as prt and vec. Therefore the selected features are: vr, sfac, vfac, t, efac, evfac, nc, nv, tm, pn, prt and vec, and they will be considered in the next analyses.

4.6 Causal Analysis Specification of causal order Initially, we created a one causal model of all selected features in the previous analyses, but the result model did not give us important information about the direct causes of the algorithm performance. The analysis of too many features in a one model could be inefficient. Therefore we recommended divide the features according to the nature of their information. One causal model was created to next classification of features: descriptive and space of problem, algorithm behavior and search trajectory. The construction of each causal model was carried out using the TETRAD software [28] and the PC [7] algorithm with a confidence level of 0.05%.

Fig 6a Causal model for problem description features Fig 6b Causal model for algorithm behavior features

In Figure 6a it is appreciated that the variable vr and evfac are direct causes of performance node; also the features vec and prt in Figure 6b; and the features nc, nv and tm in Figure 7.

Table 5. Conditional probabilities and support % Probability P(performance=1 |vr=2, evfac=2) 0.54370 P(performance =1|vec=3, prt=2) 0.04543 P(performance =1|nc=3, nv=2, tm=2) 0.07519 P(performance=2 |vr=1, evfac=1) 0.36687 P (performance =2|vec=1, prt=1) 0.17285 P(performance =2|nc=1, nv=1, tm=2) 0.31619 Relation Support 0.36728 0.15741 0.25926 0.09876 0.07716 0.13580

Fig 7 Causal model for search trajectory features

Estimation of the model Probability values of the nodes related to the performance node were calculated from the three causal models. These most important probabilities and supports are presented in Table 5.

Model interpretation Causal relations which showed greater values of conditional probability and support were interpreted; the following explanations were inferred from these relations. P(performance |vr=2, evfac=2). The version V1 wins because the weights distribution are more uniform (0, 1), the variability between the values of the fitness function of feasible solutions during the algorithm execution is big (0.1378, 1). P(performance |vec=3, prt=2). The version V1 wins because the size of the neighborhood, to create the candidate list, is big (0.2791, 1). The percentage of Tabu rejected solutions is medium (0.0003, 0.0016). P(performance |nc=3, nv=2, tm=2). The version V1 wins because the number of inflexion points in the search trajectory of the algorithm is big (0.1541, 1). The number of the founded valleys is big (0.0298, 1) and their size is big (0.0878, 1). P(performance |vr=1, evfac=1). The version V1 loses because the weights distribution are not uniform (0), the variability between the values of the fitness function of feasible solutions during the algorithm execution is small (0, 0.1366). P(performance |vec=1, prt=1). The version V1 loses because the size of the neighborhood, to create the candidate list, is small (0, 0.0924). The percentage of Tabu rejected solutions is small (0, 0.003). P(performance |nc=1, nv=1, tm=2). The version V1 loses because the number of inflexion points in the search trajectory of the algorithm is small (0, 0.0324). The number of the founded valleys is small (0, 0.0296), but their size is big (0.0878, 1). The obtained explanations indicate that a particular combination of values for the features vr, evfac, vec, prt, nc, nv, tm and the relation of the structure design of V1; characterizes an instance to won or lost against V3. We obtained an interpretation of this relation to understand why V1 algorithm wins over V3 algorithm. The use of several methods to build neighbor solutions by V1 permits: to intensify the search in the solutions space of instances with a distribution almost uniform in their weights; support a median number of Tabu rejected solutions; to generate high variability between solutions; obtain advantage of the form of the search trajectory (big number of inflexion points and valleys) to get out from valleys. The principal disadvantage of this design is doubt to management of instances with non uniform distribution, because this kind of instances can be solved by a one simple method to build neighbor solutions, it generates solution values which are too similar, permit a small number of Tabu rejected solutions and generating a flat trajectory, in which V3 takes advantage. 4.7 Model validation The theoretical validation was omitted in this example because we used the PC algorithm and it guarantees the conditions of Markov, Minimality and Faithfulness. The features of the model were validated to confirm that they are important to predict unknown instances, we run the Nave Bayes and C4.5 [29, 30] classifiers and we obtain with the cross-validation method the classification levels 82.10 % and 87.65.

5. Conclusions and Future works

This paper presented a methodology to build a complete and formal model of algorithm performance. This framework recommended the complementary use of techniques, methods and algorithms which permits to analyze the algorithm behavior, connecting statistical analysis and causal analysis. Their principal objectives are: to identify a minimal set of significant features (the prediction accuracy could be increased), considering problem description, problem space, algorithm behavior and search trajectory; to explore the relations of these features on algorithm performance; to explain with a causal model the algorithm performance, giving interpretations of why the algorithm is better on an instances set, connecting problem features and the features of the algorithm structure design. The analysis of algorithms by means of causal models allows to study the nature of complex algorithms deeply, to contribute to the generalization of the knowledge about them, to obtain open issues for a better design of them or build new algorithms.

As a future work an automatic tool to perform all steps of this methodology is proposed, including the use of the built model as prediction mechanism and evaluating its accuracy on new problem instances. Besides extending the analysis to other versions of Tabu Search algorithm with objective is to contrast its performance with different configurations. Finally the use of this methodology could allow us understanding the performance of other approximate algorithms.

References
[1] Garey, M. R., Jonson, D. S.: Computers and Intractability, a Guide to the Theory of NP-completeness. W. H. Freeman and Company, New York, USA (1979). [2] Merz, P., Freisleben, B.: Fitness Landscapes and Memetic Algorithm Design. New Ideas in Optimization, pp. 245-260, McGraw-Hill Ltd, UK (1999). [3] Wolpert, D. H., Macready, W.G.: No Free Lunch Theorems for Optimizations. IEEE Transactions on Evolutionary Computation 1, 67-82 (1997). [4] McGeoch, C.C.: Experimental Analysis of Algorithms, Handbook of Global Optimization, vol. 2, 489-513 (2002). [5] Chiarandini, M., Paquete, L., Preuss, M., Ridge, E.: Experiments on Metaheuristics: Methodological Overview and Open Issues. Preprints 2007 No. 4, ISSN No. 0903-3920, (March 2007). [6] Cohen, P.: Empirical Methods for Artificial Intelligence, The MIT Press Cambridge, Massachusetts, England, (1995). [7] Spirtes, P., Glymour, C., Scheines , R.: Causation, Prediction, and Search, MIT Press, 2nd edition (2001). [8] Hooker, J.N.: Needed: An empirical science of algorithms, Operations Research, vol. 42, (1994). [9] Borghetti , B.J.: Inference Algorithm Performance and Selection under Constrained Resources, MS Thesis, (1996). [10] Cruz R., L.: Caracterizacin de Algoritmos Heursticos Aplicados al Diseo de Bases de Datos Distribuidas, Tesis de doctorado, Centro Nacional de Investigacin y Desarrollo Tecnolgico, Cuernavaca, Morelos, (Jun., 2004). [11] Lemeire, J., Dirkx , E.: Causal Models for Performance Analysis, 4th PA3CT Symposium, Edegem, Belgica, (2004). [12] Hoos, H.: Stochastic Local Search -Methods, Models, Aplications, PhD Thesis, Department of Computer Science from Darmstadt University of Technology, Germany, (Nov. 1998). [13] Soares, C., Pinto, J.: Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results. Journal of Machine Learning Vol. 50, No. 3, 251-277, (2003). [14] Guo, H., HSU, W.: A Learning-based Algorithm Selection Meta-reasoner for the Real-time MPE Problem. In Proceedings of the 17th Australian Joint Conference on Artificial Intelligence, Cairns, Australian, Dec 2004, G. I. WEBB AND XINGHUO YU, Ed. Springer-Verlag Vol. 3339, 307-318, (2004). [15] Guerri, A., Milano, M.: Learning techniques for Automatic Algorithm Portfolio Selection. In Proceedings of the 16th Biennial European Conference on Artificial Intelligence, Valencia, Spain, Eds. IOS Press, Burke, VA, 475-479, (2004). [16] Hoos, H., Smyth, K., Stutzle, T.: Search Space Features Underlying the Performance of Stochastic Local Search Algorithms for MAX-SAT. In proceedings PPSN 2004, 51-60 (2004). [17] Konak, A.: Simulation optimization using Tabu search: An empirical study. Proceedings of Winter simulation Conference, M. E. Kuhl, N. M. Steiger, F. B. Armstrong, and J. A. Joines, eds, (2005). [18] Lemeire, J., Dirkx, E.: Causal Models for Parallel Performance Analysis. In proceedings of 4th PA3CT Symposium, Edegem, Belgium, (Sep. 2004). [19] Chevalier, R.: Balancing the effects of parameter settings on a genetic algorithm for multiple fault diagnosis. Artificial Intelligence, The university of Georgia. (2006). [20] Barr, R., Golden, B., Resende, M.: Designing and Reporting on Computational Experiments with Heuristic Methods. Journal of Heuristics, Vol. 1, No. 1, 9-32, (1995). [21] Box, G., Cox, D.: An Analysis of Transformations, Journal of the Royal Statistical Society, 211-243, (1964). [22] Chickering, D.: A transformational Characterization of Equivalent Bayesian Network Structures, 11th Conference on Uncertainly AI. San Francisco, 87-98, (1995). [23] Glover, F.: Tabu Search - Part I, First comprehensive description of tabu search. ORSA Journal on Computing, Vol. 1, No. 3, pp. 190-206, (1989). [24] Beasley, J. E.: OR-Library. Brunel University. http://people.brunel.ac.uk/~mastjjb/jeb/orlib/ binpackinfo.html (2006). [25] Multi-Interval Discretization of Continuous-Valued Attributes for Classification Learning. Usama M. Fayyad, Keki B. Irani. IJCAI, 1022-1029, (1993). [26] Montgomery, D.: Design and Analysis of Experiments, John Wiley and Sons, Inc. 5th Edition (2001). [27] Liu, H., and Setiono, R.: A probabilistic approach to feature selection - A filter solution. In 13th International Conference on Machine Learning (ICML'96), July 1996, pp. 319-327. Bari, Italy, (1996). [28] Carnegie Mellons University, Open Learning Initiative (OLI), http://www.cmu.edu/oli/. [29] Zhipeng, X., Wynne, H., Zongtian, L.: SNNB A selective neighborhood based naives for lazy learning. In Sixth PacificAsia Conference on Knowledge Discovery and Data Mining (PAKDD 2002), Taipei, Taiwan, (2002). [30] Quinlan, J.: C4.5: Programs for machine learning, Morgan Kaufmann, San Mateo, Calif. (1993)