Parameters for OPTANO Algorithm Tuner
OPTANO Algorithm Tuner (OAT) combines a small number of required parameters with a rich interface for configuration. This page describes all possible parameters and their default values.
When using OAT in a distributed fashion, multiple instances can be started as workers to speed up the tuning. The possible parameters for workers can be found in a separate section at the end of the page.
Note that when you execute a tuning in several sessions, most parameters are adopted from your previous session.
Finally, check the parameter selection page for some general advice on how to choose some of the described parameters.
Master
- --help (-h)
- Prints information about usage to console.
Required Parameters
Most parameters in OAT have a default value, but you have to specify the parameters that depend on your hardware. As part of the tuning, your algorithm is run many, many times with different configurations. You need to specify how many of these runs can be conducted in parallel per node.
Address
OAT will try to automatically detect your computing nodes' fully qualified domain names and use them for exchanging messages. If you experience connection problems on your system, you should try to set the host names explicitly.
In addition, it is possible to specify the port on which the master listens for worker connections.
- --ownHostName={HOSTNAME} [Fully Qualified Domain Name]
- The address that the master uses for incoming messages. On some systems the FQDN cannot be resolved on the fly. In that case, please provide the FQDN or an IP address.
- --port={NUMBER} [8081]
- The port on which the master listens for worker connections. Must be identical for master and respective workers, but different for different parallel runs.
- --allowLocalEvaluations={BOOLEAN} [true]
- Value indicating whether target algorithm evaluations on the master node are allowed. If false, the master will not execute any target algorithm evaluations, but will only be responsible for their distribution. In particular, the master will wait for worker nodes to join and execute the evaluations. In both cases, the master is potentially responsible for training the genome prediction random forest and the gray box random forest and should be endowed with sufficient cpu power, if genetic engineering or gray box tuning are enabled.
Target Algorithm Specific Parameters
In addition to the required parameters, you will often want to change parameters that are closely connected to the algorithm you are tuning the parameters for.
In each iteration, configurations are evaluated using a random subset of the provided training instances. The size of these subsets increases throughout the run such that the best configurations which we still evaluate in later iterations have been evaluated on the largest number of instances. The exact number of instances to use can be specified via parameters.
- --trainingInstanceFolder={PATH}
- The complete path to the folder containing instances. Handled differently by each application.
Note: For most applications, if you execute OAT in a distributed fashion, you will have to make sure that all computing nodes either can access the same instance folder or have identical instance folders stored at identical paths. - --testInstanceFolder={PATH}
- Similar to
--trainingInstanceFolder
, but only used for additional information on incumbent quality in case--scoreGenerationHistory
is provided. - --instanceNumbers={NUMBER:NUMBER} (-i) [5:100]
- The number of instances to use at the first generation and the number of instances to use at the end. Instance numbers increase linearly from the start until goalGen and then stay at maximum size until the end of the tuning.
- --addDefaultGenomeToFirstGeneration={BOOLEAN} [true]
- If set to true, a genome that uses the target algorithm's default parameter values (if specified), is included in the competitive population of the first generation. If a parameter does not have a specified default value, a randomized value from its domain is used instead.
- --addDefaultGenomeToFinalIncumbentGeneration={BOOLEAN} [true]
- If set to true, a genome that uses the target algorithm's default parameter values (if specified), is added to the final incumbent generation. If a parameter does not have a specified default value, a randomized value from its domain is used instead.
Speedups in Tuning
Racing: Loosely speaking, racing means to skip evaluating a configuration further once it becomes clear that it can not beat the current numberOfTournamentWinner-best configuration and therefore won't get to be a mini tournament winner. Of course, this ordering strongly depends on your overall tuning metric. For example, in runtime tuning with one mini tournament winner racing can skip evaluations once a configuration's total runtime exceeds the total runtime, needed by the best configuration to solve all instances. Thereby this strategy has the potential to greatly reduce the tuner's run time because the worst configurations are evaluated less often. Moreover this might lead you to introduce an evaluation priority of a configuration, based on its ability to be a racing threshold candidate. The basic version of OAT comes with two basic tuning metrices: (penalized) runtime tuning and pure value tuning. While you cannot use racing in pure value tuning, (penalized) runtime tuning comes with a full featured implementation of this strategy. When tuning for custom optimization functions you can provide your own tuning metric, racing strategy and evaluation priority score by implementing the IRunEvaluator
interface. The Gurobi example provides an exemplary implementation on how to customize your strategy.
CPU Timeout: Racing is useful, but only checks the run times between finished algorithm runs. That means that a really long run will never be cancelled, and the bad configuration will only be ignored afterwards. To avoid this problem, a CPU timeout can be set per run. Runs will be cancelled after this time and the run time will be counted as the run time until cancellation.
- --enableRacing={BOOLEAN} [false]
- Value indicating whether racing should be enabled.
- --cpuTimeout={SECONDS} (-t) [int.MaxValue]
- The CPU timeout per target algorithm run in seconds.
Scale of the Tuning
OAT uses evolutionary algorithms as the basis for tuning. They can be scaled by their population size and termination criteria.
- --popSize={NUMBER} (-p) [128]
- The total population size in the genetic algorithm.
- --numGens={NUMBER} (-g) [100]
- The number of generations to execute in the genetic algorithm.
- --goalGen={INDEX} [74]
- The first generation (0-indexed) at which the maximum number instances per genome evaluation will be reached.
- --addFinalIncumbentGeneration={BOOLEAN} [true]
- If set to true, the last regular generation is followed by a final incumbent generation, in which all incumbent genomes get evaluated on all provided training instances within a single mini tournament. By using a random subset of instances per generation, the current incumbent genome does not need to be stricly better than its predecessor. For example, the current incumbent genome may outperform its predecessor on the current instance set, but perform worse on all provided training instances. While using a random subset of instances per generation to avoid overfitting is still recommended, adding a final incumbent generation ensures that OPTANO Algorithm Tuner always returns the fittest incumbent genome. The number of additional evaluations is kept low by caching previous results and applying racing, if enabled.
- --tuningRandomSeed={NUMBER} [None]
- The random seed to control the initial population and the subset of instances, used per generation. If not specified, a time-dependent seed is used.
- --evaluationLimit={NUMBER} [2147483647]
- A maximum number of (configuration - instance) evaluations after which the program terminates.
Logging
- --verbose={0, 1, 2, 3} (-v) [1]
- The verbosity level. 0 only prints warnings, 1 regularly prints some status information, 2 prints more detailed information, e.g. calls to the target mechanism, and 3 is for debugging.
Note that workers have the same parameter for their own output. - --statusFileDir={ABSOLUTE_PATH} [current directory/status]
- Directory to which intermediate results should be written to. If --continue is set, this is also the path from which the intermediate results will be read.
- --zipOldStatus={BOOL} [false]
- Whether to zip old status files. Otherwise, old status files are overwritten.
- --logFile={ABSOLUTE_PATH} [current directory/tunerLog.txt]
- Path to which log file should be written to after every generation.
- --trackConvergenceBehavior
- Add to create
averageConvergence.csv
logging file.
- --scoreGenerationHistory
- Add to create
scores.csv
logging file.
Can significantly increase total software runtime as it adds a post-processing phase. However, best parameterization is printed before that phase.
Gray Box Tuning
To reduce the runtime, wasted on out-timing target algorithm runs, OAT provides a gray box extension, which aims for detecting and cancelling these timeouts at run time. You can adapt the behaviour of this gray box extension with the following parameters.
- --enableDataRecording={BOOL} [false]
- If this option is enabled, this OPTANO Algorithm Tuner instance will record the target algorithm's runtime features and write data log files, potentially used for gray box tuning. To enable gray box tuning, this option and the gray box tuning option need to be enabled.
- --dataRecordUpdateInterval={NUMBER} [5% of CPU timeout]
- Sets the update interval of the data recorder in seconds. Every dataRecordUpdateInterval seconds a data point is recorded and the gray box classifier is applied, if gray box tuning is enabled.
- --dataRecordDirectory={ABSOLUTE_PATH} [currentDirectory/DataLogFiles]
- Sets the path to the directory where the data log files should be written to.
- --enableGrayBox={BOOL} [false]
- If this option is enabled, this OPTANO Algorithm Tuner instance will use gray box tuning in order to minimize the overall tuning time.
- --grayBoxConfidenceThreshold={NUMBER} [0.75]
- Sets the confidence threshold of the gray box random forest. The current target algorithm run is cancelled by the gray box, if the confidence of the random forest exceeds this threshold.
- --grayBoxStartGeneration={NUMBER} [5]
- Sets the 0-indexed gray box start generation. Before this generation, no target algorithm run is cancelled by the gray box.
- --grayBoxStartTimePoint={NUMBER} [5% of CPU timeout]
- Sets the gray box start time point during a target algorithm run in seconds. Before this time point, no target algorithm run is cancelled by the gray box.
- --removeDataRecordsFromMemoryAfterTraining={BOOL} [false]
- If this option is enabled, this OPTANO Algorithm Tuner instance will remove the list of data records from memory after training the gray box random forest and read in all data log files again in every generation. This option will decrease the memory usage, but increase the time, needed to read in the data log files in every generation.
Fault Tolerance
- --faultTolerance={NUMBER} [3]
- Maximum number of consecutive failures in an evaluation before the affected instance of OAT is stopped. The most recent exception will be written to its console. If the stopped instance did not act as master, the overall tuning will continue.
- --maxRepair={NUMBER} [20]
- The maximum number of attempts to repair a genome if it is invalid after crossover or mutation. Only relevant if you have implemented a
GenomeBuilder
. - --strictCompatibilityCheck={BOOLEAN} [true]
- Option to turn off / on the compatibility check between the current and old configuration in case of a continued run. Use with care.
Tuning Algorithm
By default, OAT makes use of the GGA algorithm. There exist parameters to switch to a different tuning algorithm or even hybridize them.
- --jade
- Activates JADE.
- --cmaEs
- Activates CMA-ES.
- --maxGenerationsPerGgaPhase={GENERATIONS} [int.MaxValue]
- The maximum number of GGA(++) iterations to execute before switching to an alternative tuning algorithm. Set to 0 for pure JADE or CMA-ES.
- --maxGgaGenerationsWithSameIncumbent={GENERATIONS} [int.MaxValue]
- The maximum number of consecutive GGA generations not finding a new incumbent. If met, OAT switches to an alternative tuning algorithm. Set to a value greater or equal the total generation number to turn off this criterion.
- --maxGenerationsPerDePhase={GENERATIONS} [int.MaxValue]
- The number of JADE iterations to execute before starting a GGA(++) phase. Set to a value greater or equal the total generation number for pure JADE.
- --maxGenerationsPerCmaEsPhase={GENERATIONS} [int.MaxValue]
- The number of CMA-ES iterations to execute before starting a GGA(++) phase. Set to a value greater or equal the total generation number for pure CMA-ES.
- --focusOnIncumbent={BOOL} [false]
- Whether JADE/CMA-ES should focus on improving the incumbent or work on the complete population. Focus on incumbent is not possible without GGA(++) phases and not recommended for target algorithms without any categorical parameters.
Gender-based Genetic Algorithm
By default, OAT uses GGA, a genetic algorithm, as the basis for tuning. Therefore, some genetic algorithm parameters can be set:
- --miniTournamentSize={NUMBER} [8]
- The maximum number of participants per mini tournament.
Note: If the mini tournament size is very small, the racing framework is not very effective and tuning might take longer. - --maxGenomeAge={NUMBER} [3]
- The number of generations a genome survives in the genetic algorithm.
- --winnerPercentage={PERCENTAGE} (-w) [0.125]
- The percentage of winners per mini tournament. These are the competitive genomes allowed to reproduce.
- --enableSexualSelection={BOOL} [false]
- Set a value indicating whether an attractiveness measure should be considered during the selection of non-competitive mates. The attractiveness of a genome refers to the rank that is predicted for it by the same model that can also be used for the model based crossover
- --mutationRate={RATE} (-m) [0.1]
- The probability that a parameter is mutated in the genetic algorithm.
- --mutationVariance={PERCENTAGE} [0.1]
- The percentage of the variable's domain that is used to determine the variance for Gaussian mutation.
- --populationMutantRatio={RATIO} [0.25]
- The ratio of the non-competitive population that gets replaced by random mutants after every generation. Value is only used if
--engineeredProportion
is set to a positive value. - --crossoverSwitchProbability={PROBABILITY} [0.1]
- The probability that we switch between parents when doing a crossover and deciding on the value of a parameter that has different values for both parents and has a parent parameter in the parameter tree which also has different values for both parents.
- --trainModel
- Add if a performance model should be trained even if genetic engineering and sexual selection are turned off.
Model-Based Crossover Parameters
OAT also implements GGA++ by providing a model-based crossover operator that can be used for creating new offspring. Note that some of the parameters can have a significant impact on the time consumed for the population update. The application of this operator is also referred to as Genetic Engineering.
- --engineeredProportion={PERCENTAGE} [0]
- The proportion of offspring that should be engineered by using the model-based crossover operator.
- --startIterationEngineering={NUMBER} [3]
- Sets the iteration number in which the genetic engineering should be incorporated in the tuning.
- --targetSampleSize={NUMBER} [125]
- Sets the number of random samples to generate per reachable leaf node during genetic engineering.
- --distanceMetric={DistanceMetric} [HammingDistance]
- Sets the distance metric to use during genetic engineering. Score for offspring will be a combination of predicted performance and the "uniqueness" (i.e. distance to current population).
- --maxRanksCompensatedByDistance={NUMBER} [1.6]
- Sets the influence factor for the 'distance' between a potential offspring and the existing population when scoring the potential offspring. All measured distances will be normalized to the range [0, maxRanksCompensatedByDistance] when computing the offspring score.
- --featureSubsetRatioForDistance={RATIO} [0.3]
- Distances between Genomes during genetic engineering are only computed over a subset of the features. The subset of the given size selected at random.
- --hammingDistanceRelativeThreshold={RATIO} [0.01]
- Sets the relative threshold above which two compared genomes are considered to "be different" in a certain feature.
- --crossoverProbabilityCompetitive={PROBABILITY} [0.5]
- Sets the probability with which a non-fixed parameter will be selected from the competitive genome during the genetic engineering. Can be used to "stir" the engineered genomes away from "known" high-quality areas of the search space (i.e. set the probability < 0.5).
- --topPerformerThreshold={PERCENTAGE} [0.1]
- Sets the proportion of genomes that are considered to be 'top performers' during model based crossover. Only applied when
TopPerformerFocusSplitCriterion
is used.
The Model-Based Crossover uses a random forest in order to predict the behavior of a candidate offspring. The following parameters modify the random forest:
- --forestTreeCount={NUMBER} [75]
- The number of trees in the random forest.
- --forestFeaturesPerSplitRatio={RATIO} [0.3]
- The percentage of features to use per split during the training of the trees.
It is recommended that this number is set to a value of approx. 1/3, or 1/sqrt(#features).
The "#features" refers to the length of your parameter tree's double[]-representation. Setting "forestFeaturesPerSplitRatio" to 1/sqrt(#features) means that the trees will be trained with #features * 1/sqrt(#features) = sqrt(#features), which is a commonly used value according to the literature. - --forestMaxTreeDepth={NUMBER} [10]
- The maximum depth of a tree.
- --forestMinSplitSize={NUMBER} [2]
- The minimum size of a split in a tree.
- --forestMinInformationGain={NUMBER} [1e-6]
- The minimum information gain for a split in the trees.
- --forestSubSampleRatio={RATIO} [0.7]
- The proportion of the training set that is passed to each tree during training.
- --forestRunParallel={BOOL} [true]
- Enables parallel training of the random forest.
JADE Parameters
As detailed on the page about tuning algorithms, it is possible to employ the JADE parameter tuner instead of the default GGA(++) one.
Note that JADE only acts on the competitive population part, i.e. it is effectively exploiting only half of the specified population size.
- --jade
- Activates JADE.
- --maxGenerationsPerGgaPhase={GENERATIONS} [int.MaxValue]
- The number of GGA(++) iterations to execute before starting a JADE phase. Set to 0 for pure JADE.
- --maxGgaGenerationsWithSameIncumbent={GENERATIONS} [int.MaxValue]
- The maximum number of consecutive GGA generations not finding a new incumbent. If met, OAT switches to a JADE phase. Set to a value greater or equal the total generation number to turn off this criterion.
- --maxGenerationsPerDePhase={GENERATIONS} [int.MaxValue]
- The number of JADE iterations to execute before starting a GGA(++) phase. Set to a value greater or equal the total generation number for pure JADE.
- --minDomainSize={NUMBER} [150]
- The minimum size an integer domain needs to have to be handled as continuous. Integer parameters with a smaller number of possible values are not tuned by JADE.
- --focusOnIncumbent={BOOL} [false]
- Whether JADE should focus on improving the incumbent or work on the complete population. Focus on incumbent is not possible without GGA(++) phases and not recommended for target algorithms without any categorical parameters.
Note: If activated, JADE only works on a quarter of the complete population. - --replacementRate={RATE} [0]
- Only used if
--focusOnIncumbent=true
. Specifies the percentage of competitive genomes which get replaced by the best search points found by JADE at the end of a phase. A replacement rate of 0 indicates that only the incumbent itself should be replaced. - --fixInstances
- Ensures that the set of instances to evaluate on stays the same during a JADE phase. Setting not recommended for longer phases.
- --bestPercentage={VALUE} [0.1]
- The percentage of population members which may be used as best member in the current-to-pbest mutation strategy. Smaller values speed up convergence, but might miss good configurations. Must be in the range of (0, 1].
- --meanMutationFactor={FACTOR} [0.5]
- The initial value of the mean mutation factor. Must be in the range of [0, 1].
- --meanCrossoverRate={RATE} [0.5]
- The initial value of the mean crossover rate. Must be in the range of [0, 1].
- --learningRate={RATE} [0.1]
- The learning rate for the means. Must be in the range of [0, 1].
CMA-ES Parameters
As detailed on the page about tuning algorithms, it is possible to employ the CMA-ES parameter tuner instead of the default GGA(++) one.
Note that CMA-ES only acts on the competitive population part, i.e. it is effectively exploiting only half of the specified population size.
- --cmaEs
- Activates CMA-ES.
- --maxGenerationsPerGgaPhase={GENERATIONS} [int.MaxValue]
- The number of GGA(++) iterations to execute before starting a CMA-ES phase. Set to 0 for pure CMA-ES.
- --maxGgaGenerationsWithSameIncumbent={GENERATIONS} [int.MaxValue]
- The maximum number of consecutive GGA generations not finding a new incumbent. If met, OAT switches to a CMA-ES phase. Set to a value greater or equal the total generation number to turn off this criterion.
- --maxGenerationsPerCmaEsPhase={GENERATIONS} [int.MaxValue]
- The number of CMA-ES iterations to execute before starting a GGA(++) phase. Set to a value greater or equal the total generation number for pure CMA-ES.
- --focusOnIncumbent={BOOL} [false]
- Whether CMA-ES should focus on improving the incumbent or work on the complete population. Focus on incumbent is not possible without GGA(++) phases and not recommended for target algorithms without any categorical parameters.
- --minDomainSize={NUMBER} [150]
- Only used if
--focusOnIncumbent=true
. The minimum size an integer domain needs to have to be handled as continuous. Integer parameters with a smaller number of possible values will not be tuned. - --replacementRate={RATE} [0]
- Only used if
--focusOnIncumbent=true
. Specifies the percentage of competitive genomes which get replaced by the best search points found by CMA-ES at the end of a phase. The original incumbent is never replaced. - --fixInstances
- Ensures that the set of instances to evaluate on stays the same during a CMA-ES phase. Setting not recommended for longer phases.
- --initialStepSize={SIZE} [3]
- The step size with which to start CMA-ES phases. If using hybrid approaches, a smaller step size, e.g. 0.5, is recommended. Note that we standardize all values to fall into [0,10] at the start of a CMA-ES phase.
Worker
As OAT instances that are started as workers connect with a master and get most information from there, they need almost no parameters. Essentially, all they need to know is how to connect to the master. You may additionally change how much information they print to console.
Connection to Master
- --seedHostName={HOSTNAME} (-s)
- The host name of the node the master is running on.
- --port={NUMBER} (-p) [8081]
- The port on which the master listens for worker connections. Must be identical for master and respective workers, but different for different parallel runs.
Own Address
OAT will try to automatically detect your computing nodes' fully qualified domain names and use them for exchanging messages. If you experience connection problems on your system, you should try to set the host names explicitly.
- --ownHostName={HOSTNAME} [Fully Qualified Domain Name]
- The address that the worker uses for incoming messages. On some systems the FQDN cannot be resolved on the fly. In that case, please provide the FQDN or an IP address.
Output
- --verbose={0, 1, 2, 3} (-v) [1]
- The verbosity level. 0 only prints warnings, 1 prints some status information, 2 prints more detailed information, e.g. calls to the target mechanism, and 3 is for debugging.