OVERVIEW AND AIM

The original inspiration of artificial intelligence (AI) was to build autonomous systems that were capable of demonstrating human-like behaviours within certain application areas. However, given the present-day data deluge, rapid increase in computational resources, and key improvements in machine learning algorithms, modern AI systems have begun to far exceed humanly achievable performance across a variety of domains. Some well known examples of this reality include IBM Watson winning Jeopardy!, and Google DeepMind’s AlphaGo beating the world’s leading Go player. Given such advances, it is deemed that what we foresee for AI in the future need no longer be limited to an anthropomorphic vision. Indeed, it may be more meaningful to build AI systems that complement and augment human intelligence, excelling at those tasks for which humans are ill-equipped. In this regard, one of the long-standing goals of AI has been to effectively multitask; i.e., learning to solve many tasks at the same time [1]. It is worth noting that although humans are generally unable to tackle multiple problems simultaneously, or within short timespans – as interleaving more than one task usually entails a considerable switching cost during which the brain must readjust from one to the other – machines are largely free from such computational bottlenecks. Thus, not only can machines move more fluidly between tasks, but, when related tasks are bundled together, it is also possible for them to seamlessly transfer data (encapsulating some problem-solving knowledge) among them. As a result, while an AI attempts to solve a complex task, several other simpler ones may be unintentionally solved. Moreover, the knowledge learned unintentionally can then be harnessed for intentional use.

In line with the above, evolutionary multitasking is an emerging concept in computational intelligence that realises the theme of efficient multi-task problem-solving in the domain of numerical optimization [2-5]. It is worth noting that in the natural world, the process of evolution has, in a single run, successfully produced diverse living organisms that are skilled at survival in a variety of ecological niches. In other words, the process of evolution can itself be thought of as a massive multi-task engine where each niche forms a task in an otherwise complex multifaceted fitness landscape, and the population of all living organisms is simultaneously evolving to survive in one niche or the other. Interestingly, it may happen that the genetic material evolved for one task is effective for another as well, in which case the scope for inter-task genetic transfers facilitates frequent leaps in the evolutionary progression towards superior individuals. Being nature-inspired optimisation procedures, it has recently been shown that evolutionary algorithms (EAs) are not only equipped to mimic Darwinian principles of “survival-of-the-fittest”, but their reproduction operators are also capable of inducing the afore-stated inter-task genetic transfers in multitask optimisation settings; although, the practical implications of the latter are yet to be fully studied and exploited in the literature. The potential efficacy of this new perspective, as opposed to traditional approaches of solving each optimisation problem in isolation, has nevertheless been unveiled by so-called multi-factorial EAs (MFEAs).

Evolutionary multitasking opens up new horizons for researchers in the field of evolutionary computation. It provides a promising means to deal with the ever-increasing number, variety and complexity of optimisation tasks. More importantly, rapid advances in cloud computing could eventually turn optimisation into an on-demand service hosted on the cloud. In such a case, a variety of optimisation tasks would be simultaneously executed by the service enginewhere evolutionary multitasking may harness the underlying synergy between multiple tasks to provide consumers with faster and better solutions.

TEST SUITES

Single-objective and multi-objective continuous optimization have been intensively studied in the community of evolutionary optimization where many well-known test suites are available. As a preliminary attempt, we have designed two MTO test suites [6],[7] for single-objective and multi-objective continuous optimization tasks, respectively.

The test suite for multi-task single-objective optimization (MTSOO) contains ten MTO complex problems ten 50-task MTO benchmark problems. Each of the complex MTO problem consists of two single-objective continuous optimization tasks, while each of the 50- task MTO problem contains 50 single-objective continuous optimization tasks, which bear certain commonality and complementarity in terms of the global optimum and the fitness landscape. These MTO problems possess different degrees of latent synergy between their involved component tasks.

The test suite for multi-task multi-objective optimization (MTMOO) includes ten MTO complex problems, and ten 50-task MTO benchmark problems. Each of the complex MTO problem consists of two multi-objective continuous optimization tasks, while each of the 50- task MTO problem contains 50 multi-objective continuous optimization tasks, which bear certain commonality and complementarity in terms of the Pareto optimal solutions and the fitness landscape. The MTO problems feature different degrees of latent synergy between their involved two component tasks.

All benchmark problems included in these two test suites are developed based on the mechanisms presented in technical reports [6] and [7], respectively. The specific benchmarks can be downloadable here.

All benchmark problems included in these two test suites will be released soon.

COMPETITION PROTOCOL

Potential participants in this competition may target at either or both of MTSOO and MTMOO while using all benchmark problems in the corresponding test suites as described above for performance evaluation.

For MTSOO test suite:

(1) Experimental settings

For each of 19 benchmark problems in this test suite, an algorithm is required to be executed for 30 runs where each run should employ different random seeds for the pseudorandom number generator(s) used in the algorithm. Note: It is prohibited to execute multiple 30 runs and deliberately pick up the best one.

For all 2-task benchmark problems, the maximal number of function evaluations (maxFEs) used to terminate an algorithm in a run is set to 200,000, while the maxFEs is set to 5,000,000 for all 50-task benchmark problems. In the multitasking scenario, one function evaluation means calculation of the objective function value of any component task without distinguishing different tasks.

Note: The parameter setting of an algorithm is required to be identical for each benchmark problem in this test suite, respectively. Participants are required to report the used parametersetting for each problem in the final submission to the competition. Please refer to “SUBMISSION GUIDELINE” for more details.

(2) Intermediate results required to be recorded

When an algorithm is executed to solve a specific benchmark problem in a run, the so far achieved best function error value (BFEV) w.r.t. each component task of this problem should be recorded when the current number of function evaluations reaches any of the predefined values which are set to k*maxFEs/Z, (k =1, …, Z; Z=100 for 2-task MTO problems and Z=1000 for 50-task MTO problems), in this competition. BFEV is calculated as the difference between the best objective function value achieved so far and the globally optimal objective function value known in advance. As a result, 100 BFEVs would be recorded for every 2-task benchmark problem, while 1000 BFEVs would be recorded for every 50-task benchmark problem, w.r.t. each component task in each run.

Intermediate results for each benchmark problem are required to be saved separately into nine “.txt” files named as "MTOSOO_P1.txt", …, "MTOSOO_P9.txt" for the nine MTO complex problems, and "MTOMSO_P1.txt", …, "MTOMSO_P10.txt" for the ten 50-task MTO benchmark problems.

  1*maxFEs/Z, BFEV_{1,1}^1,…, BFEV_{1,1}^n,……, BFEV_{30,1}^1,…, BFEV_{30,1}^n

  2*maxFEs/Z, BFEV_{1,2}^1,…, BFEV_{1,2}^n,……, BFEV_{30,2}^1,…, BFEV_{30,2}^n.

  …

  m*maxFEs/Z, BFEV_{1,m}^1,…, BFEV_{1,m}^n,……, BFEV_{30,m}^1,…, BFEV_{30,m}^n

where BFEV_{j,k}^i (i = 1, …, n ;j = 1, …, 30; k = 1, …, m) stands for the BFEV w.r.t. the ith component task obtained in the jth run at the kth predefined number of function evaluations. Note: n=2 and m=100 for 2-task benchmark problems, while n=50 and m=1000 for 50-task benchmark problems.
The first column stores the predefined numbers of function evaluations at which intermediate results are recorded. The subsequent columns store intermediate results for each of 30 runs with each run occupying n consecutive columns w.r.t. n component tasks, respectively. Note: The comma is used as a delimiter to separate any two numbers next to each other in a row. As an example, “.txt” files obtained by MFEA are provided as reference.

(3) Overall ranking criterion

To derive the overall ranking for each algorithm participating in the competition, we will take into account of the performance of an algorithm on each component task in each benchmark problem under varying computational budgets from small to large. Specifically, we will treat each component task in each benchmark problem as one individual task, ending up with a total of 518 individual tasks. For each algorithm to be ranked, the median BFEV over 30 runs will be calculated at each checkpoint which corresponds to different computational budgets for each of 518 individual tasks. Based on these calculated data, the overall ranking criterion will be defined. To avoid deliberate calibration of the algorithm to cater for the overall ranking criterion, we will release the formulation of the overall ranking criterion after the competition submission deadline.

For MTMOO test suite:

(1) Experimental settings

For each of 19 benchmark problems in this test suite, an algorithm is required to be executed for 30 runs where each run should employ different random seeds for the pseudorandom number generator(s) used in the algorithm. Note: It is prohibited to execute multiple 30 runs and deliberately pick up the best one.

For all 2-task benchmark problems, the maximal number of function evaluations (maxFEs) used to terminate an algorithm in a run is set to 200,000, while the maxFEs is set to 5,000,000 for all 50-task benchmark problems. In the multitasking scenario, one function evaluation means calculation of the values of multiple objective functions of any component task without distinguishing different tasks.
Note: The parameter setting of an algorithm is required to be identical for each benchmark problem in this test suite, respectively. Participants are required to report the used parameter setting for each problem in the final submission to the competition. Please refer to “SUBMISSION GUIDELINE” for more details.

(2) Intermediate results required to be recorded

When an algorithm is executed to solve a specific benchmark problem in a run, the obtained inverted generational distance (IGD) value w.r.t. each component task of this problem should be recorded when the current number of function evaluations reaches any of the predefined values which are set to k*maxFEs/Z, (k =1, …, Z; Z=100 for 2-task MTO problems and Z=1000 for 50-task MTO problems), in this competition. IGD [8] is a commonly used performance metric in multi-objective optimization to evaluate the quality (convergence and diversity) of the currently obtained Pareto front by comparing it to the optimal Pareto front known in advance. As a result, 100 IGD values would be recorded for every 2-task benchmark problem, while 1000 IGD values would be recorded for every 50-task benchmark problem, w.r.t. each component task in each run.

Intermediate results for each benchmark problem are required to be saved into separate ".txt" files: "MTOMOO_P1.txt", …, "MTOMOO_P9.txt" for the ten MTO complex problems, and "MTOMMO_P1.txt", …, "MTOMMO_P10.txt" for the ten 50-task MTO benchmark problems. The data contained in each ".txt" file must conform to the following format:

  1*maxFEs/Z, IGD_{1,1}^1,…, IGD_{1,1}^n,……, IGD_{30,1}^1,…, IGD_{30,1}^n

  2*maxFEs/Z, IGD_{1,2}^1,…, IGD_{1,2}^n,……, IGD_{30,2}^1,…, IGD_{30,2}^n

  ...

  m*maxFEs/Z, IGD_{1,m}^1,…, IGD_{1,m}^n,……, IGD_{30,m}^1,…, IGD_{30,m}^n

where IGD_{j,k}^i (i = 1, …, n ;j = 1, …, 30; k = 1, …, m) stands for the IGD value w.r.t. the ith component task obtained in the jth run at the kth predefined number of function evaluations. Note: n=2 and m=100 for 2-task benchmark problems, while n=50 and m=1000 for 50-task benchmark problems.
The first column stores the predefined numbers of function evaluations at which intermediate results are recorded. The subsequent columns store intermediate results for each of 30 runs with each run occupying n consecutive columns w.r.t. n component tasks, respectively. Note: The comma is used as a delimiter to separate any two numbers next to each other in a row. As an example, “.txt” files obtained by MFEA are provided as reference.

(3) Overall ranking criterion

To derive the overall ranking for each algorithm participating in the competition, we will take into account of the performance of an algorithm on each component task in each benchmark problem under varying computational budgets from small to large. Specifically, we will treat each component task in each benchmark problem as one individual task, ending up with a total of 518 individual tasks. For each algorithm compared for ranking, the median IGD value over 30 runs will be calculated at each checkpoint corresponding to different computational budgets for each of 518 individual tasks. Based on these calculated data, the overall ranking criterion will be defined. To avoid deliberate calibration of the algorithm to cater for the overall ranking criterion, we will release the formulation of the overall ranking criterion after the competition submission deadline.

SUBMISSION GUIDELINE

please archive the following files into a single .zip file and then send it to mtocompetition@gmail.com before the competition submission deadline ( 30 June 2023):

• For participants in MTSOO: 19 ".txt" files (i.e., "MTOSOO_P1.txt", … , "MTOSOO_P10.txt"), "param_SO.txt" and "code.zip".
• For participants in MTMOO: 19 “.txt” files (i.e., "MTOMOO_P1.txt", … , "MTOMOO_P9.txt"), "param_MO.txt" and "code.zip".
• For participants in both MTSOO and MTMOO: 38 ".txt" files (i.e., the required ".txt" files for both MTSOO and MTMOO), "param_SO.txt", "param_MO.txt" and "code.zip".

Here, "param_SO.txt" and "param_MO.txt" contain the parameter setting of the algorithm for MTSOO and MTMOO test suites, respectively. "code.zip" contains the source code of the algorithm which should allow the generation of reproducible results.

If you would like to participate in the competition, please kindly inform us about your interest via email (mtocompetition@gmail.com) so that we can update you about any bug fixings and/or the extension of the deadline.

COMPETITION ORGANIZERS

Liang Feng
Chongqing University, College of Computer Science, China
E-mail: liangf@cqu.edu.cn
Short Bio:
Liang Feng received the PhD degree from the School of Computer Engineering, Nanyang Technological University, Singapore, in 2014. He was a Postdoctoral Research Fellow at the Computational Intelligence Graduate Lab, Nanyang Technological University, Singapore. He is currently an Assistant Professor at the College of Computer Science, Chongqing University, China. His research interests include Computational and Artificial Intelligence, Memetic Computing, Big Data Optimization and Learning, as well as Transfer Learning. He is serving as the Chair of the IEEE Task Force on “Transfer Learning and Transfer Optimization”, and also the PC member of the IEEE Task Force on “Memetic Computing”. He had co-organized and chaired the Special Session on “Memetic Computing” held at CEC’16, CEC’17, CEC’18, CEC’19, and the Special Session on "Transfer Learning in Evolutionary Computation" held at CEC’18, CEC’19.

Kai Qin
Department of Computer Science and Software Engineering
Swinburne University of Technology, Australia
E-mail: kqin@swin.edu.au
Website: http://www.alexkaiqin.org/
Short Bio:
Kai Qin is an associate professor of Department of Computer Science and Software Engineering Swinburne University of Technology, Australia. He received the PhD degree at Nanyang Technology University (Singapore) in 2007. From 2007 to 2009, he worked as a Postdoctoral Fellow at the University of Waterloo (Waterloo, Canada). From 2010 to 2012, he worked at INRIA (Grenoble, France), first as a Postdoctoral Researcher and then as an Expert Engineer. He joined RMIT University in 2012 as a Vice-Chancellor’s Research Fellow, and then worked as a Lecturer between 2013 and 2016 and a Senior Lecturer from 2017. His major research interests include evolutionary computation, machine learning, computer vision, GPU computing and services computing. Two of his authored/coauthored journal papers have become the 1st and 4th most-cited papers among all of the papers published in the IEEE Transactions on Evolutionary Computation (TEVC) over the last 10 years according to the Web of Science Essential Science Indicators. He is the recipient of the 2012 IEEE TEVC Outstanding Paper Award. One of his conference papers was nominated for the best paper at the 2012 Genetic and Evolutionary Computation Conference (GECCO’12). He won the Overall Best Paper Award at the 18th Asia Pacific Symposium on Intelligent and Evolutionary Systems (IES’14). He is serving as the Chair of the IEEE Emergent Technologies Task Force on “Collaborative Learning and Optimization”, promoting the emerging research of the synergy between machine learning and optimization. He had coorganized and chaired the Special Session on “Differential Evolution: Past, Present and Future” held at CEC’12, CEC’13, CEC’14, CEC’15, CEC’16 and CEC’17.

Abhishek Gupta
Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR), Singapore
E-mail: abhishek_gupta@simtech.a-star.edu.sg
Short Bio:
Abhishek Gupta received the Ph.D degree in Engineering Science from the University of Auckland, New Zealand, in 2014. Over the past 5 years, Dr. Gupta has been working in the area of Memetic Computation, with particular focus on developing novel theories and algorithms in the topics of evolutionary transfer and multitask optimization. His pioneering work on evolutionary multitasking, in particular, was bestowed the 2019 IEEE Transactions on Evolutionary Computation Outstanding Paper Award by the IEEE Computational Intelligence Society (CIS). He is Associate Editor of the IEEE Transactions on Emerging Topics in Computational Intelligence, and is also the founding Chair of the IEEE CIS Emergent Technology Technical Committee (ETTC) Task Force on Multitask Learning and Multitask Optimization. He is currently appointed as a Scientist in the Singapore Institute of Manufacturing Technology (SIMTech), Agency for Science, Technology and Research (A*STAR). He also jointly serves as an Adjunct Research Scientist in the Data Science and Artificial Intelligence Research Center, School of Computer Science and Engineering, Nanyang Technological University, Singapore.

Yuan Yuan
Department of Computer Science and Engineering, Michigan State University, USA
E-mail: yyuan@msu.edu
Short Bio:
Yuan Yuan is a Postdoctoral Fellow in the Department of Computer Science and Engineering, Michigan State University, USA. He received the PhD degree with the Department of Computer Science and Technology, Tsinghua University, China, in July 2015. From January 2014 to January 2015 he was a visiting PhD student with the Centre of Excellence for Research in Computational Intelligence and Applications, University of Birmingham, UK. He worked as a Research Fellow at the School of Computer Science and Engineering, Nangyang Technological University, Singapore, from October 2015 to November 2016. His current research interests include multi-objective optimization, genetic improvement, and evolutionary multitasking. Two of his conference papers were nominated for the best paper at the GECCO 2014 and GECCO 2015, respectively.

Eric Scott
Department of Computer Science, George Mason University, USA
E-mail: escott8@gmu.edu
Short Bio:
Eric Scott is a PhD candidate at George Mason University and a Senior Artificial Intelligence Engineer at MITRE Corporation in Northern Virginia. His research focuses on heuristic optimization algorithms and their applications to simulation and modeling in a variety of fields. He holds a double B.Sc. in Computer Science and Mathematics from Andrews University in Berrien Springs, Michigan, and a M.Sc. in Computer Science from George Mason University.

Yaqing Hou
School of Computer Science and Technology, Dalian University of Technology, China
E-mail: houyq@dlut.edu.cn
Short Bio:
Yaqing Hou received the Ph.D. degree from Nanyang Technological University, Singapore, in 2017. He was a Post-Doctoral Research Fellow with the Data Science and Artificial Intelligence Research Centre, Nanyang Technological University. Currently, he is an Assistant Professor with the School of Computer Science and Technology, Dalian University of Technology, Dalian, China. His current research interests include computational intelligence, memetic computing, multiagent learning systems. His publications have appeared in IEEE Transactions on Evolutionary Computation, IEEE Computational Intelligence Magazine, IEEE Transactions on Systems, Man, and Cybernetics: Systems, AAMAS, etc. He is currently an Associate Editor in Memetic Computing Journal.

Yew-Soon Ong
School of Computer Science and Engineering, Nanyang Technological University, Singapore
E-mail: asysong@ntu.edu.sg
Website: http://www.ntu.edu.sg/home/asysong/
Short Bio:
Yew-Soon Ong is Professor and Chair of the School of Computer Science and Engineering, Nanyang Technological University, Singapore. He is Director of the A*Star SIMTECH-NTU Joint Lab on Complex Systems and Programme Principal Investigator of the Data Analytics & Complex System Programme in the Rolls-Royce@NTU Corporate Lab. He was Director of the Centre for Computational Intelligence or Computational Intelligence Laboratory from 2008-2015. He received his Bachelors and Masters degrees in Electrical and Electronics Engineering from Nanyang Technological University and subsequently his PhD from University of Southampton, UK. He is founding Editor-In-Chief of the IEEE Transactions on Emerging Topics in Computational Intelligence, founding Technical EditorIn-Chief of Memetic Computing Journal (Springer), Associate Editor of IEEE Computational Intelligence Magazine, IEEE Transactions on Evolutionary Computation, IEEE Transactions on Neural Network & Learning Systems, IEEE Transactions on Cybernetics, IEEE Transactions on Big Data, International Journal of Systems Science, Soft Computing Journal, and chief editor of Book Series on Studies in Adaptation, Learning, and Optimization as well as Proceedings in Adaptation, Learning, and Optimization He is also guest editors of IEEE Transactions on Evolutionary Computation, IEEE Trans SMC-B, Soft Computing Journal, Journal of Genetic Programming and Evolvable Machines, co-edited several books, including Multi-Objective Memetic Algorithms, Evolutionary Computation in Dynamic and Uncertain Environments, and a volume on Advances in Natural Computation published by Springer Verlag. He served as Chair of the IEEE Computational Intelligence Society Emergent Technology Technical Committee (ETTC) from 2011-2012, and has been founding chair of the Task Force on Memetic Computing in ETTC since 2006 as well as a member of IEEE CIS Evolutionary Computation Technical Committee from 2008 - 2010. He was also Chair of the IEEE Computational Intelligence Society Intelligent Systems Applications Technical Committee (ISATC) from 2013-2014. His current research interests include computational intelligence spanning memetic computing, evolutionary optimization using approximation/surrogate/meta-models, complex design optimization, intelligent agents in game, and Big Data Analytics. His research grants comprises of external funding from both national and international partners that include National Grid Office, A*STAR, Singapore Technologies Dynamics, Boeing Research & Development (USA), Rolls-Royce (UK) and Honda Research Institute Europe (Germany), National Research Foundation and MDAGAMBIT. His research work on Memetic Algorithm was featured by Thomson Scientific's Essential Science Indicators as one of the most cited emerging area of research in August 2007. Recently, he was selected as a 2015 Thomson Reuters Highly Cited Researcher and 2015 World's Most Influential Scientific Minds. He also received the 2015 IEEE Computational Intelligence Magazine Outstanding Paper Award and the 2012 IEEE Transactions on Evolutionary Computation Outstanding Paper Award for his work pertaining to Memetic Computation.

REFERENCES

[1] R. Caruana, “Multitask learning”, Machine Learning, 28(1): 41-75, 1997.
[2] K. Swersky, J. Snoek and R. P. Adams, “Multi-task Bayesian optimization”, Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS'13), pp. 2004-2012, Lake Tahoe, Nevada, USA, December 5-10, 2013.
[3] A. Gupta, Y. S. Ong and L. Feng, “Multifactorial evolution: Toward evolutionary multitasking”, IEEE Transactions on Evolutionary Computation, 20(3):343-357, 2016.
[4] Y. S. Ong and A. Gupta, “Evolutionary multitasking: A computer science view of cognitive multitasking”, Cognitive Computation, 8(2): 125-142, 2016.
[5] A. Gupta, Y. S. Ong, L. Feng and K. C. Tan, “Multi-objective multifactorial optimization in evolutionary multitasking”, accepted by IEEE Transactions on Cybernetics, 2016.
[6] B. S. Da, Y. S. Ong, L. Feng, A. K. Qin, A. Gupta, Z. X. Zhu, C. K. Ting, K. Tang and X. Yao, “Evolutionary multitasking for single-objective continuous optimization: Benchmark problems, performance metrics and baseline results”, Technical Report, Nanyang Technological University, 2016.
[7] Y. Yuan, Y. S. Ong, L. Feng, A. K. Qin, A. Gupta, B. S. Da, Q. F. Zhang, K. C. Tan, Y. C. Jin and H. Ishibuchi, “Evolutionary multitasking for multi-objective continuous optimization: Benchmark problems, performance metrics and baseline results”, Technical Report, Nanyang Technological University, 2016.
[8] P. Czyzzak and A. Jaszkiewicz, “Pareto simulated annealing– a metaheuristic technique for multiple-objective combinatorial optimization”, Journal of Multi-Criteria Decision Analysis, 7:34-47, 1998.