Thesis Defense - Şahin Dirim (MSCS)

Şahin Dirim - M.Sc. Computer Science

Assoc. Prof. Hasan Sözer – Advisor

Date: 08.06.2021

Time: 12:00

Location: This meeting will be held ONLINE. Please send an e-mail to gizem.bakir@ozyegin.edu.tr in order to participate in this defense.

Prioritization and Parallel Execution of Test Cases for Certification Testing

Thesis Committee:

Assoc.Prof. Hasan Sözer, Özyeğin University

Assoc. Prof.O. Orsan Ozener, Özyeğin University

Assoc. Prof. Geylani Kardaş, Ege University

Abstract:

Systems that are developed in the consumer electronics domain are subject to testing for certification regularly. For instance, each Smart TV product must go through a certification process that is regulated by application vendors like Netflix, YouTube and Amazon. There exist a separate test suite to be used as part of the certification process pertaining to each application. Each test suite includes hundreds of test cases. Many of these test cases have to be executed manually and it might take several days to complete the execution of certification test suites. There also exists a high variation among the execution times of test cases and severities of faults detected by these test cases. Therefore, prioritization of test cases and parallel test execution can significantly impact the duration and effectiveness of the process. We measure test effectiveness by calculating the average of the percentage of faults detected. In this thesis, we introduce variants of existing metrics to take parallel test execution, varying test execution times, and fault severity levels in ordinal scale into account. We also propose an approach for prioritizing such test cases that can be executed on multiple test stations in parallel. We employ integer linear programming for optimizing the allocation of test execution tasks on a set of available test stations. The goal of this allocation is to minimize the overall test execution time. Test cases that are allocated to particular stations are executed in increasing order of their execution times. We conducted an industrial case study in the context of certification testing for Smart TV software. We used certification test suites of 3 Smart TV applications applied on 3 TV software projects as real experimental objects. We repeated our measurements for two test cycles for each project, adding up to 6 tests performed for each of the 3 test suites. So, our evaluation involves 18 test sessions. We evaluated 3 scenarios for each of these sessions, where the number of available test stations are 1, 3 and 5. We compared the effectiveness of our approach with respect to a greedy approach as the baseline. We observed that the overall test execution time can be reduced by up to 16\% even when only 3 test stations are available. Test effectiveness is also improved as a result of the optimal scheduling of test cases.

Bio: 

Şahin Dirim received his B.Sc. degree in computer engineering from Ege University, Turkey, in 2017. He has been employed as a Software Senior Test Specialist at Vestel Electronics R&D since 2017. He is working on Test automation, OTT apps(Netflix, YouTube and Prime Video) Certification, HBB Tv certification and Smart TV Test Process management areas at Vestel. Besides, he started his master's degree at Ozyegin University in 2018.