• Survey on Industrial Practice for Evaluating Quality of Open Source Software

  • Survey on Industrial Practice for Evaluating Quality of Open Source Software 

    Main Researcher: Nebi YILMAZ

    As open project repositories have become widespread, including open source software (OSS) as parts of software products and thus, evaluating OSS quality have gained attention. However, methods to evaluate OSS quality are varying and not straightforward in industry.

    You are invited to participate in this online survey about whether well-defined guidance for evaluating OSS quality would be beneficial to use while evaluating OSS quality in practice. This survey is conducted by Nebi YILMAZ (PhD candidate) under supervision of Assoc. Prof. Dr. Ayça KOLUKISA-TARHAN, at Computer Engineering Department of Hacettepe University.

    People who have knowledge and experience about at least two of the following are expected to participate in this survey: (i) OSS products, (ii) quality models, (iii) product quality evaluation or assessment, and (iv) meta-modeling.

  • Would you like to have further information about the survey?
  • CONTEXT

    Recent literature reviews by us indicated that there are 36 quality models proposed for open source software (OSS) [1] but lack of a well-defined meta-model [2]. Thus, by this survey, we are asking industrial feedback about whether the existence of a meta-model for evaluating OSS quality would be beneficial to use while evaluating OSS quality in practice. With this purpose, we have prepared a total of 26 questions that address: experiences of the participants, challenges faced in evaluating OSS quality, and whether there is a need for well-defined guidance.

    PARTICIPATION

    Your participation in this survey is voluntary, and also very valuable to understand the challenges that practitioners and/or researchers face in industry while evaluating OSS quality. You may leave the survey any time after you start. Also, you may freely decline answering a particular question that may feel you uncomfortable.

    BENEFITS

    You will not benefit financially from your participation in this survey. However, it is likely that you will feel good about sharing your unique experience on the topic and helping the community via our research. In a broader view, this may create opportunity for gathering the challenges faced while evaluating OSS quality and understanding the importance of a quality meta-model as a means of standardizing OSS quality evaluation. More specifically, the results of this survey may be useful for: (i) software researchers to create their OSS quality models in a standardized way, and (ii) software practitioners to evaluate quality of candidate OSS products that they plan to integrate into their projects in a systematic, quantifiable, and comparable way.

    COSTS

    Participating in this research will take 10-15 minutes of your time and will not cost you anything else. Thank you in advance for allocating your valuable time.

    CONFIDENTIALITY

    Only the researchers responsible for this study will have access to your answers. Your e-mail address will be stored for the sole purpose of sending you the final results of this research. Your e-mail address will be kept confidential and will not be disclosed by any means. Your e-mail address will be used only for the purpose of sending you the survey results. Information collected through this survey will only be used for research purposes, analyzed collectively, and anonymously disclosed in research reports.

    The answers to the survey will be stored in JotForm.com. Access to information is password protected and secured. We will not share your name, IP address, or any information that shall personally identify you. We assure you that any information we receive will be held in trust and confidence.

    CONTACT

    If you have questions, you may get in touch with us anytime at +90 (546) 948 22 02, or via email at yilmaz@cs.hacettepe.edu.tr.

    1- Yılmaz, N., and Tarhan, A.K.: Quality evaluation models or frameworks for open source software: A systematic literature review. J Softw Evol Proc. 2022;34(6):e2458. doi:10.1002/smr.2458, (2022).

    2- Yılmaz, N., and Tarhan, A.K.: Meta-models for Software Quality and Its Evaluation: A Systematic Literature Review. In: International Workshop on Software Measurement and the 15th International Conference on Software Process and Product Measurement, Mexico, (2020).

  • Personal Information

  • It is not mandatory to fill in the following personal information.

    Also, it is not mandatory to answer all survey questions.

  • Questions

  • Q1- What is the size of your company?
  • Q8- Do you have any previous experience with selecting the wrong OSS?
  • Q9- Do you employ a quality model in choosing the ‘right’ OSS to use in your company?
  • Q13- Do you think it is necessary that an OSS quality model enables quality evaluation from different aspects (i.e., code-based and community-based aspects)? – Some example OSS quality models are: OSMM [3], Open BRR [4], SQO-OSS [5], and QualOSS [6].

  • Please open the links in a new page:

    3- Duijnhouwer, F.W., Widdows, C.: Capgemini Expert Letter Open Source Maturity Model, Capgemini, URL: click here, (2003).

    4- Wasserman, M.P., Chan, C.: Business Readiness Rating Project, BRR Whitepaper RFC 1, URL: click here, (2006).

    5- Samoladas, I., Goussios G. and Spinellis, D.: The SQO-OSS quality model: measurement based open source software evaluation. In: IFIP international conference on open source systems. Springer, Boston, MA, URL: click here (2008).

    6- Soto, M., and Ciolkowski, M.: The QualOSS open source assessment model measuring the performance of open source communities. In: 3rd International Symposium on Empirical Software Engineering and Measurement. IEEE, URL: click here (2009).

  • Q14- A number of OSS quality models are already available. Have you employed (or at least put some effort to employ) any ofthese models (or similar ones) for evaluating OSS quality in your company?
  • Q15- Do you think it is necessary that an expert with the skills to use an OSS quality model is present in your company?
  • Q15.1- Do you have such an expert in your company?
  • Q17- Do you think it is necessary that an OSS quality model generates standardized measurement results? (i.e., measurement results obtained via different quality evaluations for the same product are comparable)
  • Q18- Do you think it is necessary that different persons, who use a same OSS quality model to evaluate a same OSS product, produce measurement results that are comparable?
  • Q19- Are you familiar about existing OSS quality models? (e.g., OSMM [3],Open BRR [4], SQO-OSS [5], and QualOSS [6])
  • Q19.2- Do you think there are inconsistencies in these models with respect to their structures or terms?(e.g., a term used in one quality model is substituted for a different term in another quality model)

  • Meta-model and Tool-support

    Meta-models are defined as “models of models” with the rules needed to instantiate specific models from more abstract, generic models. Specifically, software quality meta-models are important because they are used to derive quality models that have homogenous structure and common terms. By this way, meta-models help to disseminate related knowledge and create a common understanding among stakeholders (e.g., practitioners or researchers). They also guide stakeholders in specifying and evaluating software quality, eliminating the ambiguities involved in the process, and performing comparable measurements. In other words, the software quality meta-models support standardization as an important communication vehicle to stakeholders when interoperating with others.

  • Q20- Do you think the existence of a meta-model for OSS quality, which allows to derive standardized OSS quality models while evaluating OSS quality, would be beneficial to use in your company?
  • Should be Empty: