All posts by carminevassallo

About carminevassallo

PhD student at University of Zurich. Interested in supporting DevOps during continuous software development.

Principle-Driven Continuous Integration: Simplifying Failure Discovery and Raising Anti-Pattern Awareness

Public PhD Thesis Defense of Carmine Vassallo

Advisor: Prof. Dr. Harald C. Gall
2nd Advisor: Prof. Dr. Laurie Williams
3rd Advisor: Prof. Dr. Sebastian Proksch

Chair: Prof. Dr. Davide Scaramuzza

Date and time: Friday, September 18, 2020, 16:00 h
Location:  remotely via “Zoom” (link expired)

Extended Abstract: Continuous Integration (CI) is a software development practice that enables developers to build reliable software faster. Given its proven benefits, such as increased developer productivity and higher release frequency, most organizations have started adopting CI. This practice advocates full automation of all build steps (i.e., compilation, testing, and code quality assessment) to create a new version of the software. However, the mere introduction of an automated build infrastructure is not sufficient to practice CI well and to achieve its goals. Organizations have also to foster the application of several principles, such as commit often, that reduce conflicts in the team and ensure that the build is continuously executable. Living up to these principles is not easy especially when developers face tough deadlines. As a consequence, developers tend to deviate from principles generating anti-patterns, which are ineffective solutions to recurrent problems. Anti-patterns appear to be beneficial, but, in the end, they let CI decay and lower its effectiveness. In this dissertation, we characterize the problem of anti-patterns to implement solutions that help developers remove the root causes of anti-patterns and, therefore, follow principles. Based on the results of a preliminary study performed on opensource projects revealing the existence of deviations from core principles, we empirically derive a catalog of 79 anti-patterns encountered by developers in practice conducting semi-structured interviews with 13 practitioners and manually analyzing 2,300 posts from a well-know forum (i.e., StackOverflow) where users discuss issues related to the adoption of CI. By interpreting the resulting catalog of anti-patterns, we identify four main causes for their presence, which are (i) the poor knowledge of prerequisites for adopting CI, (ii) the difficulty of inspecting build failure logs, (iii) the presence of bad configurations, and (iv) the wrong usage of a CI process. While only better coaching in CI can efficiently remove the former, we implement several approaches to address the other causes. To improve the understandability of build failure logs, we first build a taxonomy of build failures through the manual analysis of errors contained in 34,182 build logs from open-source and closed-source projects, and then we develop a tool called BART that produces summaries for the most common build failure types. We evaluate the performance of our tool in a controlled experiment with 17 developers. To identify violations of CI principles in the form of configuration smells that developer should remove, we propose CDLinter, a semantic linter for CI/CD configuration files that is evaluated opening 145 issues in open-source projects and monitoring the acceptance of our bug reports and the removal of reported smells over a period of six months. Finally, we implement CI-Odor, an automated reporting tool that leverages information from repository and build history to monitor the presence of bad practices that slowly creep into a project over time. We evaluate its usefulness sending developers reports produced for 36 open-source projects. The results of our evaluations show that the proposed approaches are effective at removing and identifying the aforementioned causes of anti-patterns and, consequently, enforcing a principle-driven continuous integration. BART improves the understandability of the most common build failure types and developers are faster in solving build failures. In presence of build summaries, the resolution time is reduced by 23% when solving testing failures, 20% when repairing compilation errors, 43% when fixing missing dependencies, and 62% when dealing with code analysis failures. CD-Linter identifies smells that are relevant for developers. During the 6-month observation period, 53% of the project maintainers react positively to the issues detected by CD-Linter, with 9% that confirm the validity of the reported problem and 44% that fix it. Finally, the reports generated by CI-Odor are useful for monitoring anti-patterns. Many developers (67%) expect a positive effect of using our generated reports on their CI discipline and the majority (55%) is already willing to integrate CI-Odor in their CI processes.


Survey on how Developers React to CI Build Failures

Among the benefits provided by Continuous Integration (CI), increased team productivity and integration frequency are perceived as the main advantages. However, changes that contain defects or that suffer from a poor-quality can lead to build failures that stop a team from delivering. The recent Report on the State of DevOps states: “When failures occur, it can be difficult to understand what caused the problem” and previous work found that developers spend on average one hour to fix build breaks!

In our group at the University of Zurich (Switzerland), we are developing new strategies to provide developers with the right assistance to solve build failures faster and more efficiently. To achieve this, we first need to understand the state of practice from real developers and we would like to learn about your personal experience with build failures in this survey.

We would really appreciate if you could find the time to fill out the following survey to help us in our research.

Filling out the survey will take you about 7 minutes. Please note that participating in the questionnaire is completely anonymous, but we will publish the anonymized answers as part of a scientific publication.

If you have any questions about the questionnaire or our research, please do not hesitate to contact us.

Image from “Arduino Jenkins CI build monitor using car lights”, Gordons Garage, YouTube, 2016.

seal @ ICSME 2017

We are very happy to announce that our research group got two papers accepted at ICSME 2017 in Shanghai, China.

The first paper is entitled “A Tale of CI Build Failures: an Open Source and a Financial Organization Perspective” and was written in collaboration with ING Nederland, University of Sannio and TU Delft. The authors of the paper are: Carmine Vassallo, Gerald Schermann, Fiorella Zampetti, Daniele Romano, Philipp Leitner, Andy Zaidman, Massimiliano Di Penta and Sebastiano Panichella.

Abstract: Continuous Integration (CI) and Continuous Delivery (CD) are widespread in both industrial and open-source software (OSS) projects. Recent research characterized build failures in CI and identified factors potentially correlated to them. However, most observations and findings of previous work are exclusively based on OSS projects or data from a single industrial organization. This paper provides a first attempt to compare the CI processes and occurrences of build failures in 349 Java OSS projects and 418 projects from a large financial organization, ING Nederland.


Through the analysis of 34,182 failing builds (26% of the total number of observed builds), we derived a taxonomy of failures that affect the observed CI processes. Using cluster analysis, we observed that in some cases OSS and ING projects share similar build failure patterns (e.g., few compilation failures as compared to frequent testing failures), while in other cases completely different patterns emerge. In short, we explain how OSS and ING CI processes exhibit commonalities, yet are substantially different in their design and in the failures they report.


The second accepted paper is entitled “Towards Activity-Aware Tool Support for Change Tasks” and was written by Katja Kevic and Thomas Fritz.

Abstract: To complete a change task, software developers perform a number of activities, such as locating and editing the relevant code. While there is a variety of approaches to support developers for change tasks, these approaches mainly focus on a single activity each. Given the wide variety of activities during a change task, a developer has to keep track of and switch a lot between the different approaches.


By knowing more about a developer’s activities and in particular by knowing when she is working on which activity, we would be able to provide better and more tailored tool support, thereby reducing developer effort. In our research we investigate the characteristics of these activities, whether they can be identified, and whether we can use this additional information to improve developer support for change tasks. We conducted two exploratory studies with a total of 21 software developers collecting data on activities in the lab and field. An empirical analysis of the data shows, amongst other results, that activities comprise a consistently small amount of code elements across all developers and tasks (approx. 8.7 elements). Further analysis of the data shows, that we can automatically detect the boundaries and types of activities, and that the information on activity types can be used to improve the identification of relevant code elements.


Survey on Static Analysis Tools

Our research group together with the Delft University of Technology (The Netherlands), is conducting a research project aimed at understanding the Usage of Static Analysis Tools (e.g., SonarQube, PMD, Checkstyle) in practice.

Given your solid experience as developer/engineer/researcher, we kindly ask you to fill in the following (brief) survey:

It will take approximately 10-12 minutes of your time. We will handle all responses confidentially and anonymize all collected data before publishing it.

Thanks in advance for your willingness in participating in this survey and…have fun!

Please do not hesitate to contact us if you have any further questions.