7. DISCUSSION
7.3. Concluding Remarks
This work describes the first set of case studies that systematically and comparatively quantified how peer reviews are performed on successful, mature OSS projects. The work is based on Rigby et al. [2008], Rigby’s [2011] dissertation, and an earlier techni- cal report by Rigby and German [2006]. Prior to this work, there were only anecdotal descriptions and limited quantitative analyses of OSS peer review practices. There are three contributions made in this work. First, we developed measures from email and version control archives that mirrored the measures used over the last 35 years to as- sess software inspection in proprietary development. From this unstructured data, we measured the frequency of review, the level of participation in reviews, the experience and expertise of the reviewers, the size of the artifact under review, the calendar time to perform a review, and the number of issues that are discussed during review. Second, we examined the review policies of a 25 successful OSS projects and conducted a detailed quantitative analysis on the Apache httpd Server, Subversion, Linux, FreeBSD, KDE, and Gnome. Finally, we created statistical models of the efficiency and effectiveness of OSS review practices. However, descriptive statistics alone are enough to see that the parameters of OSS peer review are drastically different from traditional software inspection. We summarized our observations as a theory of OSS peer review practices.
A detailed qualitative analysis involving manual coding of reviews and interviews of top reviewers on the same six projects can be found in Rigby and Storey [2011].
OSS peer review and inspection as envisioned by Fagan in 1976 have little in common beyond a belief that peers will effectively find software defects in software artifacts. In
contrast to top-down development practices, OSS review practices evolved organically from the needs of the core development team. The recent inspection literature and our findings regarding OSS review imply that the formality and rigidity of Fagan inspections is unnecessary. Peer review practices that are conducted asynchronously, that empower experts, that provide timely feedback on small changes, and that allow developers to focus on their area of expertise are more efficient than formal peer review techniques and are still able to detect issues.
We have begun to investigate possible transfers of our findings to other develop- ment environments. An IEEE Software paper orients our findings toward practition- ers [Rigby et al. 2012]. For example, we suggest that dividing reviews into smaller, independent, complete pieces may reduce the burden placed on any individual re- viewer and divide the often unpleasant review task into more manageable chunks that can be conducted periodically throughout the day. These changes might result in indus- trial developers taking a more positive view of peer review. We are working with the Canadian Department of Defense to develop peer review practices that are lightweight while still retaining a high degree of traceability and quality assurance.
REFERENCES
A. Ackerman, L. Buchwald, and F. Lewski. 1989. Software inspections: An effective verification process.
IEEE Softw.6, 3, 31–36.
J. Asundi and R. Jayant. 2007. Patch review processes in open source software development communities: A comparative case study. InProceedings of the 40th Annual Hawaii International Conference on System Sciences (HICSS’07). 10.
V. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sørumg ˚ard, and M. Zelkowitz. 1996. The empirical investigation of perspective-based reading.Empir. Softw. Eng.1, 2, 133–164.
C. Bird, A. Gourley, and P. Devanbu. 2007. Detecting patch submission and acceptance in OSS projects. In Proceedings of the 4th International Workshop on Mining Software Repositories (MSR). IEEE Computer Society, 4.
C. Bird, A. Gourley, P. Devanbu, M. Gertz, and A. Swaminathan. 2006. Mining email social networks. In Proceedings of the 3rd International Workshop on Mining Software Repositories (MSR). ACM Press, 137–143.
D. Bisant and J. Lyle. 1989. A two-person inspection method to improve programming productivity.IEEE Trans. Softw. Eng.15, 10, 1294–1304.
S. Breu, R. Premraj, J. Sillito, and T. Zimmermann. 2010. Information needs in bug reports: Improving cooperation between developers and users. InProceedings of the ACM Conference on Computer Supported Cooperative Work. ACM Press, 301–310.
F. Buck. 1981. Indicators of quality inspections. Tech. rep. TR 21, IBM Syst. Commun. Division.
K. Burnham and D. Anderson. 2002.Model Selection and Multimodel Inference: A Practical Information- Theoretic Approach. Springer Verlag.
A. Cockburn. 2004. Crystal Clear a Human-Powered Methodology for Small Teams. Addison-Wesley Professional.
J. Cohen. 2006.Best Kept Secrets of Peer Code Review. Smart Bear Inc.
M. Crawley. 2005.Statistics: An Introduction Using R. John Wiley and Sons.
T. Dinh-Trong and J. Bieman. 2005. The FreeBSD project: A replication case study of open source develop- ment.IEEE Trans. Softw. Eng.31, 6, 481–494.
P. Eggert, M. Haertel, D. Hayes, R. Stallman, and L. Tower. 2002.diff – Compare files line by line. Free Software Foundation, Inc.
S. G. Eick, C. R. Loader, M. D. Long, L. G. Votta, and S. V. Wiel. 1992. Estimating software fault content before coding. InProceedings of the 14th International Conference on Software Engineering. 59–65.
M. Fagan. 1986. Advances in software inspections.IEEE Trans. Softw. Eng.12, 7, 744–751.
M. Fagan. 2002. A history of software inspections. InSoftware Pioneers: Contributions to Software Engineer- ing,Springer-Verlag, Inc., 562–573.
M. E. Fagan. 1976. Design and code inspections to reduce errors in program development.IBM Syst. J.15, 3, 182–211.
R. T. Fielding and G. Kaiser. 1997. The Apache HTTP server project.IEEE Internet Comput.1, 4, 88–90.
K. Fogel. 2005.Producing Open Source Software. O’Reilly.
Freshmeat. 2009. About Freshmeat: A catalog of software projects. http://freshmeat.net/about (Last accessed September 2009).
D. German. 2007. Using software distributions to understand the relationship among free and open source software projects. InProceedings of the 4th International Workshop on Mining Software Repositories (MSR).IEEE, 8.
C. Gutwin, R. Penner, and K. Schneider. 2004. Group awareness in distributed software development. In Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW). 72–81.
S. Hambridge. 1995. Netiquette guidelines.http://edtech2.boisestate.edu/frankm/573/netiquette.html.
R. Hartill. 1998. Email. http://mail-archives.apache.org/mod mbox/httpd-dev/199801.mbox%3CPine.
NEB.3.96.980108232055.19805A-100000@localhost%3E.
L. Hatton. 2008. Testing the value of checklists in code inspections.IEEE Softw.25, 4, 82–88.
H. Heinzl and M. Mittlb¨ock. 2003. Pseudo r-squared measures for poisson regression models with over- or underdispersion.Computat. Stat. Data Analy.44, 1–2, 253–271.
A. Hindle, M. W. Godfrey, and R. C. Holt. 2008. Reading Beside the Lines: Indentation as a proxy for complexity metric. InProceedings of the 16th IEEE International Conference on Program Comprehension (ICPC). IEEE Computer Society, 133–142.
J. Howison, M. Conklin, and K. Crowston. 2006. FLOSSmole: A collaborative repository for FLOSS research data and analyses.Int. J. Inf. Techno. Web Eng.1, 17–26.
G. Jeong, S. Kim, T. Zimmermann, and K. Yi. 2009. Improving code review by predicting reviewers and accep- tance of patches. ROSAEC Memo 2009-006, Research on Software Analysis for Error-Free Computing.
P. M. Johnson. 1998. Reengineering inspection.Commun. ACM41, 2, 49–52.
P. M. Johnson and D. Tjahjono. 1998. Does every inspection really need a meeting?Empir. Softw. Eng.3, 1, 9–35.
P. Kampstra. 2008. Beanplot: A boxplot alternative for visual comparison of distributions.J. Stat. Softw.
Code Snippets1 28, 1–9.
P. Kennedy. 1981. Estimation with correctly interpreted dummy variables in semilogarithmic equations.
Am. Econ. Rev.71, 4, 801.
J. C. Knight and E. A. Myers. 1993. An improved inspection technique.Commun. ACM 36, 11, 51–61.
S. Kollanus and J. Koskinen. 2009. Survey of software inspection research.Open Softw. Eng. J.3, 15–34.
O. Laitenberger and J. DeBaud. 2000. An encompassing life cycle centric survey of software inspection.
J. Syst. Softw.50, 1, 5–31.
C. Larman and V. Basili. 2003. Iterative and incremental developments: A brief history.Computer36, 6, 47–56.
G. Lee and R. Cole. 2003. From a firm-based to a community-based model of knowledge creation: The case of the Linux kernel development.Org. Sci.14, 6, 633–649.
S. Lussier. 2004. New tricks: How open source changed the way my team works.IEEE Softw.21, 1, 68–72.
F. Macdonald and J. Miller. 1999. A comparison of computer support systems for software inspection.Autom.
Softw. Eng.6, 3, 291–313.
J. Maindonald and J. Braun. 2003.Data Analysis and Graphics Using R: An Example-Based Approach.
Cambridge University Press.
J. Martin and W. T. Tsai. 1990. N-Fold inspection: A requirements analysis technique.ACM Commun.33, 2, 225–232.
V. Mashayekhi, C. Feulner, and J. Riedl. 1994. CAIS: Collaborative asynchronous inspection of software.
InProceedings of the 2nd ACM SIGSOFT Symposium on Foundations of Software Engineering. ACM Press, 21–34.
P. McCullagh and J. Nelder. 1989.Generalized Linear Models(Monographs on Statistics and Applied Prob- ability Book 37). Chapman & Hall/CRC.
A. Mockus, R. Fielding, and J. Herbsleb. 2000. A case study of open source software development: The Apache server. InProceedings of the 22nd International Conference on Software Engineering (ICSE). 262–273.
A. Mockus, R. T. Fielding, and J. Herbsleb. 2002. Two case studies of open source software development:
Apache and Mozilla.ACM Trans. Softw. Eng. Methodo.11, 3, 1–38.
A. Mockus and J. D. Herbsleb. 2002. Expertise browser: A quantitative approach to identifying expertise.
InProceedings of the 24th International Conference on Software Engineering (ICSE’05).ACM Press, 503–512.
N. Nagappan and T. Ball. 2005. Use of relative code churn measures to predict system defect density. In Proceedings of the 27th International Conference on Software Engineering (ICSE). IEEE, 284–292.