Improving Automation in Developer Testing

PROJECT SUMMARY

Developer testing, a common step in software development, involves generating sufficient test inputs and checking the behavior of the program under test during the execution of the test inputs. Complicated logics inside a method make generating appropriate arguments difficult. In testing object-oriented programs, generating method sequences to put the receiver object or argument objects into appropriate states further complicates test-input generation. After the generated test inputs are executed, program crashes or uncaught exceptions can be used to indicate program problems, especially robustness problems. However, some program problems such as producing wrong program outputs do not crash the program. This project improves automation in developer testing in collaboration between the NCSU Automated Software Engineering Research Group and Microsoft Research, developing novel ideas and tools on Pex, an automatic unit testing tool from Microsoft Research. The developed tools are released as open source at the Pex Extensions project web.

COLLABORATION VISITS WITH MICROSOFT RESEARCH

• Sept 14-18, 2009: Nikolai Tillmann from the Pex group, Microsoft Research visited the NCSU ASE group
• May-August, 2009: Suresh Thummalapenta worked as a research summer intern at the Pex group, Microsoft Research
• July-August, 2009: Tao Xie worked as a visiting researcher at the Pex group, Microsoft Research
• June-July, 2008: Tao Xie worked as a visiting researcher at the Pex group, Microsoft Research
• May-June, 2007: Tao Xie worked as a visiting researcher at the Pex group, Microsoft Research

RELEVANCE TO MILITARY/DOD

• In 2009 Summer, our undergraduate student, Justin Gorham, is working as a summer intern at a Fort Hood Army team in applying Pex and our extensions on Army code bases.
  
• In 2009 Summer, our PhD student, Kunal Taneja, is working as a summer intern at FDA in applying Pex and our extensions on a DoD code base for regulatory purposes.
  

PEOPLE

Faculty
    Tao Xie

Graduate Students
    Madhuri R Marri
    Rahul Pandita
     Yoonki Song
    Kunal Taneja
    Suresh Thummalapenta
     Xusheng Xiao

Undergraduate Students
    Khiry Arnold  
    Justin Gorham
    Dennis Rankin
    Charles Woelfel

Collaborators
   Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte (Microsoft Research)

Former Members
    Nuo Li

PUBLICATIONS

1. Nuo Li, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. Reggae: Automated Test Generation for Programs using Complex Regular Expressions. In Proceedings of the 24th IEEE/ACM International Conference on Automated Software Engineering (ASE 2009), Short Paper, Auckland, New Zealand, November 2009. [PDF][BibTeX]
   
2. Suresh Thummalapenta, Tao Xie, Nikolai Tillmann, Peli de Halleux, and Wolfram Schulte. MSeqGen: Object-Oriented Unit-Test Generation via Mining Source Code. In Proceedings of the 7th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2009), Amsterdam, the Netherlands, August 2009. [PDF][BibTeX]
3. Tao Xie, Nikolai Tillmann, Peli de Halleux, and Wolfram Schulte. Fitness-Guided Path Exploration in Dynamic Symbolic Execution. To appear in Proceedings of the 39th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2009), Lisbon, Portugal, June-July 2009. [PDF][BibTeX]
4. Kunal Taneja, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. Guided Path Exploration for Regression Test Generation. To appear in Proceedings of the 31st International Conference on Software Engineering (ICSE 2009), New Ideas and Emerging Results, Vancouver, Canada, May 2009. [PDF][BibTeX]
5. Madhuri R Marri, Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. An Empirical Study of Testing File-System-Dependent Software with Mock Objects. To appear in Proceedings of the 4th International Workshop on Automation of Software Test (AST 2009), Business and Industry Case Studies, Vancouver, Canada, May 2009. [PDF][BibTeX]
6. Tao Xie, Nikolai Tillmann, Jonathan de Halleux, and Wolfram Schulte. Mutation Analysis of Parameterized Unit Tests. To appear in Proceedings of the 4th International Workshop on Mutation Analysis (Mutation 2009), Denver, Colorado, April 2009.  [PDF][BibTeX]
7. Tao Xie, Nikolai Tillmann, Jonathan de Halleux, Wolfram Schulte. Method-Sequence Exploration for Automated Unit Testing of Object-Oriented Programs. Presented at Workshop on State-Space Exploration for Automated Testing (SSEAT 2008), Seattle, WA, July 2008. [BibTeX]
8. Tao Xie, Darko Marinov, Wolfram Schulte, and David Notkin. Symstra: A Framework for Generating Object-Oriented Unit Tests using Symbolic Execution. In Proceedings of the 11th International Conference on Tools and Algorithms for the Construction and Analysis of Systems (TACAS 2005), Edinburgh, U.K., pp. 365-381, April 2005. [PDF][BibTeX][Slides]

TUTORIALS

1. Nikolai Tillmann, Jonathan de Halleux, and Tao Xie. Parameterized Unit Testing: Theory and Practice. In Proceedings of the 32nd International Conference on Software Engineering (ICSE 2010), Tutorials, Cape Town, South Africa, May 2010. [Tutorial Web][BibTeX]
2. Nikolai Tillmann, Jonathan de Halleux, Tao Xie, and Wolfram Schulte. Parameterized Unit Testing: Principles, Techniques, and Applications in Practice. To be presented at ACM SIGPLAN International Conference on Object-Oriented Programming, Systems, Languages, and Applications (OOPSLA 2009), Tutorials, Orlando, Florida, October 2009. [Tutorial Web][BibTeX]
3. Nikolai Tillmann, Jonathan de Halleux, Tao Xie, and Wolfram Schulte. Parameterized Unit Testing: Principles, Techniques, and Applications in Practice. Presented at the 31st International Conference on Software Engineering (ICSE 2009), Tutorials, Vancouver, Canada, May 2009. [Tutorial Web][BibTeX]

COURSES

1. Tao Xie created Wiki site for Teaching/Learning Parameterized Unit Testing/Pex and Parameterized Unit Testing presentation/tutorial.
2. Tao Xie taught CSC 712  Software Testing and Reliability, Graduate Course, Fall 2009, where Microsoft Research Pex, Code Contracts, and NModel were used as the primary tools and materials of the course.
   
3. Tao Xie taught CSC 712  Software Testing and Reliability, Graduate Course, Fall 2008, where Microsoft Research Pex and NModel were used as the primary tools and materials of the course. 
4. Tao Xie taught CSC510 Software Engineering, Graduate Course, Spring 2009, where where Microsoft Research Pex and CHESS were used as tools and materials for covering testing topics in the course.
   

SELECTED PRESENTATIONS

1. Tao Xie. Automated Developer Testing: Achievements and Challenges. Keynote talk, the 2nd CREST Open Workshop Software Testing (COW), CREST Centre, King's College London , December 2009.
2. Automated Developer Testing: Achievements and Challenges. Invited talk, Department of Computer Science, Virginia Polytechnic Institute and State University, Blacksburg, VA, October 2009. [Slides]
3. Automated Developer Testing: Achievements and Challenges. Invited talk, Department of Computer and Information Science, Indiana University/Purdue University at Indianapolis, Indianapolis, IN, October 2009
4. Parameterized Unit Testing: Principles, Techniques, and Applications in Practice. Invited talk, Protocol Engineering Team, Microsoft, Beijing, June 2009.
5. Fitness-Guided Path Exploration in Automated Test Generation. Invited talk, Department of Computer Science and Engineering, Michigan State University, November 2008. [Slides]
6. Fitness-Guided Path Exploration in Automated Test Generation. Invited talk, Department of Computer Science, the College of William and Mary, October 2008.
7. Fitness-Guided Path Exploration in Dynamic Symbolic Execution. Invited talk, Foundation of Software Engineering Group, Microsoft Research, July 2008.
8. Tao Xie. Automated Testing of System Software. Presented at the NCSU SOSI Year 1 review, Raleigh, NC. [Slides]

SUBPROJECTS

• Wiki site for Teaching Parameterized Unit Testing/Pex
  
• Fitnex: Fitness-guided path exploration http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0915400
• MSeqGen: Method sequence generation based on static code mining
• Regtestgen: Guided path exploration for regression test generation
• Reggae: Automated Test Generation for Programs using Complex Regular Expressions

SPONSORS

National Science Foundation Award CCF-0915400, SHF Program (09/01/2009-08/31/2012), CCF-0845272, CAREER SHF Program (08/01/2009-07/31/2014), CCF-0725190, SoD Program (01/01/2008-12/31/2010)

Army Research Office Award W911NF-08-1-0443 (09/08/2008-08/30/2011)

NCSU Secure Open Systems Initiative (SOSI) (08/15/2008-)