5.5 The Test Gap:

In present, testing techniques have gradually involved from the practice of single programmers or small

development teams into a systematic, managed engineering discipline. Not only have there been numerous

researches on testing techniques, but also more and more considerable industry practices. There are testing

classes taught in universities. There have been special testing teams, test managers, and tester job positions

open to professional testers; there have been training programs and complete procedures for testing in large

enterprises; and there are increasing number of companies and vendors doing testing work for other

companies.

14

However, despite the numerous research results (quite a lot of them are really sound) testing remains an

awkward, time -consuming, cost-ineffective chunk of work that is always not very satisfying in most

industry practices. Only a small number of the research results have been utilized successfully in industry

practices so that the test process can be greatly improved or automated. The most common testing

exercises in industry are static analysis including code inspections, peer reviews, walkthroughs. Not

enough testing tools can be applied directly on industry projects and products without being largely

modified and even re-developed. Test plans are still written by hand, while test environment remained

simple and crude. It is always the case that testing ends up being a must-end activity because the project

runs out of budget and is beyond deadline. This inconsistency of testing research and practice has been

called the “testing gap.”

Provided many other issues involved in the testing gap, such as process and management, further

researches in testing technique are among the most significant solutions that will work for the problem.

Fundamental researches in techniques need to:

·  Demonstrate effectiveness of existing techniques

·  Address the need in new areas

·  Create new adaptive techniques

·  Facilitate transferring technology to industry

5.6 Research Strategies for Testing Techniques:

We have studied the research strategies of twelve influential papers since the year 1975 and tried to find out

the common form and successful examples of research settings that offer concrete guidance for future

research work. The results are listed in Table 2. It’s not hard to find out that most of researches on testing

techniques are motivated by questioning if there is a better method of doing something. The question is

answered by inventing, implementing, combining, refining, evaluating, alternating or proposing new ways

to do this task, and the result gets analyzed through analysis most of the times. Combined with the test gap

mentioned in last section, we contend that fundamental researches should address the challenges testing

techniques are facing in the real world, generalize them, and pursue practical solutions for them. Research

should be carried out with industry partners on real world problems, instead of simple toy systems.

Researchers in academic community and in industry should talk often to address the need for each other.

Table 2. Paradigms of testing technique researches

Paper Research Paradigm

Ref. Year Age Idea

Question Result Validation

GG75 1975 26 Fundamental theorem Evaluation Analytic Model Analysis

Huang75 1975 26 Edge approach, probe insertion Method/Means Technique Analysis

Howden76 1976 25 Path approach and its reliability Characterization Technique Analysis

WC80 1980 21 Domain testing strategy Method/Means Technique Analysis

Howden80 1980 21 Functional design abstraction Method/Means Technique Persuasion

RW85 1985 16 Data flow strategy Method/Means Technique Analysis

ROT89 1989 12 Integrate spec. and impl. testing Method/Means Technique Analysis

JM94 1994 7 Coverage reliability estimation Method/Means Technique Analysis

BBL97 1997 4 Probabilistic functional testing Method/Means Technique Analysis

BIMR97 1997 4 Testing based on architectural Method/Means Technique Persuasion

HIM00 2000 1 UML based testing Method/Means Technique Experience

TBG01 2001 0 Component based testing Method/Means Technique Analysis

6 Conclusion 

testing has been widely used as a way to help engineers develop high-quality systems, and the techniques

for testing have evolved from an ad hoc activities means of small group of programmers to an organized

discipline in software engineering. However, the maturation of testing techniques has been fruitful, but not

adequate. Pressure to produce higher-quality software at lower cost is increasing and existing techniques

used in practice are not sufficient for this purpose. Fundamental research that addresses the challenging

15

problems, development of methods and tools, and empirical studies should be carried out so that we can

expect significant improvement in the way we test software. Researchers should demonstrate the

effectiveness of many existing techniques for large industrial software, thus facilitating transfer of these

techniques to practice. The successful use of these techniques in industrial software development will

validate the results of the research and drive future research. The pervasive use of software and the

increased cost of validating it will motivate the creation of partnerships between industry and researchers to

develop new techniques and facilitate their transfer to practice. Development of efficient testing techniques

and tools that will assist in the creation of high-quality software will become one of the most important

research areas in the near future.

7 Annotated bibliography

[1] G. Bernet, L. Bouaziz, and P. LeGall, “A Theory of Probabilistic Functional Testing,” Proceedings of

the 1997 International Conference on Software Engineering, 1997, pp. 216 –226

[BBL97] A framework for probabilistic functional testing is proposed in this paper. The authors

introduce the formulation of the testing activity, which guarantees a certain level of confidence into the

correctness of the system under test. They also explain how one can generate appropriate distributions

for data domains including most common domains such as intervals of integers, unions, Cartesian

products, and inductively defined sets. A tool assisting test case generation according to this theory is

proposed. The method is illustrated on a small formal specification.

Question: Method/Means

Result: Technique

Validation: Analysis

[2] B. Beizer, “Software Testing Techniques,” Second Edition, Van Nostrand Reinhold Company Limited,

1990, ISBN 0-442-20672-0

[Beizer90] This book gives a fairly comprehensive overview of software testing that emphasizes

formal models for testing. The author gives a general overview of the testing process and the reasons

and goals for testing. In the second chapter of this book, the author classifies the different types of

bugs that could arise in program development. The notion of path testing, transaction flowgraphs,

data-flow testing, domain testing, and logic-based testing are introduced in detail in the chapters

followed. The author also introduces several attempts to quantify program complexity, and more

abstract discussion involving paths, regular expression, and syntax testing. How to implement

software testing based on the strategies is also discussed in the book.

[3] S. Beydeda and V. Gruhn, “An integrated testing technique for component-based software,” ACS/IEEE

International Conference on Computer Systems and Applications, June 2001, pp 328 – 334

[BG01] Testing is made complicated with features, such as the absence of component source code, that

are specific to component-based software. The paper proposes a technique combining both black-box

and white-box strategies. A graphical representation of component software, called component-based

software flow graph (CBSFG), which visualizes information gathered from both specification and

implementation, is described. It can then be used for test case identification based on well-known

structural techniques.

Question: Method/Means

Result: Technique

Validation: Analysis