Testing Methodology

Usage " It involves testing of all the functions performed by the people while preparing the data and using these data from automated system.

Objectives " Verify - manual support documents and procedures are correct.

" Determine -

Manual support responsibility is correct

Manual support people are adequately trained.

Manual support and automated segment are properly interfaced.

How to Use " It involves:

Evaluation of adequacy of process

Execution of process

" Process evaluated in all segments of SDLC.

" Execution of the can be done in conjunction with normal system testing.

" Instead of preparing, execution and entering actual test transactions the clerical and supervisory personnel can use the results of processing from application system.

" It involves several iterations of process.

" To test people it requires testing the interface between the people and application system.

When to use " Verification that manual systems function properly should be conducted throughout the SDLC.

" Should not be done at later stages of SDLC.

" Best done at installation stage so that the clerical people do not get used to the actual system just before system goes to production.

Examples " Provide input personnel with the type of information they would normally receive from their customers and then have them transcribe that information and enter it in the computer.

" Users can be provided a series of test conditions and then asked to respond to those conditions. Conducted in this manner, manual support testing is like an examination in which the users are asked to obtain the answer from the procedures and manuals available to them.

Background " Application systems are frequently interconnected to other application system.

" The interconnection may be data coming from another application system, leaving for another application system or both.

" Frequently multiple systems (applications) sometimes called cycles or functions are involved.

Usage " To ensure interconnection between application functions correctly.

Objectives " Determining -

Proper parameters and data are correctly passed between the applications

Documentation for involved system is correct and accurate.

" Ensure Proper timing and coordination of functions exists between the application system.

How to Use " Operations of multiple systems are tested.

" Multiple systems are run from one another to check that they are acceptable and processed properly.

When to use " When there is change in parameters in application system

" The parameters, which are erroneous then risk associated to such parameters, would decide the extent of testing and type of testing.

" Intersystem parameters would be checked / verified after the change or new application is placed in the production.

Examples " Develop test transaction set in one application and passing to another system to verify the processing.

" Entering test transactions in live production environment and then using integrated test facility to check the processing from one system to another.

" Verifying new changes of the parameters in the system, which are being tested, are corrected in the document.

Disadvantage " Time consuming

" Cost may be expensive if system is run several times iteratively.

Software testing is more than just error detection;

Testing software is operating the software under controlled conditions, to (1) *verify* that it behaves "as specified"; (2) to *detect* *errors*, and (3) to *validate* that what has been specified is what the user actually wanted.

1. *Verification* is the checking or testing of items, including software, for conformance and consistency by evaluating the results against pre-specified requirements. [*V**erification: Are we building the system right?*]

2. *Error Detection*: Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn't or things don't happen when they should.

3. *Validation* looks at the system correctness – i.e. is the process of checking that what has been specified is what the user actually wanted. [*Validation: Are we building the right system?*]

In other words, validation checks to see if we are building what the customer wants/needs, and verification checks to see if we are building that system correctly. Both verification and validation are necessary, but different components of any testing activity.

The definition of testing according to the ANSI/IEEE 1059 standard is that testing is the process of analysing a software item to detect the differences between existing and required conditions (that is defects/errors/bugs) and to evaluate the features of the software item. Remember: The purpose of testing is verification, validation and error detection in order to find problems – and the purpose of finding those problems is to get them fixed.

Software Testing

Testing involves operation of a system or application under controlled conditions and evaluating the results. Every Test consists of 3 steps :

Planning : Inputs to be given, results to be obtained and the process to proceed is to planned.

Execution : preparing test environment, Completing the test, and determining test results.

Evaluation : compare the actual test outcome with what the correct outcome should have been.

23. Printers

Users like to print. The concept behind the web should save paper and reduce printing, but most people would rather read on paper than on the screen. So, you need to verify that the pages print properly. Sometimes images and text align on the screen differently than on the printed page. You need to at least verify that order confirmation screens can be printed properly.

24. Combinations

Now you get to try combinations. Maybe 600x800 looks good on the MAC but not on the IBM. Maybe IBM with Netscape works, but not with Lynx.

If the web site will be used internally it might make testing a little easier. If the company has an official web browser choice, then you just need to verify that it works for that browser. If everyone has a T1 connection, then you might not need to check load times. (But keep in mind, some people may dial in from home.) With internal applications, the development team can make disclaimers about system requirements and only support those systems setups. But, ideally, the site should work on all machines so you don't limit growth and changes in the future.

25. Load/Stress

You will need to verify that the system can handle a large number of users at the same time, a large amount of data from each user, and a long period of continuous use. Accessibility is extremely important to users. If they get a "busy signal", they hang up and call the competition. Not only must the system be checked so your customers can gain access, but many times crackers will attempt to gain access to a system by overloading it. For the sake of security, your system needs to know what to do when it's overloaded and not simply blow up.

Many users at the same time

If the site just put up the results of a national lottery, it better be able to handle millions of users right after the winning numbers are posted. A load test tool would be able to simulate large number of users accessing the site at the same time.

Large amount of data from each user

Most customers may only order 1-5 books from your new online bookstore, but what if a university bookstore decides to order 5000 different books? Or what if grandma wants to send a gift to each of her 50 grandchildren for Christmas (separate mailing addresses for each, of course.) Can your system handle large amounts of data from a single user?

Long period of continuous use

If the site is intended to take orders for flower deliveries, then it better be able to handle the week before Mother's Day. If the site offers web-based email, it better be able to run for months or even years, without downtimes.

You will probably want to use an automated test tool to implement these types of tests, since they are difficult to do manually. Imagine coordinating 100 people to hit the site at the same time. Now try 100,000 people. Generally, the tool will pay for itself the second or third time you use it. Once the tool is set up, running another test is just a click away.

26. Security

Even if you aren't accepting credit card payments, security is very important. The web site will be the only exposure some customers have to your company. And, if that exposure is a hacked page, they won't feel safe doing business with you.

27. Directory setup

The most elementary step of web security is proper setup of directories. Each directory should have an index.html or main.html page so a directory listing doesn't appear.

One company I was consulting for didn't observe this principal. I right clicked on an image and found the path "...com/objects/images". I went to that directory manually and found a complete listing of the images on that site. That wasn't too important. Next, I went to the directory below that: "...com/objects" and I hit the jackpot. There were plenty of goodies, but what caught my eye were the historical pages. They had changed their prices every month and kept the old pages. I browsed around and could figure out their profit margin and how low they were willing to go on a contract. If a potential customer did a little browsing first, they would have had a definite advantage at the bargaining table.

SSL Many sites use SSL for secure transactions. You know you entered an SSL site because there will be a browser warning and the HTTP in the location field on the browser will change to HTTPS. If your development group uses SSL you need to make sure there is an alternate page for browser with versions less than 3.0, since SSL is not compatible with those browsers. You also need to make sure that there are warnings when you enter and leave the secured site. Is there a timeout limit? What happens if the user tries a transaction after the timeout?

28 Logins

In order to validate users, several sites require customers to login. This makes it easier for the customer since they don't have to re-enter personal information every time. You need to verify that the system does not allow invalid usernames/password and that it does allow valid logins. Is there a maximum number of failed logins allowed before the server locks out the current user? Is the lockout based on IP? What if the maximum failed login attempts is three, and you try three, but then enter a valid login? What are the rules for password selection?

29. Log files

Behind the scenes, you will need to verify that server logs are working properly. Does the log track every transaction? Does it track unsuccessful login attempts? Does it only track stolen credit card usage? What does it store for each transaction? IP address? User name?

30. Scripting languages

Scripting languages are a constant source of security holes. The details are different for each language. Some exploits allow access to the root directory. Others allow access to the mail server. Find out what scripting languages are being used and research the loopholes. It might also be a good idea to subscribe to a security newsgroup that discusses the language you will be testing.

31. Web Server Testing Features

* Feature: Definition

* Transactions: The nunber of times the test script requested the current URL

* Elapsed time: The number of seconds it took to run the request

* Bytes transferred: The total number of bytes sent or received, less HTTP headers

* Response time: The average time it took for the server to respond to each individual request.

* Transaction rate: The average number of transactions the server was able to handle per second.

* Transferance: The average number of bytes transferred per second.

* Concurrency: The average number of simultaneous connections the server was able to handle during the test session.

* Status code nnn: This indicates how many times a particular HTTP status code was seen.

Compatibility and configuration testng is performanced to check that an application functions properly across various hardware and software environments. Often, the stragegy is to run the functional acceptance simple tests or a subset of the task-oriented functional tests on a range of software and hardware configurations. Sometimes, another strategy is to create a specific test that takes into account the error risks associated with configuration differences. For example, you might design an extensive series of tests to check for browser compatibility issues.

Software compatibility configurations include variances in OS versions, input/output (I/O) devices, extension, network software, concurrent applications, online services and firewalls. Hardwere configurations include variances in manufacturers, CPU types, RAM, graphic display cards, video capture cards, sound cards, monitors, network cards, and connection types(e.g. T1, DSL, modem, etc..).

Click stream Testing is to show which URLs the user clicked, The Web site's user activity by time period during the day, and other data otherwise found in the Web server logs. Popular choice for Click-Stream Testing statisticss include KeyNote Systems Internet weather report , WebTrends log analysis utility, and the NetMechanic monitoring service.

Disadvantage: Click-Stream Testing statistics reveal almost nothing about the user's ability to achieve their goals using the Web site. For example, a Web site may show a million page views, but 35% of the page views may simply e pages with the message "Found no search results," With Click-Stream Testing, there's no way to tell when user reach their goals.

Usage

" To determine whether the system achieves the desired level of proficiency in the production status.

" Used to verify -

Response time

Turn around time

Design performance.

" Test execution can be done using the simulated system and actual system.

" The system either can be tested as a whole or in parts.

Objectives " To determine whether the system can meet the specific performance criteria

" Verify whether system make optimum use of hardware and software.

" Determining response time to online use requests

" Determining transaction processing turnaround time.

How to Use

" Can be performed in any phase of SDLC

" Used to evaluate single aspect of system

" Executed in following manner -

Using h/w and s/w monitor

Simulation of functioning using simulation model

Creating quick or dirty programs to evaluate approximate performance of completed system.

When to use " Should be used early in SDLC

" Should be performed when it is known that the results can be used to make changes to the system structure.

Examples " Transaction turnaround time adequacy

" Optimum use of h/w and s/w.

Black Box Testing is testing without knowledge of the internal workings of the item being tested. The Outside world comes into contact with the test items, --only through the application interface ,,, an internal module interface, or the INPUT/OUTPUT description of a batch process. They check whether interface definitions are adhered to in all situation and whether the product conform to all fixed requirements. Test cases are created based on the task descriptions.

Black Box Testing assumes that the tester does not know anything about the application that is going to be tested. The tester needs to understand what the program should do, and this is achieved through the business requirements and meeting and talking with users.

Funcional tests: This type of tests will evaluate a specific operating condition using inputs and validating results. Functional tests are designed to test boundaries. A combination of correst and incorrect data should be used in this type of test.

Overwhelm the product for performance, reliability, and efficiency assessment; To find the breakpoint when system is failure; to increase load regressively to gather information for finding out maximum concurrent users.

Stress tests force programs to operate under limited resource conditions. The goal is to push the upper functional limits of a program to ensure that it can function correctly and handle error conditions gracefully. Examples of resources that may be artificially manipulated to create stressful conditions include memory, disk space, and network bandwidth. If other memory-oriented tests are also planned, they should be performed here as part of the stress test suite. Stress tests can be automated.

Breakpoint:

the capabilites and weakness of the product:

* High volunmes of data

* Device connections

* Long transation chains

Stress Test Environment:

As you set up your testing environment for a stress test, you need to make sure you can answer the following questions:

* Will my test be able to support all the users and still maintain performance?

* Will my test be able to simulate the number of transactions that pass through in a matter of hours?

* Will my test be able to uncover whether the system will break?

* Will my server crash if the load continues over and over?

The test should be set up so that you can simulate the load; for example:

* If you have a remote Web site you should be able to monitor up to four Web sites or URLs.

* There should be a way to monitor the load intervals.

* The load test should be able to simulate the SSL (Secure Server)

* The test should be able to simulate when a user submits the Form Data (GET method)

* The test should be set up to simulate and authentical the keyword verification.

* The test should be able to simulate up to six email or pager mail addresses and an alert should occur when there is a failure.

It is important to remember when stressing your Web site to give a certain number of users a page to stress test and give them a certain amount of time in which to run the test.

Some of the key data features that can help you measure this type of stress test, determine the load, and uncover bottlenecks in the system are:

* Amount of memory available and used

* The processor time used

* The number of requests per second

* The amount of time it takes ASP pages to be set up.

* Server timing errors.

Background " One half of total system development effort is directly attributable to controls.

" Controls include:

Data validation

File integrity

Audit trail

Back up and recovery

Documentation.

Other aspects of system related to integrity

" Control is system within a system.

" Control looks at the totality of the system.

Usage " Control is a management tool to ensure that processing is performed in accordance to what management desire or intents of management.

Objectives " Accurate and complete data

" Authorized transactions

" Maintenance of adequate audit trail of information.

" Efficient, effective and economical process.

" Process meeting the needs of the user.

How to Use " To test controls risks must be identified.

" Develop risk matrix, which identifies the risks, controls; segment within application system in which control resides.

" Testers should have negative approach i.e. should determine or anticipate what can go wrong in the application system.

When to use " Should be tested with other system tests.

Examples " file reconciliation procedures work

" Manual controls in place.

Background " Pre determination of Error handling features is the basic difference between Automated and manual systems.

" Manual System: can deal with problems as they occur.

" Automated Systems: Must pre program error handling.

Usage " It determines the ability of applications system to process the incorrect transactions properly

" Errors encompass all unexpected conditions.

" In some system approx. 50% of programming effort will be devoted to handling error condition.

Objectives " Determine:

" Application system recognizes all expected error conditions.

" Accountability of processing errors has been assigned and procedures provide a high probability that errors will be properly corrected.

" During correction process reasonable control is maintained over errors.

How to Use " A group of knowledgeable people is required to anticipate what can go wrong in the application system.

" It is needed that all the application knowledgeable people assemble to integrate their knowledge of user area, auditing and error tracking.

" Then logical test error conditions should be created based on this assimilated information.

" The error handling testing technique should test -

Error

Processing of error

Control condition

Reentry of condition is proper or not.

" The iterative process should be used where first the errors in the system trapped should be corrected and then the corrected system should be re-tested to check the authenticity of application operation and to complete the error handling testing cycle.

" Tester should think negatively to trap errors.

" Testers should determine how the system should fail so that they can test to determine if the software can properly process the erroneous data.

When to use " Throughout SDLC

" Impact from errors should be identified and should be corrected to reduce the errors to acceptable level.

" Used to assist in error management process of system development and maintenance.

Examples " Create a set of erroneous transactions and enter them into the application system then find out whether the system is able to identify the problems.

" Using iterative testing enters transactions and trap errors. Correct them. Then enter transactions with errors, which were not present in the system earlier.