Non-Functional Testing

Introduction

Non-functional testing is the testing of a software application or system for its non-functional requirements: the way a system operates, rather than specific behaviors of that system.

Non-functional testing refers to aspects of the software that may not be related to a specific function or user action such as scalability or security

The names of many non-functional tests are often used interchangeably because of the overlap in scope between various non-functional requirements.

Different Types of Non-Functional Testing

• Baseline Testing
• Performance Testing and Load Testing
• Compatibility Testing and Usability Testing
• Volume Testing
• InstallationTesting
• Interoperability Testing
• Recovery Testing
• Power Consumption/ Battery Life Testing
• Security Check
• Internationalization Testing and Localization Testing
• Stress Testing
• Maintainability Testing
• Endurance Testing
• Reliability Testing

Baseline Testing

The requirement specification validation is Baseline testing. Baseline testing also helps a great deal in solving most of the problems that are discovered. A majority of the issues are solved through baseline testing. Following is the scenario need to consider while testing:

• Validate all the requirement specifications and make sure that which is designed properly.

Performance Testing and Load Testing

Performance testing is the testing, which is performed, to ascertain how the components of a system are performing, given a particular situation. Resource usage, scalability and reliability of the product are also validated under this testing. This testing is the subset of performance engineering, which is focused on addressing performance issues in the design and architecture of software product.

Load testing is meant to test the system by constantly and steadily increasing the load on the system till the time it reaches the threshold limit. It is the simplest form of testing which employs the use of automation tools such as LoadRunner or any other good tools, which are available. Load testing is also famous by the names like volume testing and endurance testing.

• Verify that the app take minimum time to sync up data from the server.
• Retrials in Case of network disconnect – Verify the performance of the app when network is lost.
• Accelerometer – Ensure that the application properly responds to rotations.
• App Sleep Mode – Test the application response in the sleep mode.
• Verify that app take minimum time to launch.
• Verify the performance of the app in different OS/iPad versions.
• Verify that the response of the app for someone to come back to using the app after they have not used it for a period of time.
• Verify the behavior of application while device under test (DUT) is under production load – 100 Users. The application should not hang.

Compatibility Testing and Usability Testing

Compatibility testing is a type of software testing used to ensure compatibility of the system/application/website built with various other objects such as other web browsers, hardware platforms, users (in case if it’s very specific type of requirement, such as a user who speaks and can read only a particular language), operating systems etc. This type of testing helps find out how well a system performs in a particular environment that includes hardware, network, operating system and other software etc.

Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system.

• Verify response of each function and measure the performance of each response.
• Verify all settings available for the application and test the functionality of each setting.
• Validate the behavior of App in Low connectivity mode and verify proper behavior.
• Verify the compatibility of the app in different OS/iPad versions.
• Verify that for a new user to accomplish tasks the first time they use the app
• Verify that how quickly users can complete tasks on the app after they are familiar with its use.
• Verified that Alignment of Views should be proper.
• Check each and every buttons, images Pixels as per wire frame/page schematic or screen blueprint.
• Verify that Proper images size displayed in well manner with Orientation.
• Verify that all Spell-check are correct on Alert, error popup, Error messages etc.
• Verified that Spinner (Size, Types) should be suitable as per screen.
• Verify that application Logo should not be blurred and App title should not be misspelled.
• Verify that Font size should be consistent.
• Verify that any kind of text should not be cutting off.
• Verify that any kind of graphics should not be blurred, Check with different resolution Devices (Like BB have the Different resolution for all devices, iPhone 3Gs or iPhone 4)
• Verify that On Taping (Single Tap) Application Logo, Application Splash should display.
• Verify that Application Splash should not display more than 4 seconds.
• Verify that each screen should be visible for the time necessary to comfortably read all its information by moving between screens of an application.
• Verify that error messages in the Application must be clearly understandable.
• Verify that any function selected in the Application should start within the time specified in client’s requirement.
• Verify that if the screen requires data entry on a specific field should identify the required fields with a red asterisk and generate a friendly warning if the data is left blank.
• Verify that if the screen contains dates, numeric, currency or other specific data types, ensure that only valid data can be entered.
• Verify that if the screen contains text boxes that allow data entry, ensure that the width of data entered does not exceed the width of the table field.
• Verify that the screen takes more than 5 seconds to display the results/page; it should contain a progress bar so that the user understands the processing is continuing.
• Verify that a person deletes an item; it is a good idea to confirm the delete.
• Verify that the user interface uses combo boxes (drop down lists); be sure to include type ahead.
• Verify that icons are consistent throughout your application by using a common icon set. Avoid free clip-art icons.
• Verify that application has links on the screen; ensure that the links have consistent spacing between them and other links, and that the color of the links is consistent.
• Verify that application has menu items, ensure that menu items that are not applicable for the specific screen are disabled and the order in which each menu item appears is consistent.
• Verify that application has buttons (e.g. Submit, OK, Cancel, etc), ensure that the buttons appear in a consistent order from screen to screen (e.g. Submit then Cancel).
• Verify that tabbing should go onto the next field on the screen.
• Check for social networking options such as sharing, posting,etc…

Volume Testing

Volume testing refers to testing a software application or the product with a certain amount of data. E.g., if we want to volume test our application with a specific database size, we need to expand our database to that size and then test the application’s performance on it. The purpose of volume testing is to determine system performance with increasing volumes of data in the database.

• Response Memory Handling – Verify the Measure memory usage of the application and identify all memory leaks
• Verify that fill up the full phone memory with other files and data, and then try to install the APP on the phone.
• Verify that fill up the memory with files and data and leave exactly the same amount of memory required for the installation of the APP.
• Verify with leave only a small amount (5-7 KB) of space for the app to run. While you are in the app and continuously, making server calls this memory will also be filled up.
• Verify that the application should not crash while running the heavy memory consuming applications in the background.

Installation Testing

Installation testing is a kind of quality assurance work in the software industry that focuses on what customers will need to do to install and set up the new software successfully. The testing process may involve full, partial or upgrades install/uninstall processes

• Verify that the platforms and Operating Systems support.
• Ensure the test device is not the same as used for development or is not set up as the development environment.
• Verify that application can be installed successfully following normal installation procedures.
• Verify that version number matches the version specified during submission.
• Verify the application is seen in the installed applications list.
• Verify whether proper alert is displayed when we does not follow the normal installation procedures.
• Check installation with low Wi-Fi connectivity.
• Test installation behavior with Wi-Fi in disconnected mode.
• Check uninstallation and reinstallation.
• Check Application start/stop behavior — Start the application by selecting the icon or following the steps outlined in the submission statement.
• Check installation behavior when receiving voice calls — while installation process is in progress, make a call to the test device.
• Check installation behavior when receiving text messages — while installation process is in progress; send a text message to the test device.
• Check if the app is supported on an older firmware (i.e.: iOS 4.x), especially if it is part of the requirement, else an intelligent message should be displayed to the user.

Interoperability Testing

Interoperability testing involves testing whether a given software application is compatible with others and promotes cross-use functionality. This kind of testing is now important as many different kinds of technology are being built into architectures made up of many diverse parts.

• Phone call interrupt – when application in foreground: When the user is running the application he should be able to send/receive a call.
• Phone call interrupt – when application in foreground: When the user is running the application, he should be able to send/receive a call and once it has ended, the user should be taken back to the application automatically.
• Verify that the application installed and running, send a text message to the test device.
• Verify that user can get SMS alert when application is running.
• Verify that the location service alert should be shown while running the GPS related applications.
• Verify that application should not crash using Screen Lock button.
• Verify that when application is running then inserting and removing charger do not cause any problem and proper message is displayed when charger is inserted in device.

Recovery Testing

Recovery testing is the activity of testing how well an application is able to recover from crashes, hardware failures and other similar problems.

• Verify that the application should not crash after re-booting the device.
• Verify that the application should not crash while launch the application in offline/Airplane active mode.
• Network unavailable: When the application cannot connect to the server, appropriate message should be shown like ‘The server cannot be reached, please try again’.
• Network unavailable: Device is in non-wifi mode: When the application cannot connect to the network due to non-wifi, appropriate message should be shown.
• Network unavailable: Device is in Airplane mode: When the application cannot connect to the network, appropriate message should be shown.
• In 3G or cellular data: Device connected: When the application connected to the cellular data mode, it should work properly without any interruption.
• In Wi-Fi + VPN mode: Device connected: When the application connected to the wi-fi + VPN, it should work properly without any interruption.
• In 3G or cellular data + VPN mode: Device connected: When the application connected to the cellular data mode + VPN, it should work properly without any interruption.
• In Wi-Fi mode: Device connected: When the application connected to the network using wi-fi, it should work properly without any interruption.
• Network change: Verify that the application should work properly while switching from Wi-Fi mode to 3G or cellular data and vice versa.
• Verify that the session should be expire with the time constrain specified in the requirement (necessary for specific applications) and recover into its original state.
• Application should gracefully handle the condition when incoming communication is made via Bluetooth [Send a file using Bluetooth(if applicable) to the device application presents the user]

Power Consumption/ Battery Life Testing

Power Consumption/ Battery Life Testing are the activity of testing that helpful to identify the battery consumption of the application. Also, it helps to measure the behavior of the application under different battery levels if the application runs in background as well as in foreground.

• Battery Usage – Measure the battery usage and will provide feedback on increasing battery life. Start on a full battery and use the application for 6-12 hours and measure the battery level at the end of each ½ or 1 hour.
• Turn off the screen lock and power saver modes on the device. Then start on a full battery and keep the application running on its main, home or dashboard screen as appropriate, and measure the battery level at ½ or 1 hour intervals.
• iPhone, iPod & iPad settings are adjusted; screen brightness, minimize use of location services, turn off push notifications, turn off other downloaded applications, fetch new data less frequently and turn off push mail; run the application to determine the rate it took for the battery life to decrease.
• Verify that when application is running and battery is low then proper message is displayed to the user.

Security Check

Aim of Security testing is to check whether the application or the product is secured or not. It checks to see if the application is vulnerable to attacks, if anyone hack the system or login to the application without any authorization. It is a process to determine that an information system protects data and maintains functionality as intended.

• Prevent storage of credit card and other sensitive account data within the mobile device.
• Prevent unauthorized logical-device access by implementing passcode or face detection to access the application (auto lock application if idle for x seconds and can be opened only with a passcode).
• Block screen capturing/Copy paste kind of activities in the application.
• Create the ability to remotely disable payment application. A server-side push notification system can be developed to lock and unlock the application.
• Once the app detects lost or stolen device, the app should be intelligent enough to wipe the data. Even if the app is not used for a pre-defined amount of time, it should be able to wipe the device.
• Protect against known OS vulnerabilities, for example app can detect whether the device is rooted (Jail broken) and if found rooted, it can stop working.
• Provide an indication of secured state, an android application detects other application running on the device and if it detects any unusual activity, it can report to the user and warn the user about the security.
• Conform to secure coding, engineering, and testing. As a guideline, account information should not be stored locally nor write in the log file for debugging purpose.
• Cipher the android database, since android does not come up hardware encryption it makes sense to use ciphered database instead plain sql-lite.

Internationalization Testing and Localization Testing

Aim of Internationalization and Localization testing is to make sure usability, acceptability, reliability to audience and users worldwide & check whether the application under test is ready for world-readiness. First application under test needs to be localized & then tested for many other counts like locale, copy text, language, compatibility, reliable functionality and interoperability.

• Make sure all UI elements are isolated in thelocalizable resources. Pseudo-localization should not leave anyunlocalized text in the UI. Text that remains untranslated after pseudo-localization most likely is hard-coded in the application’s code and creates alocalizability problem. In addition to the text that appears in the graphical user interface (GUI), verify that the following text islocalizable:
  • Parts of the command-line interface
  • Messages in the Event Log and Error Log
  • Error messages
  • Tool tips

• Check that static UI text (such as names of dialog boxes, Field labels, tabs, and other things the user cannot change) is displayed in the language of the application’s UI. There should be no question marks, default glyphs, or random characters in the UI, regardless of the locale or language settings of the system.
• Applications should not rely on consistency of the resource text in different parts of the system. If the functionality of an application relies on several strings being identical in different parts of the application, either those strings have to originate from a single source, or their relationship should be clearly commented for future localization and support. Make sure that random pseudo-localization does not affect the functionality of the application.
• Font properties must not be in localizable resources. Applications violating this rule are likely to use an incorrect font face and size if pseudo-localized.
• Verify that applications are not affected if the size of the shell font or screen resolution is changed.
• Growth of the UI text length due to pseudo-localization should not cause text clipping; compensate for this problem by having the UI adjusted manually (by localizers) or automatically (with pseudo-localization). Applications should not assume the UI dimensions are fixed.
• Verify that the dynamic-string composition can hold if the order of parts of the output must be changed. Make sure the composition rules are not based on a single language.
• Verify that application crashes or not when change the language and re-launch the application from background.

Stress Testing

Stress testing is the process of determining the ability of software application to maintain a certain level of effectiveness under unfavorable conditions.

• Verify that the application does not crash in conditions of insufficient computational resources (such as memory or disk space).
• Verify the application’s response time under stress testing. It should be normal.

Maintainability Testing

Maintainability testing basically defines that how easy it is to maintain the software application. This means that how easy it is to analyze, change and test the application or product. Maintainability testing shall use a model of the maintainability requirements of the software applications.

• Verifying the development standards such as structured programming, standards for database approach, recognizable nomenclature and standards for the user interfaces.
• Verify if the data processing split up into sub-transactions
• Verify if the input, the processing and the output have been implemented separately.
• Verify if the programs have been parametrize under necessary conditions to promote reusability.
• Verify if the systems are distributed.

Endurance Testing

Endurance testing is usually done to determine if the system can sustain the continuous expected load. During soak tests, memory utilization is monitored to detect potential leaks.

The goal is to discover how the system behaves under sustained use. That is, to ensure that the throughput and/or response times after some long period of sustained activity are as good as or better than at the beginning of the test.

• Verify the application’s behavior for 1 hour sustained running.
• Ensure that the throughput and/or response times after some long period of sustained activity are as good as or better than at the beginning of the test. Verify any memory leaks happens.

Reliability Testing

Reliability testing is a measure of stability and overall performance of software application during an extended period of time under various specific sets of test conditions.  This type of testing incorporates the results from non-functional testing such as stress testing, security testing, network testing, along with functional testing.  It is a combined metric to define software’s overall reliability.

• Verify the time to failure for a non-repairable application.
• Verify the recovery time of the application from a failure (MTTR).