Scope In Test Plan | What Do You Measure Test Coverage Against?

Conclusion

So, in short, a Master Test plan is a high-level test plan for a project/product that joins all other test plans, on the other hand, Testing levels are specific like a Unit Test plan or Integration Test plan.

In the next tutorial, we will dig into System testing, plan, and others.

Happy learning, until then.

Performance testing is a non-functional software testing technique that determines how the stability, speed, scalability, and responsiveness of an application holds up under a given workload. It’s a key step in ensuring software quality, but unfortunately, is often seen as an afterthought, in isolation, and to begin once functional testing is completed, and in most cases, after the code is ready to release.

The goals of performance testing include evaluating application output, processing speed, data transfer velocity, network bandwidth usage, maximum concurrent users, memory utilization, workload efficiency, and command response times.

Learn how to adopt a combined “shift left” and “shift right” performance engineering approach to build a highly productive software development organization.

Learn more

Organizations run performance testing for at least one of the following reasons:

The specific steps of performance testing will vary from one organization and application to the next. It depends on what performance indicators the business considers most important. Nevertheless, the general goals of performance testing are largely the same across the board so there’s a certain workflow most testing plans will follow.

Identify the test environment and tools

Identify the production environment, testing environment, and testing tools at your disposal. Document the hardware, software, infrastructure specifications, and configurations in both test and production environments to ensure coherence. Some performance testing may occur in the production environment but there must be rigorous safeguards that prevent the testing from disrupting production operations.

Define acceptable performance criteria

Determine the constraints, goals, and thresholds that will demonstrate test success. The major criteria will be derived directly from the project specifications, but testers should be adequately empowered to set a wider set of tests and benchmarks.

Plan and design tests

Think about how widely usage is bound to vary then create test scenarios that accommodate all feasible use cases. Design the tests accordingly and outline the metrics that should be captured.

Prepare test environment and tools

Configure the testing environment before you execute the performance tests. Assemble your testing tools in readiness.

Run the performance tests

Execute the tests. Capture and monitor the results.

Resolve and retest

Consolidate and analyze test results. Share the findings with the project team. Fine tune the application by resolving the performance shortcomings identified. Repeat the test to confirm each problem has been conclusively eliminated.

Create a testing environment that mirrors the production ecosystem as closely as possible. Without that, the test results may not be an accurate representation of the application’s performance when it goes live.

Performance testing and performance engineering are two closely related yet distinct terms. Performance Testing is a subset of Performance Engineering, and is primarily concerned with gauging the current performance of an application under certain loads.

To meet the demands of rapid application delivery, modern software teams need a more evolved approach that goes beyond traditional performance testing and includes end-to-end, integrated performance engineering. Performance engineering is the testing and tuning of software in order to attain a defined performance goal. Performance engineering occurs much earlier in the software development process and seeks to proactively prevent performance problems from the get-go.

Since performance testing seeks to establish how well a system runs when subjected to different workloads, it’s difficult to execute such tests efficiently without using automated testing tools. Testing tools vary in their capability, scope, sophistication and automation. Find out how OpenText Testing Solutions can move the effectiveness of your performance testing to the next level.

“If you spot any confusing or contradicting information, please ask the Team Leader via the chat rather than making assumptions.”

Motivation

The scope of a test defines what areas of a customer’s product are supposed to get tested, what functionalities to focus on, what bug types the customer is interested in, and what areas or features should not be tested by any means.

If something is in scope, please test it; if something is out of scope, it should not be tested. Understanding the scope of a test is crucial to be a successful tester on our platform.

What information makes up the scope of a test?

How to Improve Test Coverage?

#Remove Dead Code

Total coverage can be defined as a ratio of code coverage and test coverage (covered_code/total_code). You can increase coverage by decreasing the denominator which is a total code. This can be possible by removing the Dead code or Zombie code. Usually, Dead code can be found in program history in which feature code was added, deleted or disabled, and the debugging code was likewise added and deleted. In this way, you can increase your total code coverage ratio without writing any additional tests.

Dead code can be found easily by manual testing or using automation tools. Before removal of dead code, you need to perform functional testing and if it performs exactly as per requirements than unused can be removed. You can also use static test coverage analysis tools to identify the unused dead code from the source code. The effectiveness of static analysis, however, depends on tools, language and architecture, but it’s a good start.

#Remove Redundant/Cloned Code

Removing cloned code can improve test coverage ratios in the same way as removing dead code.

The long program contains base code and code blocks which have clones in the programme. If you find these clones and remove it, you can increase test coverage with two improvements, one is size reduction and the second is “free” additional testing. You can increase your test coverage 5-10% by removing the cloned code from your source code.

#Increase device coverage

The digital space is complex more than ever. Mobile devices and OS versions are also being introduced to the market in a similar cadence while desktop browsers like Chrome and Firefox are releasing a public beta or GA on a regular basis. While each release not just introduces a test coverage challenge, but also introduces new features and functionalities which require additional testing efforts.

To measure maximum testing coverage, Perfecto Mobile has built a methodology that is based on market usage analytics and looks at the market devices from various angles.

The way to measure mobile testing coverage is to understand through usage and customer analytics. Make a list of top 30 devices that are mostly used in your geography. Also, make sure to include devices from both iOS and Android OS version perspective as well as device ages (New and trending, Most popular, Legacy).

For iOS, you can include top 3 or 4 OS families based on current OS and devices statistics when you start measuring it. Currently, top 3 families are iOS 11, iOS 10 and iOS 9. The problem is that there are devices that can only run on iOS9.3.5 (iPhone 4S, iPad Mini), and old devices like the iPhone 5, 5C, iPad 4th Gen will be left behind with OS upgradations.

For Android, the complication is even higher. To access the latest version of Android OS, developers and test engineers must use Google devices such as Nexus/Pixel or other brands who get the latest updates.

To measure and keep up with the mobile market coverage, you can consider 3 layer methodology of Essential, Enhanced and Extended segments as an Index of top used devices per geography. This approach provides data-driven test coverage by combining multiple Devices/OSs with characteristics like ageing, screen parameters and other testing related guidelines.

To project the upcoming releases of devices, OS versions, browsers you need to build a mobile market calendar, use it with existing analytics and validate and calibrate this data on a quarterly basis as a continuous practice.

What Is Code Coverage And How Is It Different From Test Coverage?

Code coverage is the measure of degree to which the software code has been executed. It covers automated as well as manual tests and gives you the percentage of code that has been executed. Based on this information, you can implement additional testing for the product and make it a lot more comprehensive.

The primary difference between test coverage and code coverage is that while test coverage is a qualitative measure, code coverage is more about quantitative measurement in terms of testing.

Code coverage helps determine how much of the code is executed while the application is running, whereas test coverage is helpful in describing test cases that can be written against the requirements document.

With the code coverage v/s test coverage debate addressed, let’s now take a look at various test coverage techniques.

Master Test Plan

The master test plan is a document that defines how the testing is being projected and how it will be handled across different test levels. It gives an overview of the key decisions taken, the strategies to be implemented, and the testing efforts included in the project.

Details provided by Master Test Plan

In simple words, the master test plan for software testing provides the following details:

• List of tests to be executed.
• Testing levels to be assured.
• Relationship between different test levels and related coding activity.
• Test execution strategy.
• Justify testing effort, which is a section of the project.
• The master test plan should adjust with test policy and test strategy and list out all exceptions or errors and their possible impact.

Example – If regression testing is always mandatory before the release of a product, but no regression testing is needed before the release of a product. That information should be in the master plan document with justification, and it also points out the impact of skipping regression testing, if any. Besides, It should also list out the steps taken to lessen any risks that occur from skipping regression testing.

Also read Test Plan and Test case

The structure of the master test plan and its content depends on the following factors:

• Type of organization
• Documentation rules followed by the organization
• Level of project formality

Content of Master Test Plan

A master test plan should include the following points:

• List of items to be tested.
• List of items not to be tested.
• List of product quality features that will be tested.
• List of product quality features that will not be counted for testing.
• Test execution cycles.
• Test budgets according to the project or overall operational budget.
• Testing schedule.
• Interaction between testing cycles and release plan.
• Interaction between other departments and the testing team.
• Scope of each test item.
• Predefined entry, continuation, and exit conditions for each test level.
• Risks associated with the test project.
• Management and control of testing.
• Team members are responsible for each test level.
• Inputs and outputs for each test level.

Test instructions

Test instructions are displayed below the Access section and they consist of three sections:

• The goal of this test: This section usually covers the general purpose of the test. It often names what to focus on, e.g. new feature that is about to be released.

• Out of scope: What is not supposed to get tested is specifically stated here. On live websites, you may oftentimes not complete orders at the end of a checkout process, or some areas of a website or an app might not be ready yet. Make sure to stick to this information – disregarding them can have big consequences.

• Additional requirements: If there is anything else to point out, information will be given in this section. This might be information concerning the whole test, credentials for user accounts, dummy payment information, etc.

Test plan là gì?

• Test plan là tài liệu chi tiết mô tả chiến lược kiểm thử, mục tiêu, lịch trình và ước tính cùng khả năng cung cấp các nguồn lực cho kiểm thử (nhân lực, phần mềm, phần cứng). Nói cách khác đây là tài liệu giúp Tester xác định nỗ lực cần thiết để kiểm chứng chất lượng của phần mềm đang được kiểm thử.

• Nó có vai trò như một bản kế hoạch chi tiết để thực hiện các hoạt động kiểm thử phần mềm. Hay quy trình xác định được quản lý và giám sát từng bước bởi Test Manager.

• Test plan là tài liệu bắt buộc đối với bất kỳ quy trình kiểm thử phần mềm nào. Bởi nó chính là chiếc la bàn giúp Tester định hướng đường đi của mình trong suốt quá trình kiểm thử.

Chat information

The team leader or Customer Success Manager might have provided additional information about the scope or posted important reminders. Please always check the chat if you see that there are new messages. Whenever you have any questions about the environment or the scope of the cycle, please use the chat to ask for the Team Leader or even other testers’ help.

Out of scope

Out of scope means that something is not supposed to be tested. Note that the following points are always out of scope (unless stated differently in the test description):

• Legal problems. We are not legal advisors and the severity of a bug is not determined by legal provisions, frameworks, or standards.

• Problems related to browser extensions, ad-blockers, or virus scanners, e.g. blocking certain contents or even the execution of apps.

• Website content should not be changed, e.g. by using the Google Translator service to translate the website into a different language, as it may lead to unexpected behaviour.

• Set up problems in tests.

Setup problems

Setup problems are not legitimate bugs – it only means there is a problem with the test setup or the test environment. Please report them to Team Leader via the test chat who will in turn contact the CSM. Here are a few examples:

• The URL given in the test environment section of a test is wrong (404 error).

• The test environment can only be accessed with credentials, but none are given or the given ones don’t work.

• The test environment requires a proxy or VPN to be accessed but this information is missing in the instructions (403, 1020 errors).

• An app cannot be downloaded or installed.

• The environment is not available for some time while testing. Remember the customer might update the environment during a test run, so make sure your issues are not temporary ones.

• The customer provided login or payment credentials for user accounts but they don’t work.

• A link on a staging environment redirects to their live website. The link was not updated to point to the staging environment during the test environment setup.

• The device requirements for a test e.g. list iOS 9 but the app under test can only be installed on iOS 10 and above.

The scope is unclear or contains contradicting information

If you don’t understand something about the scope or if you find contradicting information, please contact the test’s Team Leader via the test chat as described in the article Confusing Instructions. Sharing such information helps us improve the setups of future tests.

Features

Each test contains at least one of what we call features. A feature can describe an area of a product, e.g. a landing page, product overview page or product detail page of a webshop, it can describe a process like a checkout process of a website or an app, or it can even be a particular functionality that should be tested. For example:

When a customer sets up a new test, so-called standard features will be added by default. Customers can use these standard features and modify them if they want or add completely new features. You can learn more about some of our standard features here.

You can only submit bugs if a corresponding feature exists in the test. For example, if none of the given features covers the checkout process, you cannot submit bugs related to it.

Bug types and severities

Customers are usually only interested in certain bug types and/or severities, not in all of them at once (Functional, Content, Visual, Usability). You can see which bug types and severities are in scope by checking out the payout table in the right sidebar on the test overview page:

If a bug type/severity is not listed there, you won’t be able to select it in the bug form, and it is thus out of scope. E.g. if the payout table does not list the bug type Visual, you may not report visual bugs.

Why is information sometimes not consistent?

When customers intend to run a new test but the test setup of a previous test can be reused somehow, they will often duplicate that old test and adjust severities as well as additional requirements. Features and instructions are usually not adjusted, so sometimes irrelevant information might be shown to you or feature descriptions and additional requirements may contradict each other. Since features were duplicated but additional requirements were specifically written for the new test, additional requirements will often override feature descriptions.

If you spot any confusing or contradicting information, please ask the Team Leader via the chat rather than making assumptions.

How do you define the scope and objectives of a software testing project?

Software testing is a crucial part of software development, but how do you plan and execute it effectively? In this article, you will learn how to define the scope and objectives of a software testing project, and why they are important for delivering quality software.

The scope of testing is the extent and boundaries of what you will test and what you will not test. It includes the features, functions, requirements, risks, and assumptions of the software under test. The scope of testing helps you focus on the most important and relevant aspects of the software, and avoid wasting time and resources on unnecessary or out-of-scope testing. To define the scope of testing, you need to understand the context, purpose, and expectations of the software, and communicate with the stakeholders, developers, and users.

• Sivaprasad Kunjanpillai

  Director @ The QE Hub l Programme / Portfolio Test Manager l CTAL – TM | CSTE | PRINCE2 Practitioner

  Identifying the accurate testing scope of a project is the important first step in laying out quality driven delivery approach. This can be achieved by: – Conduct an in-depth analysis of requirements to gain insights into functionalities and objectives, thus consolidate the functional and non-functional requirements that must undergo thorough testing. – Determine the specific goals of the project and identify suitable test levels and types to aligns with it. – Analyse dependencies on external systems and services to ensure right level of test coverage to validate them. – Identify data driven requirements, regulatory standards, industry-specific criteria, and high-risk areas within the application, to plan focused testing activities.

• NARAYANAN PALANI

  🏦Platform Engineering Lead 💳Retail, Commercial and Investment Banking 📚Best Seller 👉FOLLOW

  Boundary Value Analysis and Cyclomatic Complexity are still two important methods to adapt high quality coverage on testing! Having a test plan is necessory when documenting such inscope and out of scope detailed. Hence ISTQB foundation is still a valuable exam for those starting a career in QA roles.

The objectives of testing are the specific goals and outcomes that you want to achieve from testing. They are derived from the scope of testing, and they guide your testing strategy, methods, techniques, and criteria. The objectives of testing should be SMART: specific, measurable, achievable, relevant, and time-bound. Some examples of testing objectives are verifying the functionality of the software, identifying and reporting defects, measuring the quality and coverage of testing, and providing feedback and recommendations.

• Md. Farhan Labib

  Specialist SQA Engineer @10 Minute School // Unearthing Software Flaws with Precision

  In a project involving an online educational platform, I applied SMART testing objectives effectively. Our goal was to validate the platform’s quiz feature: Specific: We focused on the quiz feature. Measurable: We tracked quiz success rates and score calculations. Achievable: Testing quizzes were within our scope and resources. Relevant: Quizzes were crucial for user engagement. Time-bound: We adhered to a set testing timeline. We created diverse test cases, covering quiz types and user scenarios. During testing, we found a mobile display bug, promptly reported and resolved it, enhancing the user experience. This emphasized the importance of SMART testing objectives in ensuring feature quality.

The testing strategy is the high-level approach and plan for testing. It defines the scope and objectives of testing, the roles and responsibilities of the testers, the testing resources and tools, and the testing environment and data. The testing strategy aligns with the project scope, objectives, and constraints, and it provides a clear and consistent direction for testing.

The testing methods are the ways of applying the testing strategy to the software under test. They include the testing techniques, processes, standards, and best practices that you use to design, execute, and evaluate test cases. The testing methods depend on the scope and objectives of testing, the characteristics and complexity of the software, the testing levels and types, and the testing tools and resources. Some examples of testing methods are black-box testing, white-box testing, exploratory testing, automated testing, and regression testing.

The testing criteria are the measures and indicators that you use to evaluate the results and outcomes of testing. They include the test-case design criteria, the test execution criteria, the test result analysis criteria, and the test completion criteria. The testing criteria help you determine the quality, effectiveness, efficiency, and completeness of testing, and to decide when to stop testing. The testing criteria should be defined in advance based on the scope and objectives of testing, the testing strategy and methods, and the stakeholder expectations.

The testing feedback is the information and communication that you provide and receive during and after testing. It includes the test reports, defect reports, test metrics, test reviews, test lessons learned, and test recommendations. The testing feedback helps you monitor and control the testing process, identify and resolve issues, improve the software quality, and enhance the testing performance. The testing feedback should be timely, accurate, clear, and actionable, and it should address the scope and objectives of testing, the testing strategy and methods, and the testing criteria.

What is the Importance of Test Plan?

Making Test Plan document has multiple benefits

• Help people outside the test team such as developers, business managers, customers understand the details of testing.
• Test Plan guides our thinking. It is like a rule book, which needs to be followed.
• Important aspects like test estimation, test scope, Test Strategy are documented in Test Plan, so it can be reviewed by Management Team and re-used for other projects.

Test environment

The URL in the Access section at the top of the test overview page defines what website or app should be tested. Any other websites or apps are not under test (unless stated differently in the test description), including other builds of the same app. For more information, see Testing Environment.

You can only see access information once the test starts (mind the countdown at the top), confirm you want to participate in the cycle, and then agree with the cycle terms.

What is Test Coverage Matrix?

TCM is a matrix table which is used to make sure that all the possible conditions/features that are to be tested are thought of. It also helps in identifying the probable gaps. It can be seen as a checklist which ensures that the functionality of the given screen is verified in all the possible combinations.

To get started with test coverage matrix, there are no prerequisites. You may create your own table, considering the following table as an example.

A Detailed Guide on Test Coverage

The world has witnessed some of the disastrous events due to the errors prevailing in the software. One such event, which I personally recall, is the opening of Heathrow Terminal 5, the UK in 2008.

Engineers were pretty confident about the working of terminal pertaining to their rigorous testing. Though the baggage handling system couldn’t cope up when it faced some real-life scenarios; which resulted in complete shut down of the system. Over the following 10 days, some 42,000 bags failed to travel with their owners, and over 500 flights were cancelled.

All this due to the failure of engineers to carry out the test coverage of possible real-life scenarios. And it went down as “Disastrous Opening Day for Terminal 5” But we, at Simform, have no room for any mistakes in our test coverage practices.

In this blog, you will discuss all the aspects of test coverage and how it directly affects the production, whether it is custom software development or software testing.

Why is information sometimes not consistent?

When customers intend to run a new test but the test setup of a previous test can be reused somehow, they will often duplicate that old test and adjust severities as well as additional requirements. Features and instructions are usually not adjusted, so sometimes irrelevant information might be shown to you or feature descriptions and additional requirements may contradict each other. Since features were duplicated but additional requirements were specifically written for the new test, additional requirements will often override feature descriptions.

If you spot any confusing or contradicting information, please ask the Team Leader via the chat rather than making assumptions.

Benefits of Test Coverage

#Eliminates defects at early stages

You can identify gaps in requirements, test cases and defects at early stages of your product development life cycle. It saves you from a lot of headaches later.

#Better coverage

Test coverage creates more test cases to ensure superior coverage. This leads to fewer defects and work to do at later stages. Moreover, you get to increase customer satisfaction with a refined product.

#Removes redundant cases

Test coverage is especially useful in identifying and eliminating test cases that don’t make much sense in the current project. Your developers can report these cases to remove them and make the overall code lighter.

#Higher ROI

With fewer defects at production stages and lesser user acceptance testing defects, test coverage can have a significant impact on the ROI. All the resources which would’ve been spent on addressing defects, now translate into your profits.

#Discovers uncovered areas

Test coverage helps you unearth areas of a program that have not been covered by a set of test cases. It helps make your program more robust and error-free.

#Superior control

Test coverage gives you a better control over the resources during the product development lifecycle. You save time by eliminating defects earlier and faster. The saved time allows you to keep a tab of costs. And most importantly, you get to have a firm grip on the scope of the project.

#Smoother testing cycles

You can prevent defect leakage using Test coverage analysis. Test coverage also helps in Regression testing, test case prioritization, test suite augmentation and test suite minimization. All this leads to smoother yet efficient testing cycles.

Before we dive deeper into test coverage and it’s techniques, let’s also discuss another wildly popular testing methodology— code coverage.

Bug types and severities

Customers are usually only interested in certain bug types and/or severities, not in all of them at once (Functional, Content, Visual, Usability). You can see which bug types and severities are in scope by checking out the payout table in the right sidebar on the test overview page:

If a bug type/severity is not listed there, you won’t be able to select it in the bug form, and it is thus out of scope. E.g. if the payout table does not list the bug type Visual, you may not report visual bugs.

What is Test Coverage?

Test coverage is defined as a technique which determines whether our test cases are actually covering the application code and how much code is exercised when we run those test cases.

If there are 10 requirements and 100 tests created and if 90 tests are executed then test coverage is 90%. Now, based on this metric, testers can create additional test cases for remaining tests.

In software development, a test plan defines your testing team’s test strategy, goals, and scope, which ultimately work together to ensure that all your software components are tested sufficiently before a release.

Follow these six steps to create an efficient test plan:

1. Define the release scope
2. Schedule timelines
3. Define test objectives
4. Determine test deliverables
5. Design the test strategy
6. Plan test environment and test data

Test plan là gì?

Một kế hoạch kiểm thử dự án phần mềm (test plan) là:

• Một tài liệu mô tả các mục tiêu, phạm vi, phương pháp tiếp cận, và tập trung vào nỗ lực kiểm thử phần mềm.
• Quá trình chuẩn bị test plan là một cách hữu ích để suy nghĩ tới những nỗ lực cần thiết để xác nhận khả năng chấp nhận một sản phẩm phần mềm.
• Các tài liệu đã hoàn thành sẽ giúp mọi người bên ngoài nhóm test hiểu được ‘tại sao’ và ‘như thế nào’ chấp nhận sản phẩm.
• Nó cần phải hoàn hảo đủ để dùng được nhưng không đủ hoàn hảo vì không ai bên ngoài nhóm test sẽ đọc nó.

Nói một cách khác cho dễ hiểu thì Test Plan là: Tài liệu tổng quan về việc test 1 project. Scope của project, hướng tiếp cận, STLC(Software Testing Life Cycle), resource và nhân lực cần có, các features cần được test và không phải test, các tool test và môi trường test cần có. Có thể ví test plan là 1 cái xương sống của 1 testing project và là cái được chuẩn bị đầu tiên khi có 1 project.

Sau đây là một số hạng mục có thể được bao gồm trong một test plan, tùy thuộc vào từng dự án cụ thể:

• Tiêu đề
• Định nghĩa version của phần mềm (version release)
• Lưu lại quá trình hiệu chỉnh tài liệu như tác giả, ngày cập nhật, duyệt
• Mục lục
• Mục đích của tài liệu, ý kiến chung
• Mục tiêu của chi phí kiểm thử (test)
• Giới thiệu tổng quan về sản phẩm
• Danh sách tài liệu liên quan như spec, tài liệu thiết kế, các kế hoạch test khác,…
• Các tiêu chuẩn thích hợp, các yêu cầu hợp lệ
• Nguồn gốc của các sự thay đổi
• Assumptions and dependencies
• Phân tích rủi ro của dự án
• Các vấn đề ưu tiên và tập trung test
• Phạm vi và giới hạn test -Test phác thảo – phân tích cách tiếp cận theo loại test, tính năng, chức năng, quy trình, hệ thống, mô đun, v.v … khi áp dụng
• Phân tích giá trị tương đương đầu vào, phân tích giá trị biên, các trường hợp lỗi
• Môi trường test – Phần cứng, hệ điều hành, phần mềm yêu cầu khác, cấu hình dữ liệu, giao diện với các hệ thống khác
• Phân tích tính hợp lệ của môi trường test – sự khác nhau giữa các hệ thống test – product và ảnh hưởng của chúng đối với tính hợp lệ của việc test.
• Các vấn đề thiết lập môi trường và cấu hình
• Quá trình chạy phần mềm
• Kiểm tra yêu cầu thiết lập dữ liệu
• Yêu cầu thiết lập cơ sở dữ liệu
• Test tự động – giải thích và tổng quan
• Các công cụ test được sử sụng, bao gồm các version, bản vá lỗi,…v.v
• Các qui trình bảo trì và quản lý version của test script/test code
• Theo dõi và giải quyết vấn đề – Các công cụ và qui trình
• Các thước đo về test sản phẩm được sử dụng
• Báo cáo các yêu cầu và khả năng giao test
• Điều kiện đầu vào và đầu ra của phần mềm
• Giai đoạn và điều kiện test ban đầu
• Điều kiện dừng test và test lại
• Sự bố trí nhân sự
• Những người cần training trước khi tham gia
• Nơi test
• Các tổ chức test bên ngoài sẽ sử dụng và mục đích, trách nhiệm, khả năng hoàn thành, người liên hệ và các vấn đề hợp tác của họ
• Các vấn đề độc quyền thích hợp, phân loại, bảo mật và bản quyền.
• Các vấn đề mở
• Phụ lục – bảng chú giải, các từ viết tắt, …v.v…

Attachments

In some cases, one or more attachments might be provided. They will be found in the right sidebar on the test overview page. Those are usually Spreadsheets, Documents, Images, Videos or PDF files given by the customer directly. The attachments might contain important information or requirements from the customer so make sure you checked them carefully before starting to test the app or website.

Tầm quan trọng của test plan

Việc lập test plan mang đến nhiều lợi ích:

• Giúp những người ngoài nhóm kiểm thử: nhà phát triển, quản lý doanh nghiệp hay khách hàng có thể hiểu chi tiết về quá trình kiểm thử.

• Test plan định hướng suy nghĩ của bạn và nó giống như quyển sách với nội dụng về các quy tắc, điều luật cần phải tuân theo để đạt được hiệu cao trong công việc.

• Các yếu tố quan trọng được lưu lại trong test plan gồm: Test Estimation, Test Scope, Chiến lược test. Vì vậy nó có thể được xem xét lại bởi nhóm quản lý có thể xem xét và tái sử dụng cho các dự án khác.

How to write a Test Plan

You already know that making a Test Plan is the most important task of Test Management Process. Follow the seven steps below to create a test plan as per IEEE 829

1. Analyze the product
2. Design the Test Strategy
3. Define the Test Objectives
4. Define Test Criteria
5. Resource Planning
6. Plan Test Environment
7. Schedule & Estimation
8. Determine Test Deliverables

Step 1) Analyze the product

How can you test a product without any information about it? The answer is Impossible. You must learn a product thoroughly before testing it.

The product under test is Guru99 banking website. You should research clients and the end users to know their needs and expectations from the application

• Who will use the website?
• What is it used for?
• How will it work?
• What are software/ hardware the product uses?

You can use the following approach to analyze the site

Now let’s apply above knowledge to a real product: Analyze the banking website

http://demo.guru99.com/V4

.

You should take a look around this website and also review product documentation. Review of product documentation helps you to understand all the features of the website as well as how to use it. If you are unclear on any items, you might interview customer, developer, designer to get more information.

Step 2) Develop Test Strategy

Test Strategy is a critical step in making a Test Plan in Software Testing. A Test Strategy document, is a high-level document, which is usually developed by Test Manager. This document defines:

• The project’s testing objectives and the means to achieve them
• Determines testing effort and costs

Back to your project, you need to develop Test Strategy for testing that banking website. You should follow steps below

Step 2.1) Define Scope of Testing

Before the start of any test activity, scope of the testing should be known. You must think hard about it.

• The components of the system to be tested (hardware, software, middleware, etc.) are defined as “in scope“
• The components of the system that will not be tested also need to be clearly defined as being “out of scope.”

Defining the scope of your testing project is very important for all stakeholders. A precise scope helps you

• Give everyone a confidence & accurate information of the testing you are doing
• All project members will have a clear understanding about what is tested and what is not

How do you determine scope your project?

To determine scope, you must –

• Precise customer requirement
• Project Budget
• Product Specification
• Skills & talent of your test team

Now should clearly define the “in scope” and “out of scope” of the testing.

• As the software requirement specs, the project Guru99 Bank only focus on testing all the functions and external interface of website Guru99 Bank (in scope testing)
• Nonfunctional testing such as stress, performance or logical database currently will not be tested. (out of scope)

Problem Scenario

The customer wants you to test his API. But the project budget does not permit to do so. In such a case what will you do?

Well, in such case you need to convince the customer that Api Testing is extra work and will consume significant resources. Give him data supporting your facts. Tell him if Api Testing is included in-scope the budget will increase by XYZ amount.

The customer agrees and accordingly the new scopes, out of scope items are

• In-scope items: Functional Testing, Api Testing
• Out of scope items: Database Testing, hardware & any other external interfaces

Step 2.2) Identify Testing Type

A Testing Type is a standard test procedure that gives an expected test outcome.

Each testing type is formulated to identify a specific type of product bugs. But, all Testing Types are aimed at achieving one common goal “Early detection of all the defects before releasing the product to the customer”

The commonly used testing types are described as following figure

There are tons of Testing Types for testing software product. Your team cannot have enough efforts to handle all kind of testing. As Test Manager, you must set priority of the Testing Types

• Which Testing Types should be focused for web application testing?
• Which Testing Types should be ignored for saving cost?

Which Testing Types should you focus in this case?

Select All that Apply

B) API Testing

C) Integration Testing

D) System Testing

E) Install/Uninstall Testing

F) Agile testing

We only select

B) API Testing

C) Integration Testing

D) System Testing

for Guru99 project

Step 2.3) Document Risk & Issues

Risk is future’s uncertain event with a probability of occurrence and a potential for loss. When the risk actually happens, it becomes the ‘issue’.

In the article Risk Analysis and Solution, you have already learned about the ‘Risk’ analysis in detail and identified potential risks in the project.

In the QA Test Plan, you will document those risks

Risk	Mitigation
Team member lack the required skills for website testing.	Plan training course to skill up your members
The project schedule is too tight; it’s hard to complete this project on time	Set Test Priority for each of the test activity.
Test Manager has poor management skill	Plan leadership training for manager
A lack of cooperation negatively affects your employees’ productivity	Encourage each team member in his task, and inspire them to greater efforts.
Wrong budget estimate and cost overruns	Establish the scope before beginning work, pay a lot of attention to project planning and constantly track and measure the progress

Step 2.4) Create Test Logistics

In Test Logistics, the Test Manager should answer the following questions:

• Who will test?
• When will the test occur?

Who will test?

You may not know exact names of the tester who will test, but the type of tester can be defined.

To select the right member for specified task, you have to consider if his skill is qualified for the task or not, also estimate the project budget. Selecting wrong member for the task may cause the project to fail or delay.

Person having the following skills is most ideal for performing software testing:

• Ability to understand customers point of view
• Strong desire for quality
• Attention to detail
• Good cooperation

In your project, the member who will take in charge for the test execution is the tester. Base on the project budget, you can choose in-source or outsource member as the tester.

When will the test occur?

Test activities must be matched with associated development activities.

You will start to test when you have all required items shown in following figure

Step 3) Define Test Objective

Test Objective is the overall goal and achievement of the test execution. The objective of the testing is finding as many software defects as possible; ensure that the software under test is bug free before release.

To define the test objectives, you should do 2 following steps

1. List all the software features (functionality, performance, GUI…) which may need to test.
2. Define the target or the goal of the test based on above features

Let’s apply these steps to find the test objective of your Guru99 Bank testing project

You can choose the ‘TOP-DOWN’ method to find the website’s features which may need to test. In this method, you break down the application under test to component and sub-component.

In the previous topic, you have already analyzed the requirement specs and walk through the website, so you can create a Mind-Map to find the website features as following

This figure shows all the features which the Guru99 website may have.

Based on above features, you can define the Test Objective of the project Guru99 as following

• Check that whether website Guru99 functionality(Account, Deposit…) is working as expected without any error or bugs in real business environment
• Check that the external interface of the website such as UI is working as expected and & meet the customer need
• Verify the usability of the website. Are those functionalities convenient for user or not?

Step 4) Define Test Criteria

Test Criteria is a standard or rule on which a test procedure or test judgment can be based. There’re 2 types of test criteria as following

Suspension Criteria

Specify the critical suspension criteria for a test. If the suspension criteria are met during testing, the active test cycle will be suspended until the criteria are resolved.

Test Plan Example: If your team members report that there are 40% of test cases failed, you should suspend testing until the development team fixes all the failed cases.

Exit Criteria

It specifies the criteria that denote a successful completion of a test phase. The exit criteria are the targeted results of the test and are necessary before proceeding to the next phase of development. Example: 95% of all critical test cases must pass.

Some methods of defining exit criteria are by specifying a targeted run rate and pass rate.

• Run rate is ratio between number test cases executed/total test cases of test specification. For example, the test specification has total 120 TCs, but the tester only executed 100 TCs, So the run rate is 100/120 = 0.83 (83%)
• Pass rate is ratio between numbers test cases passed / test cases executed. For example, in above 100 TCs executed, there’re 80 TCs that passed, so the pass rate is 80/100 = 0.8 (80%)

This data can be retrieved in Test Metric documents.

• Run rate is mandatory to be 100% unless a clear reason is given.
• Pass rate is dependent on project scope, but achieving high pass rate is a goal.

Test Plan Example:Your Team has already done the test executions. They report the test result to you, and they want you to confirm the Exit Criteria.

In above case, the Run rate is mandatory is 100%, but the test team only completed 90% of test cases. It means the Run rate is not satisfied, so do NOT confirm the Exit Criteria

Step 5) Resource Planning

Resource plan is a detailed summary of all types of resources required to complete project task. Resource could be human, equipment and materials needed to complete a project

The resource planning is important factor of the test planning because helps in determining the number of resources (employee, equipment…) to be used for the project. Therefore, the Test Manager can make the correct schedule & estimation for the project.

This section represents the recommended resources for your project.

Human Resource

The following table represents various members in your project team

No.	Member	Tasks
1.	Test Manager	Manage the whole project Define project directions Acquire appropriate resources
2.	Tester	Identifying and describing appropriate test techniques/tools/automation architecture Verify and assess the Test Approach Execute the tests, Log results, Report the defects. Tester could be in-sourced or out-sourced members, base on the project budget For the task which required low skill, I recommend you choose outsourced members to save project cost.
3.	Developer in Test	Implement the test cases, test program, test suite etc.
4.	Test Administrator	Builds up and ensures Test Environment and assets are managed and maintained SupportTester to use the test environment for test execution
5.	SQA members	Take in charge of quality assurance Check to confirm whether the testing process is meeting specified requirements

System Resource

For testing, a web application, you should plan the resources as following tables:

No.	Resources	Descriptions
1.	Server	Install the web application under test This includes a separate web server, database server, and application server if applicable
2.	Test tool	The testing tool is to automate the testing, simulate the user operation, generate the test results There are tons of test tools you can use for this project such as Selenium, QTP…etc.
3.	Network	You need a Network include LAN and Internet to simulate the real business and user environment
4.	Computer	The PC which users often use to connect the web server

Step 6) Plan Test Environment

What is the Test Environment

A testing environment is a setup of software and hardware on which the testing team is going to execute test cases. The test environment consists of real business and user environment, as well as physical environments, such as server, front end running environment.

How to setup the Test Environment

Back to your project, how do you set up test environment for this banking website?

To finish this task, you need a strong cooperation between Test Team and Development Team

You should ask the developer some questions to understand the web application under test clearly. Here’re some recommended questions. Of course, you can ask the other questions if you need.

• What is the maximum user connection which this website can handle at the same time?
• What are hardware/software requirements to install this website?
• Does the user’s computer need any particular setting to browse the website?

Following figure describes the test environment of the banking website

http://demo.guru99.com/V4


Step 7) Schedule & Estimation

In the article Test estimation, you already used some techniques to estimate the effort to complete the project. Now you should include that estimation as well as the schedule to the Test Planning

In the Test Estimation phase, suppose you break out the whole project into small tasks and add the estimation for each task as below

Task	Members	Estimate effort
Create the test specification	Test Designer	170 man-hour
Perform Test Execution	Tester, Test Administrator	80 man-hour
Test Report	Tester	10 man-hour
Test Delivery	20 man-hour
Total	280 man-hour

Then you create the schedule to complete these tasks.

Making schedule is a common term in project management. By creating a solid schedule in the Test Planning, the Test Manager can use it as tool for monitoring the project progress, control the cost overruns.

To create the project schedule, the Test Manager needs several types of input as below:

• Employee and project deadline: The working days, the project deadline, resource availability are the factors which affected to the schedule
• Project estimation: Base on the estimation, the Test Manager knows how long it takes to complete the project. So he can make the appropriate project schedule
• Project Risk : Understanding the risk helps Test Manager add enough extra time to the project schedule to deal with the risks

Let’s practice with an example:

Suppose the boss wants to complete the project Guru99 in one month, you already estimated the effort for each tasks in Test Estimation. You can create the schedule as below

Step 8) Test Deliverables

Test Deliverables is a list of all the documents, tools and other components that has to be developed and maintained in support of the testing effort.

There are different test deliverables at every phase of the software development lifecycle.

Test deliverables are provided before testing phase.

• Test plans document.
• Test cases documents
• Test Design specifications.

Test deliverables are provided during the testing

• Test Scripts
• Simulators.
• Test Data
• Test Traceability Matrix
• Error logs and execution logs.

Test deliverables are provided after the testing cycles is over.

• Test Results/reports
• Defect Report
• Installation/ Test procedures guidelines
• Release notes

Resources

Download a sample Test Plan Template

Tìm hiểu về test plan

Bài đăng này đã không được cập nhật trong 2 năm

Test plan – Kế hoạch kiểm thử

Attachments

In some cases, one or more attachments might be provided. They will be found in the right sidebar on the test overview page. Those are usually Spreadsheets, Documents, Images, Videos or PDF files given by the customer directly. The attachments might contain important information or requirements from the customer so make sure you checked them carefully before starting to test the app or website.

Ví dụ về testplan mẫu

Giới thiệu

Test plan được tạo để truyền đạt phương pháp kiểm thử cho các thành viên trong nhóm. Nó bao gồm các mục tiêu, phạm vi, lịch trình, rủi ro và cách tiếp cận. Tài liệu này sẽ xác định rõ ràng những sản phẩm kiểm thử sẽ là gì và những gì được coi là trong và ngoài phạm vi.

1.1 Mục tiêu

Test Case Tamer là một công cụ quản lý kiểm thử dựa trên web được sử dụng để tạo và lưu trữ các bài kiểm thử cũng như kết quả thực hiện. Công cụ này là một sản phẩm mới được viết bằng Ruby on Rails sử dụng cơ sở dữ liệu MySQL. Nhóm kiểm thử chịu trách nhiệm kiểm thử sản phẩm và đảm bảo nó đáp ứng nhu cầu. Nhóm kiểm thử đóng vài trò vừa là khách hàng vừa là tester trong dự án này.

Giai đoạn 1 của dự án sẽ cung cấp TCT (Test Case Tamer) với chức năng tạo và lưu trữ các bài kiểm thử thủ công. Điều này sẽ cho phép nhóm kiểm thử đầu chuyển các kiểm thử sang hệ thống mới.

1.2 Thành viên tham gia

Mục tiêu

Giai đoạn ban đầu sẽ bao gồm tất cả các yêu cầu ‘phải có’. Tất cả những yêu cầu này và bất kỳ yêu cầu nào khác được đưa vào đều phải được kiểm thử. Vào cuối Giai đoạn 1, tester phải :

Tạo kiểm thử thủ công với các bước cần thiết


Lưu trữ


Truy xuất và có khả năng xem lại khi chạy kiểm thử


Nhập kết quả và nhận xét phù hợp


Xem kết quả

Khi nhóm làm việc với sản phẩm, họ sẽ xác định nhu cầu cho giai đoạn thứ hai.

Load testing sẽ không được coi là một phần của dự án này vì cơ sở người dùng đã được biết và không phải là vấn đề.Viết lại, di chuyển hoặc chuyển các test case hiện có từ các tài liệu Word không được coi là một phần của dự án này.

Giả định và vấn đề

3.1 Giả định

Phần này liệt kê các giả định cụ thể cho dự án này.

Việc phân phối sản phẩm ở định dạng mà nhóm kiểm thử có thể thực hiện nó trong CVS.

3.2 Rủi ro

Các rủi ro sau đây đã được xác định và hành động thích hợp được xác định để giảm thiểu tác động của chúng đối với dự án. Tác động (hoặc mức độ nghiêm trọng) của rủi ro dựa trên việc dự án sẽ bị ảnh hưởng như thế nào nếu xảy ra. Trigger là cột mốc hoặc sự kiện nào sẽ khiến rủi ro trở thành vấn đề cần xử lý.

Rủi ro Mức độ Trigger Kế hoạch giảm thiểu Scope Creep – khi tester trở nên quen thuộc hơn với công cụ, họ sẽ muốn có nhiều chức năng hơn Cao Delay ngày thực hiện Mỗi lần lặp lại, chức năng sẽ được giám sát chặt chẽ. Các ưu tiên sẽ được thiết lập và thảo luận bởi các bên liên quan. Vì trình điều khiển là chức năng chứ không phải thời gian, nên có thể cần phải lùi ngày. Các thay đổi đối với chức năng có thể phủ nhận các bài kiểm thử đã được viết và chúng ta có thể mất các test case đã viết Cao – đúng tiến độ và chất lượng Mất hết test case Xuất dữ liệu trước khi nâng cấp, giảm thiểu khi cần thiết và nhập lại sau khi nâng cấp. Không thể cung cấp sản phẩm hàng tuần vì nhà phát triển làm việc ngoài trang web Trung bình Sản phẩm không được giao đúng tiến độ

Phương pháp kiểm thử

Dự án đang sử dụng một cách tiếp cận linh hoạt, với các lần lặp lại hàng tuần. Vào cuối mỗi tuần, các yêu cầu được xác định cho lần lặp đó sẽ được chuyển đến nhóm và sẽ được kiểm thử.

Exploratory testing phá sẽ đóng vai trò quan trọng trong quá trình kiểm thử vì nhóm chưa bao giờ sử dụng loại công cụ này và sẽ học hỏi trong quá trình thực hiện. Các kiểm thử cho chức năng theo kế hoạch sẽ được tạo và thêm vào TCT khi chúng ta nhận được các lần lặp lại của sản phẩm.

Test Automation

Kiểm thử đơn vị tự động là một phần của quá trình phát triển, nhưng không có kiểm thử chức năng tự động nào được lên kế hoạch tại thời điểm này.

Môi trường kiểm thử

Cần có một máy chủ mới cho máy chủ web, ứng dụng và cơ sở dữ liệu.

Cột mốc / Sản phẩm bàn giao

6.1 Test Schedule

Nhiệm vụ Bắt đầu Kết thúc Nỗ lực Nhận xét Kế hoạch kiểm thử Xem lại tài liệu yêu cầu 2đ Tạo ước tính kiểm thử ban đầu 1đ Nhân viên và đào tạo tài liệu kiểm thử mới Lần đầu tiên triển khai môi trường kiểm thử QA Kiểm thử chức năng – Lặp lại lần 1 Lặp lại lần 2 – Triển khai môi trường kiểm thử QA Kiểm thử chức năng – Lặp lại lần 2 Kiểm thử hệ thống Kiểm thử hồi quy – Regression Testing UAT Giải quyết các lỗi cuối và kiểm thử bản build cuối cùng Triển khai vào môi trường Staging Kiểm thử hiệu năng Phát hành để sản xuất

6.2 Bàn giao

Bàn giao Tới Ngày Test Plan Quản lý dự án, QA Director, Nhóm kiểm thử Traceability Matrix – Ma trận truy xuất nguồn gốc Quản lý dự án, QA Director Kết quả kiểm thử Quản lý dự án Báo cáo tình trạng kiểm thử QA manager, QA Director Số liệu Tất cả thành viên trong nhóm

Đến đây thì bạn đọc đã hiểu rõ hơn về test plan và cách thức xây dựng kế hoạch thử nghiệm hiệu quả. Hy vọng rằng bài viết đã cung cấp những thông tin hữu ích và thiết thực để hiểu rõ hơn về nghề nghiệp bạn đang theo đuổi. Nếu có bất cứ vấn đề gì thắc mắc hãy để lại bình luận phía dưới, Testerpro sẽ giải đáp ngay lập tức.

Nhận ngay ưu đãi lên đến 20 % học phí khi đăng ký Khóa học tester trong tháng này!

TEST PLAN in Software Testing (Example)

Test environment

The URL in the Access section at the top of the test overview page defines what website or app should be tested. Any other websites or apps are not under test (unless stated differently in the test description), including other builds of the same app. For more information, see Testing Environment.

You can only see access information once the test starts (mind the countdown at the top), confirm you want to participate in the cycle, and then agree with the cycle terms.

Cách viết test plan trong 8 bước

Bước 1: Phân tích sản phẩm (Analyze the product)

Đây là bước đầu tiên nhưng rất quan trọng trong quá trình kiểm thử. Việc nghiên cứu và phân tích sản phẩm sẽ hạn chế được tối thiểu những sai xót không đáng có xảy ra. Analyze the product quyết định cho các tiến trình kiếm thử tiếp theo. Để phân tích sản phẩm thì người lập test plan phải trả lời được những câu hỏi sau đây:

• Đối tượng sử dụng sản phẩm này là ai?

• Mục đích của sản phẩm này dùng để làm gì?

• Sản phẩm này sẽ làm việc ra sao?

• Cần có những phần cứng và phần mềm nào của sản phẩm?

Bước 2: Lập chiến lược kiểm thử (Develop Test Strategy)

Define Scope of Testing ( Xác định phạm vi kiểm thử)

Một Scope (phạm vi kiểm thử) đúng sẽ giúp đội nhóm có được những thông tin chính xác về quá trình kiểm thử. Đồng thời họ sẽ biết được những nội dung nào được kiểm thử (in-scope) và những gì không (out of scope).

Phần quy trình xác định phạm vi kiểm thử trong việc xây dựng Test plan gồm 4 giai đoạn:

• Precise customer requirement: Tại đây bạn cần nắm được yêu cầu chính xác của khách hàng

• Project Budget: Tiến hành thiết lập ngân sách dự án.

• Product Specification: Nằm được đặc điểm kỹ thuật sản phẩm

• Skills & talent of your test team: Kỹ năng & tài năng của nhóm kiểm thử của bạn

Identify Testing Type (Xác định loại kiểm thử)

Mỗi testing type được xây dựng để tìm ra một loại Bug cụ thể. Tuy nhiên tất cả các loại kiểm thử đều hướng tới mục tiêu chung là phát hiện sớm các lỗi trước khi phát hành sản phẩm cho khách hàng. Tùy theo loại sản phẩm hay thuộc tính ở giai đoạn test mà người tạo plan sẽ chọn testing type khác nhau.

Là người quản lý kiểm thử thì bạn cần phải cân nhắc mức độ ưu tiên của các loại kiểm thử. Loại kiểm thử nào được tập trung để kiểm thử ứng dụng web hoặc loại nào nên bỏ qua để tiết kiệm chi phí.

Document Risk & Issues (Rủi ro và vấn đề về Tài liệu)

Risk là sự kiện có xác suất xảy ra trong tương lai rất thấp và có khả năng thua lỗ. Khi Risk xảy ra thì nó sẽ trở thành issue. Ở QA Test Plan thì bạn sẽ ghi lại những rủi ro đó:

Risk	Giải pháp giảm rủi ro
Các thành viên trong nhóm thiếu những kỹ năng cơ bản để kiểm thử trang web	Lúc này bạn cần lên kế hoạch đòa tạo để năng cao kỹ năng cho các thành viên.
Project schedule quá eo hẹp để hoàn thành dự án này đúng thời thời gian	Bạn cần đặt mức độ ưu tiên (Test Priority) cho từng hoạt động kiểm thử.
Test Manager có kỹ năng quản lý kém	Cần lên kế hoạch đào tạo cho người quản lý kiểm thử.
Thiếu sự hợp tác ảnh hưởng tiêu cực đến năng suất làm việc của các thành viên	Hãy động viên, khuyến khích mỗi thành viên trong nhóm thực hiện nhiệm vụ của mình và truyền cảm hứng để họ nỗ lực hơn.
Dự toán sai ngân sách và chi phí tăng	Bạn hãy thiết lập scope trước khi bắt đầu công việc, chú ý nhiều đến việc lập planning dự án và liên tục theo dõi và đo lường tiến độ.

Tạo Test Logistics

Ở Test Logistics thì người quản lý kiểm thử cần trả lời các câu hỏi:

• Who will test ? (Ai là người thực hiện kiểm thử): Để lựa chọn được thành viên phù hợp bạn cần xem xét năng lực của họ có đáp ứng được điều kiện cho nhiệm vụ cùng ước tính ngân sách dự án. Chọn sai thành viên cho nhiệm vụ sẽ gây ra các dự án thất bại hay chậm trễ. Người thực hiện kiểm thử cần có những kỹ năng sau: hiểu được quan điểm của khách, mong muốn chất lượng tốt. Chú ý đến các chi tiết và có sự hợp tác. Trong dự án thì người chịu trách nhiệm kiểm thử là Tester và dựa trên ngân sách dự án thì bạn có thể chọn thành viên trong nội bộ hoặc outsource Tester.

• When will the test occur? (Khi nào sẽ thực hiện kiểm thử): Các hoạt động kiểm thử cần phải được kết hợp với các hoạt động phát triển liên quan. Bạn hãy bắt đầu kiểm thử khi các mục yêu cầu thỏa mãn 3 yếu tố: tài liệu kiểm thử, nhân lực cùng môi trường Test.

Bước 3: Xác định mục tiêu kiểm thử (Define Test Objective)

• Test Objective (Đối tượng kiểm thử) là mục tiêu tổng thể của toàn bộ dự án. Đó là tìm ra càng nhiều lỗi của phần mềm càng nhiều càng tốt để đảm bảo rằng phần mềm không có lỗi trước khi phát hành. Việc xác định đúng mục tiêu sẽ giúp cho việc kiểm thử sản phẩm diễn ra nhanh chóng và suôn sẻ hơn.

• Nếu muốn xác định Test Objective thì bạn cần liệt kê các tính năng phần mềm (functionality, performance, GUI…) có thể cần kiểm thử. Từ đó xác định mục tiêu, mục đích của kiểm thử dựa trên các tính năng trên.

Bước 4: Xác định tiêu chí kiểm thử (Define Test Criteria)

Tiêu chuẩn hoặc quy tắc để quá trình kiểm thử sản phẩm diễn ra đúng chuẩn đó chính là Test Criteria (Tiêu chí kiểm thử). Gồm có 2 loại tiêu chí chính:

Suspension Criteria (Tiêu chí đình chỉ)

Đây là tiêu chí phát hiện ra bugs trong quá trình kiểm thử. Nếu trong quá trình test xuất hiện Suspension Criteria thì chu kỳ kiểm thử hoạt động sẽ bị ngưng lại cho đến khi các tiêu chí được xử lý.

Exit Criteria (Tiêu chí thoát)

Nhằm mục đích xác định các tiêu chí thể hiện sự hoàn thành thành công của giai đoạn kiểm thử. Exit Criteria là kết quả được hướng tới là mục tiêu của test để chuyển sang giai đoạn phát triển tiếp theo. Cách thức giúp xác định tiêu chí kết thúc của test dựa trên 2 tỷ lệ:

• Run rate là tỉ số giữa số trường hợp đã kiểm thử trên tổng số trường hợp kiểm thử trên kế hoạch. Chỉ số này bắt buộc là 100%

• Còn Pass rate là tỷ số giữa số các trường hợp kiểm thử pass trên số trường hợp được thực hiện kiểm thử. Yếu tố này phụ thuộc vào phạm vi dự án và Pass rate càng cao càng tốt.

Bước 5: Hoạch định nguồn lực (Resource Planning)

Ở bất cứ dự án nào thì việc hoạch định nguồn lực là điều cần thiết bởi nguồn lực luôn có giới hạn và riêng biệt dựa theo từng Project. Các nhà quản lý kiểm thử nên liệt kê và xác định rõ nhân lực cùng tài nguyên hệ thống cho dự án để lên kế hoạch để chạy, hoàn thành dự án hợp lý.

Bước 6: Lập kế hoạch môi trường kiểm thử (Plan Test Environment)

Test Environment là một thiết lập của phần mềm và phần cứng để nhóm test sẽ tiến hành các trường hợp kiểm thử. Môi trường test bao gồm: người dùng cuối, môi trường kinh doanh, môi trường chạy UI, máy chủ,… Để thiết lập Test Environment thì bạn cần có sự hợp tác chặt chẽ giữa Test Team và Development Team. Môi trường kiểm thử lý tưởng sẽ cho phép Tester giám sát mọi biến động của phần mềm trong điều kiện sử dụng thực tế.

Bước 7: Lập lịch trình và dự toán (Schedule & Estimation)

Ở bước này bạn cần thiết lập project schedule vững chắc trong Test Planning. Bạn có thể phân chia cả quá trình thành những nhiệm vụ (task) nhỏ. Để Test Manager dễ dàng phân bổ thời gian cùng nhiệm chi tiết cho từng task theo dõi tiến độ dự án và kiểm soát chi phí. Để tạo project schedule, người quản lý kiểm thử cần các yếu tố đầu vào gồm:

• Employee và project deadline: Thể hiện ở ngày làm việc, deadline của dự án cùng tài nguyên sẵn có là những yếu tố ảnh hưởng đến lịch trình.

• Project estimation (Dự toán công trình): Dựa vào estimation, Test Manager sẽ biết được thời gian hoàn thành dự án để xây dựng project schedule phù hợp.

• Project Risk(Rủi ro dự án): Việc hiểu được rủi ro giúp Test Manager thêm đủ thời gian vào project schedule để đưa ra phương án xử lý hiệu quả

Bước 8: Xác định phân phối thử nghiệm (Determine Test Deliverables)

Sản phẩm hoàn chỉnh đã trải qua quá trình kiểm thử là có thể phát hành. Khi tiến hành kiểm thử phần mềm thì các phân phối thử nghiệm là danh sách tất cả các tài liệu, công cụ. Cùng các hiện vật được khai thác để phát triển và dùng để hỗ trợ quá trình Test. Các loại thử nghiệm có thể phát hành gồm:

• Phân phối trước khi thử nghiệm gồm tài liệu kế hoạch kiểm tra và bộ thử nghiệm các trường hợp thử nghiệm.

• Phân phối trong quá trình test: tập lệnh thử nghiệm, dữ liệu test, ma trận truy xuất nguồn gốc cùng nhật ký thực thi và lỗi.

• Phân phối sau thử nghiệm: Kết quả thử nghiệm và báo cáo, quy trình cài đặt, ghi chú phát hành, báo cáo sai xót.

Test instructions

Test instructions are displayed below the Access section and they consist of three sections:

• The goal of this test: This section usually covers the general purpose of the test. It often names what to focus on, e.g. new feature that is about to be released.

• Out of scope: What is not supposed to get tested is specifically stated here. On live websites, you may oftentimes not complete orders at the end of a checkout process, or some areas of a website or an app might not be ready yet. Make sure to stick to this information – disregarding them can have big consequences.

• Additional requirements: If there is anything else to point out, information will be given in this section. This might be information concerning the whole test, credentials for user accounts, dummy payment information, etc.

Test Plan

A Test Plan is a detailed document that describes the test strategy, objectives, schedule, estimation, deliverables, and resources required to perform testing for a software product. Test Plan helps us determine the effort needed to validate the quality of the application under test. The test plan serves as a blueprint to conduct software testing activities as a defined process, which is minutely monitored and controlled by the test manager.

As per ISTQB definition: “Test Plan is A document describing the scope, approach, resources, and schedule of intended test activities.”

Let’s start with following Test Plan example/scenario: In a meeting, you want to discuss the Test Plan with the team members, but they are not interested – .

In such case, what will you do? Select your answer as following figure

A) I am Manager do everything as I said

B) OK, let’s me explain why we need a Test Plan

Incorrect

As a Test Manager, you must explain them the importance of Test Plan rather than force the team to do what you want.

Correct

As a Test Manager, you must explain them the importance of Test Plan rather than force the team to do what you want.

Level Test Plans

As the name suggests, a level test plan describes in detail the testing activities that must be carried out for every test level or, sometimes, test type. Normally, the test levels listed in the master testing plan are explained in the level test plan.

They should provide the testing schedule, targets, test activities, test templates, etc., which are not mentioned in the master plan.

Test planning in a test management tool

Test management tools can help your team with their test planning efforts. Whether your goal is to be more agile or accelerate test planning, there are tactical approaches to making a comprehensive and agile test plan with a test management tool like TestRail.

1. Milestones

In TestRail, milestones are containers for aggregating test artifacts and tracking different testing activities related to the same outcome. For example, if you create a milestone in TestRail to keep track of a release and connect all of the related test runs and test plans, then you can see all of the different activities you will do for that release in one location.

Image: Manage all your milestones and ongoing test projects in TestRail.

Within TestRail’s milestones, the description field provides a great place to build a one-page test plan. By having all of your test plan information connected to the milestone from the beginning, you can constantly refer back to it as you are planning and ask yourself questions such as, “What was in scope?” “What was out of scope?” “What are we testing or not testing?” and “Are we going to do automated testing in this milestone?”

Milestones give you an efficient way to quickly refer back to your test plan information from within your actual test management tool.

1. Test case priority and type

Test cases define what you will test beforehand; essentially, they outline what you will do before you do it. In TestRail, you can organize test cases based on many hierarchies. This is a key way to start building your test plan.

For example, if you’re developing an advanced messaging app, the highest risk area to your app is that it has to be able to be installed and run. In this example, you might start with that as your smoke test and then do more in-depth functional or exploratory testing.

By capturing test case priority and what type of testing approach you plan to take with a particular test case ahead of time, you are starting to think about planning that test case in practice.

Image: Organize your TestRail test case repository based on priority.

1. Test Reports

Using a test management tool like TestRail for your test execution is beneficial because you can use real-time reports. For example, if you built your plan out in TestRail and you’re executing around that plan, a report like TestRail’s milestone summary report is going to show you your initial test objectives, your initial one-page test plan, all the test runs and plans added within that milestone, the priority you assigned to them, and more. You can also download reports to share with your team or other stakeholders.

Image: The Milestone (Summary) report allows users to attach project criteria and quickly visualize a clean, shareable interpretation of milestone activity.

As projects become more complex, using spreadsheets as test plan templates can become unwieldy, and a more sophisticated approach is needed. Use a test management tool like TestRail to make your test plans as flexible as possible.

Requested devices

You will usually be invited to the cycle to use a specific device of yours. This device is displayed in the right sidebar on the test overview page. For ideal device coverage, our distribution algorithm invites testers for different devices. You may only use the requested devices that were assigned to you. When testing websites, you must not only use the correct device but also one of the available browsers on the list.If the seats for the requested device are taken, you may be able to join the test with a different device if there are other slots opened. On the ‘Submit a Bug’ page you can check with which device you are in the test.

More articles on Software Testing

• How can you ensure testing stays within user story scope in Scrum?

  1 contribution

• How can AI testing solutions improve your ROI?

  60 contributions

• What are the most effective testing tools for API testing with Postman?

  119 contributions

• You need to test your mobile app. What tools can help you ensure quality?

  249 contributions

• Your developers are stuck in a testing rut. What software tools can help them break free?

  85 contributions

• Your team is facing a difficult software testing challenge. How can you overcome it?

  141 contributions

• Your software testing automation is underperforming. What steps can you take to improve it?

  81 contributions

• You’re looking for software testing automation tools. How do you know which features to prioritize?

  85 contributions

• Your enterprise needs software testing automation. What are the best tools to get started?

  133 contributions

• You’re looking to improve your API testing. What tools should you consider?

  133 contributions

• You’re managing a team of remote workers. How can you keep everyone engaged and motivated?

  101 contributions

• You’re tasked with API testing. What’s the best software to help you do it right?

  86 contributions

• You’re in charge of testing a new web app. How do you ensure it’s bug-free?

  136 contributions

• You need to test your software on a tight budget. What are your options?

  200 contributions

Features

Each test contains at least one of what we call features. A feature can describe an area of a product, e.g. a landing page, product overview page or product detail page of a webshop, it can describe a process like a checkout process of a website or an app, or it can even be a particular functionality that should be tested. For example:

When a customer sets up a new test, so-called standard features will be added by default. Customers can use these standard features and modify them if they want or add completely new features. You can learn more about some of our standard features here.

You can only submit bugs if a corresponding feature exists in the test. For example, if none of the given features covers the checkout process, you cannot submit bugs related to it.

More relevant reading

• Software TestingWhat’s the best way to plan and execute software testing for a complex project?
• Software TestingHow have you overcome software testing challenges?
• Internet ServicesWhat are the most effective ways to manage software testing for your team?
• Software Project ManagementHow do you add software testing to the software project life cycle?

What is Test Plan?

What is a Test Plan?

A Test Plan is a critical and exhaustive document that outlines the strategies, goals, timeline, estimations, deadlines, and resources needed for the successful completion of a project. It provides a framework that is designed by QA managers to provide clarity about the necessary tests that you need to verify to ensure the proper functioning of the software.

Why is Test Plan creation important?

Did you know according to the Department of Commerce’s National Institute of Standards and Technology (NIST) software failures cost the US economy about USD 59.5 billion annually or about 0.6% of the GDP (gross domestic product).

You can eliminate 1/3 of the cost with improved testing infrastructure and planning. Even though it may not be able to eliminate all errors cannot be eliminated through.

• Reduction in defects: Identify and mitigate defects early on in the software lifecycle.
• Improved quality: Studies indicate that projects equipped with thorough test plans generally experience reduced development and maintenance expenses. Identifying and addressing issues in the early stages is more economical compared to dealing with them after the software release.
• Resource utilization: Data indicates that the presence of a test plan facilitates improved planning and allocation of resources, averting bottlenecks and ensuring the efficient execution of testing activities.
• Faster time to market: The implementation of effective testing strategies specified in the plan plays a role in streamlining the development process. This leads to quicker time to market.
• Mitigate risks: A well structured plan helps in identifying risks earlier in the process. These can be cross-collaborative risks as well as risks that are internal to the team.

Who needs Test Planning?

Test plan is an important artifact which, when well drafted, can act as an anchor for the product development process. However, these personas benefit more from its creation.

• Developers, who get insights into the testing scope and requirements. This helps them anticipate the tests that will be executed on their code and understand the success criteria for testing.
• QAs, who create and execute test cases to find bugs and document them. Test planning helps them establish testing strategies, allocate resources, and track the overall advancement of the testing process. Test Plan functions as a roadmap for carrying out test cases, monitoring test advancement, and guaranteeing thorough coverage.
• Product Managers, who use test plans to make informed decisions about release timelines, resource allocation, and overall product quality. It helps them coordinate release activities, assess customer impact, and foster effective collaboration between development and testing teams to deliver high-quality software.
• Business Analysts, who ensure all the test cases are well aligned with the business requirements specs at every stage, keeping users’ interests in mind. Through test planning, they get a clear vision of the relevance of test cases with current user requirements.
• Compliance Teams, ensure that the testing procedures conform to predefined standards as per regulatory demands.
• Support Teams, who communicate with users about known issues, provide timely solutions and offer guidance on workarounds. Test Plans help them anticipate potential issues or bugs that have been identified during the testing phase and figure out their workarounds to help users.

Difference between Test Strategy and Test Plan

A test strategy is a high-level outline that provides an approach and framework for testing. A test plan is a detailed document that provides guidance around specific test activities, scope, deliverables, and resource projections for the project.

Here’s a detailed comparison of Test Strategy vs Test Plan:

Components of a Test Plan

A test plan usually comprises various elements that offer in-depth details regarding the testing procedures for a particular software application or system.

The essential components of a test plan include:

• Scope: Outlines the goals of the specific project and provides information about user scenarios intended for testing purposes. The scope may also specify scenarios or issues that will not be addressed by the project if required.
• Schedule: Specifies the commencement dates and deadlines for testers to provide outcomes.
• Resource Allocation: Specifies the allocation of specific tests to individual testers.
• Environment: Describes the characteristics, setup, and accessibility of the test environment.
• Tools: Specifies the tools that will be employed for testing, reporting bugs, and other pertinent activities.
• Defect Management: Outlines the procedure for reporting bugs, including the designated recipients and the required accompanying elements for each bug report. This may include specifications such as whether bugs should be reported with screenshots, textual logs, or videos demonstrating their occurrence in the code.
• Risk Management: Specifies potential risks that could arise during the testing of the software and the risks the software might face if released without adequate testing.
• Exit Parameters: Specifies the cessation point for testing activities. This section delineates anticipated outcomes from QA operations, providing testers with a standard against which they can measure actual results.

How to create a Test Plan?

When you use test management software, you get an organized view of your test cases and clarity of steps for execution. With BrowserStack Test Management, you get an added advantage of not just defining your test cases, but also directly monitoring and running your tests using the configurations of devices and operating systems.

Best Practices for creating an effective Test Plan

Some of the key best practices for creating effective test plan are:

• Understand project requirements and map to test cases.
• Clearly state the objectives of the testing effort.
• Clearly define the scope of testing, outlining which features and functionalities will be tested.
• Document the expected deliverables of the testing process.
• Define the test environment, detailing the hardware, software, and network configurations.
• Identify potential risks associated with the testing process and the project.
• Create a detailed testing schedule with milestones and deadlines.
• Create a realistic and achievable testing schedule.
• Maintain flexibility to tweak the plan, if required.
• Include scope for retrospection and avoid pitfalls in the future.
• Define key metrics to be collected during testing.
• Conduct retrospectives to identify areas for improvement in the testing process.

Conclusion

Test planning in software testing is the backbone of the entire project. Without a sufficiently extensive and well-crafted plan, QAs are bound to get confused with vague, undefined goals and deadlines. This unnecessarily hinders fast and accurate testing, slowing results and delaying release cycles.

BrowserStack Test Management lets you manage the creation of test cases, importing test cases, syncing with bug tracking software, and running tests across devices, operating systems, and browsers through a unified interface.

Chat information

The team leader or Customer Success Manager might have provided additional information about the scope or posted important reminders. Please always check the chat if you see that there are new messages. Whenever you have any questions about the environment or the scope of the cycle, please use the chat to ask for the Team Leader or even other testers’ help.

Out of scope

Out of scope means that something is not supposed to be tested. Note that the following points are always out of scope (unless stated differently in the test description):

• Legal problems. We are not legal advisors and the severity of a bug is not determined by legal provisions, frameworks, or standards.

• Problems related to browser extensions, ad-blockers, or virus scanners, e.g. blocking certain contents or even the execution of apps.

• Website content should not be changed, e.g. by using the Google Translator service to translate the website into a different language, as it may lead to unexpected behaviour.

• Set up problems in tests.

Setup problems

Setup problems are not legitimate bugs – it only means there is a problem with the test setup or the test environment. Please report them to Team Leader via the test chat who will in turn contact the CSM. Here are a few examples:

• The URL given in the test environment section of a test is wrong (404 error).

• The test environment can only be accessed with credentials, but none are given or the given ones don’t work.

• The test environment requires a proxy or VPN to be accessed but this information is missing in the instructions (403, 1020 errors).

• An app cannot be downloaded or installed.

• The environment is not available for some time while testing. Remember the customer might update the environment during a test run, so make sure your issues are not temporary ones.

• The customer provided login or payment credentials for user accounts but they don’t work.

• A link on a staging environment redirects to their live website. The link was not updated to point to the staging environment during the test environment setup.

• The device requirements for a test e.g. list iOS 9 but the app under test can only be installed on iOS 10 and above.

The scope is unclear or contains contradicting information

If you don’t understand something about the scope or if you find contradicting information, please contact the test’s Team Leader via the test chat as described in the article Confusing Instructions. Sharing such information helps us improve the setups of future tests.

How to create a test plan

Define the release scope

Before any test activity occurs, it’s important to define the scope of testing for your release. This means defining the features or functions that need to be included in the release, considering any constraints and dependencies that can affect the release, and determining what type of release it is.

Examples of questions to ask when defining the release scope include:

• Are there new features being released in this version?
• What are the risk areas?
• Are there any particularly sticky areas where you’ve seen regressions in the past?
• What type of release is it? Is this a maintenance release that includes bug fixes? Is this a minor feature release? Is this a major feature release?
• What does being “done” actually look like for your team?

For example, what information would you require if your organization has just launched a new e-commerce site and wants to test it before it launches?

Whether it’s talking with the developers to understand the scope of the project, or working with a product manager to walkthrough new functionalities and user flow, defining the scope ensures that accurate information is being shared and that there is a common understanding of the product’s goals, expectations, and features.

Schedule timelines

Specify release deadlines to help you decide your testing time and routine. Here are some pointers for determining timelines:

• Consult your project manager to understand the current release timeline.
• Look at past release times and schedules.
• Consider extraneous elements: Does the release need to coincide with outside variables, such as conferences or events? Factor those into your release date prediction.
• Consider the timeframes for development: Your development team might have a set schedule for finishing development work. Make sure you comprehend that timeframe so you can adjust the testing schedule.
• Add some extra wiggle room: It’s common to encounter unexpected delays. Including extra time for unforeseen events can help you stick to your plan.
• Review and update the schedule frequently to ensure the test timetable is attainable.

Define test objectives

A test objective is a reason or purpose for designing and executing a test. These objectives ultimately help guide and define the scope of testing activities.

Examples of general test objectives include:

• Identifying and reporting defects
• Testing new features
• A certain level of test coverage

Examples of objectives for specific types of testing include:

• Functional testing objectives: Ensure the software works as it should. Examples of goals for this objective include: Validating user workflows, data processing, and verifying input/output parameters.
• Performance testing objectives: Ensure the software is efficient and can handle various loads. Examples of goals for this objective include: Verifying software reaction time, throughput, and scalability.
• Security testing objectives: Uncover program security flaws. Examples of goals for this objective include: Verifying authentication and authorization features and identifying potential threats.
• Usability testing objectives: Concentrate on ease of use and user experience. Examples of goals for this objective include: Validating software accessibility, verifying user flow, and identifying user-related issues.

Measure testing with the right metrics

Metrics assess the overall quality of a release, the progress of your testing, and the effectiveness of your testing (for a particular test cycle or the entirety of your testing).

They provide visibility into your testing process and overall product quality, ultimately helping your team decide if your release is ready to ship. Here are some metric formulas you might consider:

• Defect Density = Defect count/size of the release (lines of code)

Example: If your software has 150 defects and 15,000 lines of code, its defect density is 0.01 defects per line of code.

• Test Coverage = (Total number of requirements mapped to test cases / Total number of requirements) x 100.

Defect Detection Efficiency (DDE)

• DDE = The percentage of defects detected during a phase / Total number of defects

Time to Market

• TTM = The time it takes for your company to go from idea to product launch

Determine test deliverables

Test deliverables are the products of testing that help track testing progress. Deliverables should meet your project’s and client’s needs, be identified early enough to be included in the test plan, and be scheduled accordingly. There are different test deliverables at every phase of the software development lifecycle. Here are important deliverables to focus on before, during, and after testing:

Before testing

• Test plan document: The scope, objectives, and approach of the testing endeavor are all outlined in the test plan.
• Test suite: Test cases illustrate how to run a test, including input data, expected output, and pass/fail criteria.
• Test design and environment specifications: The test environment outlines the hardware and software configurations used for testing.

During testing

• Test log: The test log records each test case’s results, including issues and resolutions.
• Defect report: A defect report lists testing issues by severity, priority, and reproducibility.
• Test data: According to the International Software Testing Qualifications Board (ISTQB), test data is data created or selected to satisfy the execution preconditions and input content required to execute one or more test cases.
• Test summary report: The test summary report lists the number of tests run, passed, and failed, as well as open defects.

After testing

• Test completion report: Covers the testing scope, product quality, and lessons discovered.
• User acceptance test (UAT) report: Points to any issues found and fixed.
• Release notes: List information about what the release includes. Examples include any new features for development, advancements, or fixes.

A test plan’s content and structure differ depending on its context. Although there isn’t one cookie-cutter way to write a test plan, following best practices for test plan development can help companies deliver high-quality software.

TestRail is test plan software designed to make it easy to follow best practices for test plan development. In TestRail, you can enter test cases with preconditions, test instructions, expected results, priorities, and effort estimates.

Image: Organize and structure reusable test cases in folders, create agile test plans, and track test execution progress in TestRail.

This level of flexibility and visibility into your testing process makes TestRail an easy fit into any organization’s test plan — Try TestRail for free to see how it can help with your test planning. Or, learn more about how to build and optimize your testing processes—from test design to test planning and execution with this free TestRail Academy course, Fundamentals of Testing with TestRail.

Design the test strategy

Test strategy helps determine test cost, test effort, and which features will be in-scope (planned to be tested) versus out-of-scope (not planned to be tested).

Identify testing types

It is critical to identify when to perform what type of testing, what should be tested manually vs. automated, the scope of automated tests, how much work will be required to create new test cases, and who will be doing that work.

Depending on several factors, there may be various types of testing to include in your test plan.

Examples of factors to consider when choosing the right testing type to perform include:

• Test objectives
• Your project’s feature requirements
• The complexity of your product
• Your team’s experience levels
• Regulatory requirements
• Time and budget

Here are commonly used types of testing to consider including in your test plan:

Document risks and issues

It’s essential to document risks that may occur during testing and the effect of these risks. Risks can include:

• Strict deadlines
• Insufficient or inaccurate budget estimates
• Poor management
• Problems with the code
• Changes in the business environment
• Limited resources for testing
• Unexpected delays during testing

Document test logistics

Test logistics should answer the “Who, what, where, when, and how.” Documenting test logistics ensures that all human and system-related testing resources are available. For example, it may be important that your team identifies who is available to do testing and who will support them if needed during testing. Moreover, when resource planning, it can be helpful to identify alternative resources or build slack into your plan to ensure your project gets completed.

Establish test criteria

Test Criteria is a standard that regulates all activities within a testing project. The two main types of test criteria include suspension and exit criteria.

• Suspension Criteria: Establishes the conditions for suspending all tests.
• Exit Criteria: Exit criteria are established items or goals to complete that define the end of a test phase. The exit criteria of a test are the predetermined results that must be achieved to move on to the next testing phase. For example, 92% of all critical test cases must pass before a feature can be deemed suitable for release to your customers.

Plan the test environment and test data

Planning a test environment guarantees precise and robust testing. The test environment includes hardware, software, and network configurations for software testing. Follow these procedures to set up the test environment:

• Determine your hardware and program requirements: Select test environment devices and software, including operating systems, browsers, databases, and testing tools.
• Install the required software: Once prerequisites are established, install the necessary tools on the test environment. This may require setting up a separate server to host the application and installing a database management system or other tools.
• Configure the network: Make sure that firewall protocols, IP addresses, and DNS settings, among other network configurations, are identical between the test and production environments.
• Create the test data: Prepare the test material for the application’s testing. Test data can be created manually with data from the production environment, retrieved from an existing production environment and database, or, created via automated Data Generation Tools.
• Access the builds: Ensure that the builds that the testers will be testing are accessible. One example is setting up a file-sharing or version control system to allow testers access to the most current builds.
• Verify the test environment: After setting it up, check that your test environment fulfills the requirements.

What do you Measure Test Coverage Against?

The one thing many QA teams don’t consider while measuring test coverage: What Do You Measure Test Coverage Against? Or How to Measure the Test Coverage? Or How to Ensure Test Coverage?

Test coverage measured against lines of code. It is a Test Execution Coverage Percentage which is discussed above. For example, If you have executed 800 lines of code through test cases, out of 1000 lines of code, then your test coverage is 80%.

Now, you need to measure test coverage by requirements to focus on good test cases in your test repository. A good test case traces a requirement (happy and unhappy flows included) to fulfilment. All you have to do to ensure you mainly have good test cases is to establish Requirements Traceability.

You can achieve traceability in your projects by mapping out test scenarios to cover all the requirements in scope for that release, project, iteration, sprint. By establishing Requirements Traceability, you know – at any given point in time – test coverage by requirements.

Test Plan

Schedule a personalized demo.

Our team will get in touch with you within 12 hours

Get in touch with us

Please share some details regarding your query

Thank you

Request received!

We will respond back shortly to

Request received!

We will respond back shortly to

In the meantime, here are some resources that might interest you:

Để quá trình kiểm thử đi theo đúng hướng đã định ra thì Test plan có vai trò rất quan trọng. Công việc Kiểm thử đòi hỏi người thực hiện phải có kỹ thuật, kinh nghiệm và tiến hành theo quy trình định trước. Bất kỳ Tester nào cũng cần hiểu rõ khái niệm Test plan và biết cách xây dựng kế hoạch chi tiết và hiệu quả. Theo dõi bài viết sau đây của Testerpro.vn để hiểu rõ hơn nhé!

Conclusion

Development is more systematic nowadays and organizations seek measures of testing completeness and effectiveness to show test completion criteria. Of them all, test coverage is considered especially valuable.

To ensure that our software testing services are at par, we believe in a structured approach since the software test coverage is a critical factor to measure and ensure the quality of software.

How do you use test coverage to ensure the software quality? Let me know in the comments.

What are Master Test Plans & Level Test Plan?

Welcome to the Manual Testing Free Training series.

In the last tutorial, we talked about the test plan documents. Please refer to the following link:

Read Test plan document

One-page test plan template

If your test plan doesn’t fit onto one page, don’t worry. The intention is to minimize extraneous information and capture the necessary information that your stakeholders and testers need to execute the plan.

Test Coverage Techniques

#Statement Coverage

Statement coverage ensures that all the statements in the source code have been tested at least once. It provides the details of both executed and failed code blocks out of total code blocks.

Let’s understand it with the example of the flow diagram. In the given example, this path 1A-2C-3D-E-4G-5H covers all the statements and hence it requires only on a test case to cover all the requirements. One test case means one statement coverage.

In complex code, a single path is not sufficient to cover all the statements. In that case, you need to write multiple test cases to cover all the statements.

Advantages:

• It can be applied directly to object code and does not require processing source code.
• It verifies what the written code is expected to do and not to do

Disadvantages:

• It covers only true conditions of every statement.
• Does not report when loop reaches to the terminations.
• Statement coverage is completely insensitive to the logical operators (|| and &&).

Statement coverage is the basic coverage and hence does not guarantee 100% coverage.

#Decision/Branch coverage

It is impossible for developers to write code in a continuous mode, at any points they need to branch out the code to meet the functionality requirements. Branching in the code is actually a jump from one decision point to another point. Branch Coverage checks every possible path or branch in the code is covered.

Branch coverage can be calculated by finding the minimum number of paths which ensure that all the edges have been covered. In the given example, there is no single path that ensures coverage of all the edges at one go.

For example, if you follow this path 1A-2C-3D-E-4G-5H which covers the maximum number of edges(A, C, D, E, G and H), you will still miss the two edges B and F. You need to follow another path 1A-2B-E-4F to cover the remaining two edges. By combining the above two paths you can ensure of travelling through all the paths. For this test coverage example, our branch coverage is 2 as we are following two paths and it requires two test cases to meet the requirements.

Advantages:

• It covers both the true and false conditions unlikely the statement coverage.
• It validates if all branches are tested.

Disadvantages:

• It ignores branches within Boolean expressions which occur due to short-circuit operators.

#Path Coverage

Path testing is a structural testing method that involves using the source code of a program in order to find every possible executable path.

Path Coverage ensures coverage of all the paths from start to end. In this example, there are four possible paths:

1. 1A-2B-E-4F
2. 1A-2B-E-4G-5H
3. 1A-2C-3D-E-4G-5H
4. 1A-2C-3D-E-4F

Advantages:

• It helps to reduce redundant tests.
• Path coverage provides high test coverage because it covers all statements and branches in the code.

Disadvantages:

• Testing each path is challenging as well as time-consuming because a number of paths are exponential to the number of branches. For example, a function containing 10 if-statements has 1024 paths to test.
• Sometimes many paths are impossible to exercise due to relationships of data.

#Condition Coverage

Condition coverage checks if both the outcomes(“true” or false”) of every condition have been exercised. The outcome of the decision point is only relevant for checking the conditions. It requires two test cases per condition for two outcomes.

Advantages:

• Condition coverage measures the conditions independently of each other.
• It has better sensitivity to the control flow.

Disadvantages:

• Similar to the branch/decision coverage.

#Boundary Value Coverage

This coverage metric is very useful for those applications in which error occurred due to input numbers. And these errors occurred at boundary values. In boundary value coverage, test cases are selected at the endpoints of the equivalence classes. For this test coverage example, below are boundary values for an application which requires 3-digit number as an input.

1. 100(Minimum)
2. 99(Just below the minimum boundary value)
3. 999(Maximum)
4. 1000(Just above the maximum boundary value)

Advantages:

• It is quite easy to test a small set of data in place of testing a whole lot of data sets.
• It is easy to use because of it’s easy to automate nature and uniformity of identified tests.

Disadvantages:

• It can not test dependencies between two inputs.
• It can not cover code which contains boolean functions.

#Product Coverage

Product coverage refers to the process of testing a software solution from a product perspective and evaluating specific product areas. It involves evaluating the performance of software across various scenarios by

• Defining and prioritizing requirements
• Developing a checklist
• Implementing effective test automation

#Risk Coverage

The risk coverage technique is used to comprehensively evaluate all software-related risks and rigorously test the application. It involves identifying and assessing all potential risks and the issues that may arise from them, including low-probability scenarios that could significantly impact the software.

#Requirements Coverage

Requirements coverage is used to assess whether the end-product encompasses all the committed functionalities and fulfills the customer’s stipulated requirements. It measures the number of requirements addressed and the types of tests executed to validate them.

#Compatability Coverage

Compatibility coverage is a technique used to ensure the software works seamlessly across all browsers, operating systems, and devices without any issues or complications. It involves testing the product’s compatibility-related problems, including mobile testing, hardware testing, browser testing, network testing, and other subtypes.

Test coverage in Test Driven Development (TDD)

Following TDD makes sure each new development has automation test coverage and the behaviour is safeguarded iteration after iteration. Generally, writing tests first is strongly correlated with better coverage, and better coverage is strongly correlated with fewer bugs. It also tells us that writing tests first correlates with better API designs.

Let’s take an example to understand how TDD can facilitate better test coverage. At Simform, we follow 2-week sprint as a part of agile methodology in software development.

• Sprint – Week 1

In the planning session, our engineering team shortlists the x most important stories on the backlog for delivery. The Quality Assurance team analyses the new sprint backlog and maps out all the test scenarios they would be covering in their test cases. These scenarios are then passed over to the development team for doing unit testing.

• Sprint – Week 2

While the QA team was busy preparing initial test cases, the development team has kicked off programming and looking into the test case scenarios. In the second week, testers have completely scripted new test cases and prioritized the test scenarios. Thus at this moment, the tester are pretty confident about the test coverage.

• Sprint – Week 3

Developers have used the test scenarios and test cases to complete unit testing of their codes. This has led to weeding out of easy to find and conspicuous bugs. Further the code is deployed using continuous integration and continuous deployment tools to facilitate end-to-end functional testing. Tester use the detailed test cases to execute end-to-end functional testing and identify critical bugs.

As we get to the end of Sprint, the team is ready for the product release with a good 85%+ test coverage and no critical bugs. Thus Test driven development makes a high test coverage possible which leads to robust product and satisfied customer.

However, if you are doing TDD by the book you should always end up with 100% test code coverage because you will never add new functionality to the application, a new branch in code, without first having failing tests for that specific new piece of code.

What are Test Coverage Metrics?

Test coverage metrics measure the test effort and help answer, “How much of the application was tested?” Test coverage metrics can be divided into three parts: code-level metrics, feature testing metrics and application level metrics. There are various test coverage formulas to calculate these results and generate test coverage reports.

For example, “Of these tests that are passing or failing, what are the artefacts or area of my application that they are designed to ensure my products are produced in high quality.” Below are 5 key test coverage metrics.

Code level metrics

#Test Execution Coverage Percentage

It also called as executed tests and it is a percentage of passed/executed tests out of the total number of tests.

The advantage is that you get an overview of testing progress by counting the number of passed and failed tests.

The disadvantage is that counting test cases passed doesn’t say anything about the quality of those tests. For example, some tests might pass because it checks the trivial condition or some error in the test code while it’s not functioning as per requirement.

Feature testing metrics

#Requirements coverage

Requirements coverage is used to determine how well the test cases cover the software requirements. For that, you simply need to divide the number of requirements covered by the total number of scoped requirements for a sprint, release or project.

In the Requirements module, you create test coverage by linking tests to a requirement. Test coverage assesses the impact of a change in the test or requirement.

Instead of covering each requirement only at the level of the test, you can cover a requirement by test configurations. A test configuration represents a specific use-case of a test. Covering test configurations with requirements provides finer granularity by covering multiple use cases.

#2. Test cases by the requirement

This metric is used to see what features are being tested and the number of tests that aligned with a requirement. Most of the requirements contain multiple test cases. It is very important to know which test cases are failed for a particular requirement to rewrite the test cases for specific requirements.

This metric is very important for stakeholders as it shows the progress of the app/software development.

Application level metrics

#Defect Density

Defect density is a measure of the total known defects divided by the size of the software entity being measured.

It is used to identify the areas which require automation. If defect density is high for the specific functionality than it requires retesting. To reduce the efforts of retesting, test cases for known defects can be automated.

It is important to consider the priority of the defect (low or high) while evaluating the defects. For example, multiple low priority defects may pass because the acceptance criteria have been satisfied. And on the other hand, only one high priority defect may prevent acceptance criteria from being satisfied. Higher-priority defects are generally weighted more heavily as part of this metric.

#2. Requirements without Test Coverage

After calculating the requirements coverage, you will find some requirements which are not covered. Now, it is important to know about each requirement which has not been covered and what stage the requirement is in.

This metric helps test engineers and developers to identify and eliminate uncovered requirements from total requirements before they send them to the production phase.

Requested devices

You will usually be invited to the cycle to use a specific device of yours. This device is displayed in the right sidebar on the test overview page. For ideal device coverage, our distribution algorithm invites testers for different devices. You may only use the requested devices that were assigned to you. When testing websites, you must not only use the correct device but also one of the available browsers on the list.If the seats for the requested device are taken, you may be able to join the test with a different device if there are other slots opened. On the ‘Submit a Bug’ page you can check with which device you are in the test.

Keywords searched by users: scope in test plan

Categories: Phát hiện thấy 79 Scope In Test Plan

See more here: kientrucannam.vn

See more: https://kientrucannam.vn/vn/