SW Testing: 64 Essential Test Metrics For Measuring Software Quality, Testing & More

Should you measure things while testing? Sure. In order to be able to manage the phase (and the resources available to execute it) it is mandatory... What should you measure? Here is an interesting article on the topic:

• 64 Essential Test Metrics For Measuring Software Quality, Testing & More!

Quoting:

"Software testing metrics are a way to measure and monitor your test activities. More importantly, they give insights into your team’s test progress, productivity, and the quality of the system under test. When we ask ourselves “What have we tested?”, metrics will give us better answers than just “we have tested it.” Different teams measure various aspects depending on what they want to track and control or improve.

A metric usually conveys a result or a prediction based on the combination of data.

Result Metrics:  metrics that are mostly an absolute measure of an activity/process completed.

Example: Time taken to run a set of test cases in a suite

Predictive Metrics:  metrics that are derivatives and act as early warning signs of an unfavorable result.

Example: Defects created vs. Resolved chart shows the rate of defect fixing.  This grabs the team’s attention if this rate is slower than the rate desired.

Are you using the right software test metrics? Get answers, best practices and strategies in our 14-page guide to improving your software quality through data and analytics.

Why Test Metrics? Why Should You Care?

The aim of collecting test metrics is to use the data for improving the test process, rather than to just show fancy reports. This includes finding tangible answers to the questions:

• How long will it take to test?
• How much money will it take to test?
• How bad are the bugs?
• How many bugs found were fixed? reopened? closed? deferred?
• How many bugs did the test team did not find?
• How much of the software was tested?
• Will testing be done on time? Can the software be shipped on time?
• How good were the tests? Are we using low-value test cases?
• What is the cost of testing?
• Was the test effort adequate? Could we have fit more testing in this release?

Good answers to these questions need measurement. This post includes 64 of the absolute, derivative, result, and predictive metrics that testers and QA managers use the most often.

The Fundamental Metrics 

As a tester, your road to metric creation has to start somewhere.  Fundamental QA metrics are a combination of absolute numbers that can then be used to produce derivative metrics.

Absolute Numbers:

• Total number of test cases
• Number of test cases passed
• Number of test cases failed
• Number of test cases blocked
• Number of defects found
• Number of defects accepted
• Number of defects rejected
• Number of defects deferred
• Number of critical defects
• Number of planned test hours
• Number of actual test hours
• Number of bugs found after shipping"

For the derivative and other type of metrics, follow the link above.

REUSE: The Elements of Reusable Code (Chris Wash)

An excellent (oldie but goldie) article summing up the elements of reusable code:
The Elements of Reusable Code

More from Chris Wash:
https://www.captechconsulting.com/search#q=Chris Wash

SECURITY: The visible face of your projects could be the wrong one

Customer are always important but some customers and its projects will need additional care. Sometimes the visible face of your work gets into the news and this is good. Unless the news are bad, or even worse, are incorrect.

Only then you'll probably be aware that sometimes it is just not a matter of being the best in business and let your technical (and management) competency excel.
It's a matter of surviving what it is being said about your work, the way you got your contract, the way you delivered your work, the way you could get the blame for issues that are not part of the scope of your work and so on. It's then that you'll have to have your executive management support (to shield you from the outer world interferences and to manage communication with that outer world).

The oddest experience I've ever had was having to work with a point of contact that was to help us execute a specific project and from minute one his intentions were to undermine the project's success. I had 20+ years of experience in the software industry, and in my innocence, I was thinking I had nothing more to learn. I was so wrong,

What if the visible face of your project start to be in the news? The news could be correct and/or incorrect; presenting sensitive data and/or confidential data (that no one knows who's disclosing). Your competition could be interested in it. Internal stakeholders could be interested in it (feeling "invaded"). Reporters could be getting it all wrong. It can get messy...

Some examples (in PT, sorry guys):

• http://www.publico.pt/sociedade/noticia/citius-tem-um-mes-para-ficar-bom-colapso-era-afinal-previsivel-1670159
• http://www.computerworld.com.pt/2013/05/27/critical-reage-a-noticias-sobre-citius/
• http://www.publico.pt/sociedade/noticia/auditoria-ja-tinha-alertado-para-problemas-na-rede-informatica-dos-tribunais-1669185
• http://observador.pt/especiais/como-a-rede-de-toupeiras-do-benfica-foi-descoberta-pela-policia-judiciaria/ (2018-03: you could be wondering what BENFICA has to do with the rest of this article, but it does).
• http://silvaonsoftware.blogspot.pt/2016/07/resource-security-is-process.html

Maybe some more can be told on this topics later on. Lessons learned are for sure more useful if properly shared. And you bet that we've managed to deliver successfully that particular [migration] project.

Revisions: 2018-03 - updated links. 2018-04 - added the "drop money out of the window" link.
PS: Additional info or how to not execute a "data migration" project:

• https://www.publico.pt/2014/09/01/sociedade/noticia/site-citius-continua-indisponivel-no-arranque-do-mapa-judiciario-1668298#/0
• https://www.publico.pt/2014/09/01/sociedade/noticia/sistema-informatico-dos-tribunais-esta-a-voltar-a-normalidade-assegura-ministerio-da-justica-1668350
• https://tek.sapo.pt/noticias/computadores/artigos/citius-investimento-de-26-milhoes-de-euros-na-plataforma-tera-sido-desperdicado (or, how to drop out of the window ~2.5 M EUR) 

TOOLS: Modeling with Sparx Systems EA (Reloaded)

Talking of using EA for modeling? Strategic Modeling? Business Modeling?
Take a look at these free PDFs from the tool maker:

• http://www.sparxsystems.com.au/downloads/quick/strategic_modeling_with_enterprise_architect.pdf
• http://www.sparxsystems.com.au/resources/user-guides/guidebooks/business-modeling-techniques.pdf

Modelling: EA initial management tasks

Some common management tasks for EA (to be performed by the EA UML repository manager, which could be the Technical Manager of your project) include:

Activate Security - Turn on the option that allows you to define who alters what part of the model and who does not (lock, unlock):
http://www.sparxsystems.com/enterprise_architect_user_guide/8.0/projects_and_teams/usersecurity.html

Managing Users - Create profiles, users belonging to profiles and defining permissions for those:
http://www.sparxsystems.com/enterprise_architect_user_guide/8.0/projects_and_teams/managingusers.html

Import Users from Active Directory - Integrating existing (e.g. domain) users into EA 
http://www.sparxsystems.com/enterprise_architect_user_guide/8.0/projects_and_teams/import_user_ids_from_active_di.html

Initial EA Scripts - The DB schema that has to be loaded into the SQL DB initially: 
http://silvaonsoftware.blogspot.pt/2016/10/ea-initial-scripts.html


(@2017-06-23: Minor rephrases)

Mr. Scrum wrote yet another book: "Scrum: The Art of Doing Twice the Work in Half the Time"

Jeff Sutherland (one of the inventors of Scrum) wrote one more book that looks interesting:

• Scrum: The Art of Doing Twice the Work in Half the Time (Barnes and Noble link)
• https://www.amazon.com/Scrum-Doing-Twice-Work-Half/dp/038534645X/ref=sr_1_1?ie=UTF8&qid=1518784352&sr=8-1&keywords=jeff+sutherland (Amazon link)

Quoting:
"We live in a world that is broken. For those who believe that there must be a more agile and efficient way for people to get things done, here from Scrum pioneer Jeff Sutherland is a brilliantly discursive, thought-provoking book about the leadership and management process that is changing the way we live.

In the future, historians may look back on human progress and draw a sharp line designating “before Scrum” and “after Scrum.” Scrum is that ground-breaking.  It already drives most of the world’s top technology companies. And now it’s starting to spread to every domain where leaders wrestle with complex projects.

If you’ve ever been startled by how fast the world is changing, Scrum is one of the reasons why. Productivity gains of as much as 1200% have been recorded, and there’s no more lucid – or compelling – explainer of Scrum and its bright promise than Jeff Sutherland, the man who put together the first Scrum team more than twenty years ago.

The thorny problem Jeff began tackling back then boils down to this: people are spectacularly bad at doing things with agility and efficiency. Best laid plans go up in smoke. Teams often work at cross purposes to each other. And when the pressure rises, unhappiness soars. Drawing on his experience as a West Point-educated fighter pilot, biometrics expert, early innovator of ATM technology, and V.P. of engineering or CTO at eleven different technology companies, Jeff began challenging those dysfunctional realities, looking  for solutions that would have global impact.

In this book you’ll journey to Scrum’s front lines where Jeff’s system of deep accountability, team interaction, and constant iterative improvement is, among other feats, bringing the FBI into the 21st century, perfecting the design of an affordable 140 mile per hour/100 mile per gallon car, helping NPR report fast-moving action in the Middle East, changing the way pharmacists interact with patients, reducing poverty in the Third World, and even helping people plan their weddings and accomplish weekend chores.

Woven with insights from martial arts, judicial decision making, advanced aerial combat, robotics, and many other disciplines, Scrum is consistently riveting. But the most important reason to read this book is that it may just help you achieve what others consider unachievable – whether it be inventing a trailblazing technology, devising a new system of education, pioneering a way to feed the hungry, or, closer to home, a building a foundation for your family to thrive and prosper."

http://www.barnesandnoble.com/w/scrum-jeff-sutherland/1117655162

Updated - 2018-02-16, Amazon link;

RESOURCE: EA additional documentation

Sparx's Enterprise Architect (EA) UML Modelling Tool has loads of documentation in the WWW (aside from the local online help documentation that typically is very self-contained and informative). Several task-oriented docs exist and are made available via the WWW web site. Some interesting examples are presented here (direct links for download):

Reference Booklets (PDF List):

• http://www.sparxsystems.com.au/resources/booklets/

Getting Started:

• http://www.sparxsystems.com.au/downloads/resources/booklets/getting_started_with_enterprise_architect.pdf (Getting Started)
• http://www.sparxsystems.com.au/downloads/resources/booklets/using_enterprise_architect.pdf  (Using EA)
• http://www.sparxsystems.com.au/bin/EAUserGuide.pdf (User Guide)
• https://www.sparxsystems.com.au/resources/user-guides/basics/tutorial.pdf (Tutorial)

Requirements management with EA:

• https://sparxsystems.com.au/downloads/whitepapers/Requirements_Management_in_Enterprise_Architect.pdf

Data Modelling:

• http://www.sparxsystems.com.au/resources/uml_datamodel.html
• http://www.sparxsystems.com.au/downloads/whitepapers/Database_Modeling_In_UML.pdf
• https://community.sparxsystems.com/white-papers/669-data-modeling-from-conceptual-model-to-dbms (from Conceptual to Physical)

When and how to use Version Control with EA:

• http://www.sparxsystems.com.au/WhitePapers/Version_Control.pdf

UML additional resources: 

• http://www.sparxsystems.com.au/resources/links.html

(Updated: @2016-10-10: Added version control PDF; 2018-03: updated broken links)

INDEX: The Agile/Scrum index of useful links

To gather some resources and information presented on this site (and mostly available in the internet) here it goes, the Agile/Scrum index of useful links (not exhaustive):

1. http://silvaonsoftware.blogspot.pt/2016/05/scrum-agile-main-documents-and-5-line.html (Resumo do SCRUM Official Guide: 16p. em 5 linhas ;)
2. http://silvaonsoftware.blogspot.pt/2016/07/book-sbok-guide-guide-to-scrum-body (Scrum Body of Knowledge)
3. http://silvaonsoftware.blogspot.pt/2016/04/resource-agile-videos.html (learning through videos)
4. http://silvaonsoftware.blogspot.pt/2016/06/book-actionable-agile-tools-jeff.html (BOOK with additional tools to use with Scrum, e.g. during some Scrum events)
5. http://silvaonsoftware.blogspot.pt/2016/04/quality-attributes-invest-on-proper.html (about the INVEST Model that each US shall respect)
6. http://silvaonsoftware.blogspot.com/2016/06/the-state-of-scrum-2015-report.html (The State of Scrum Report)

Scrum Definitive Guide | Scrum.org (Agile)

This is the link to "The Definitive Guide to Scrum" (developed and sustained by Ken Schwaber and Jeff Sutherland, scrum.org, and also known as Scrum Official Guide or SOG), 16 pages that changed the way we produce (some) software:

• http://www.scrumguides.org/docs/scrumguide/v1/scrum-guide-us.pdf: July 2013 version; older versions include May 2009 and October 2011. PS. In 2016 a new version of the guide was provided by the authors.
• The 2017 version is here: http://www.scrumguides.org/docs/scrumguide/v2017/2017-Scrum-Guide-US.pdf#zoom=100 and can be browsed online here: http://www.scrumguides.org/index.html
• In PT BR: http://www.scrumguides.org/docs/scrumguide/v1/Scrum-Guide-Portuguese-BR.pdf

(@2016-06-08: Added aka; 2016-10-17: added PS; 2018-02-21: rephrased, added link to V2017; added link to PT BR version)

The State of Scrum 2015 Report - Scrumalliance.org (PDF)

Interesting PDF report from the organization that manages the framework:

https://www.scrumalliance.org/scrum/media/scrumalliancemedia/files%20and%20pdfs/state%20of%20scrum/scrum-alliance-state-of-scrum-2015.pdf

Quoting:
"The State of Scrum reports offer comprehensive coverage of global usage of Scrum -- who is using Scrum, where, how, and why. The report also looks at how and where Scrum is best adopted and is growing. "

PS. Heard of it from the "Fluffy Side of Scrum" book authors @ Agile Portugal 2015 (2016-06-05).
(@2016-10-13: Added quote)