REPORT: Standish Group 2015 Chaos Report - Q&A with Jennifer Lynch

Project failures, overall, by project size, Agile vs Waterfall and some contributing factors to success (in the opinion of the participants):
Standish Group 2015 Chaos Report - Q&A with Jennifer Lynch

Quoting:

"The 2015 CHAOS Report has recently been released by the Standish Group.  The CHAOS Reports have been published every year since 1994 and are a snapshot of the state of the software development industry.  This year the report studied 50,000 projects around the world, ranging from tiny enhancements to massive systems re-engineering implementations.  This year the report includes an enhanced definition of success looking at some additional factors which were covered in previous surveys.

The results indicate that there is still work to be done around achieving successful outcomes from software development projects. This table summarises the outcomes of projects over the last five years using the new definition of success factors (on time, on budget with a satisfactory result)"
(...)

Listed factors that could help in project success include:

"The definitions for these factors are:
Executive Support: when an executive or group of executives agrees to provide both financial and emotional backing. The executive or executives will encourage and assist in the successful completion of the project.

Emotional maturity is the collection of basic behaviors of how people work together. In any group, organization, or company it is both the sum of their skills and the weakest link that determine the level of emotional maturity.

User Involvement: takes place when users are involved in the project decision-making and information-gathering process. This also includes user feedback, requirements review, basic research, prototyping, and other consensus-building tools.

Optimization is a structured means of improving business effectiveness and optimizing a collection of many small projects or major requirements. Optimization starts with managing scope based on relative business value.

Skilled staff are people who understand both the business and the technology. A skilled staff is highly proficient in the execution of the project’s requirements and deliver of the project or product.

SAME is Standard Architectural Management Environment. The Standish Group defines SAME as a consistent group of integrated practices, services, and products for developing, implementing, and operating software applications.

Agile proficiency means that the agile team and the product owner are skilled in the agile process. Agile proficiency is the difference between good agile outcomes and bad agile outcomes.

Modest execution is having a process with few moving parts, and those parts are automated and streamlined. Modest execution also means using project management tools sparingly and only a very few features.

Project management expertise is the application of knowledge, skills, and techniques to project activities in order to meet or exceed stakeholder expectations and produce value for the organization.

Clear Business Objectives is the understanding of all stakeholders and participants in the business purpose for executing the project. Clear Business Objectives could also mean the project is aligning to the organization’s goals and strategy"

(...)

Success, namely project success is (re)defined as:

"The Standish Group has redefined project success as onTime, onBudget, with a satisfactory result.
Success is hard to define and we had a hard time coming to this conclusion.
Merriam-Webster dictionary defines success as the fact of getting or achieving wealth: respect or fame: the correct or desired result of an attempt; someone or something that is successful: or a person or thing that succeeds.
The Project Management Institute (PMI) has defined success as onTime, onBudget, and onTarget also known as the Triple Constraints and the Iron Triangle. However, we have seen many projects that have met the Triple Constraints and did not return value to the organization or the users and executive sponsor were unsatisfied."

SW Testing: Congestion?

Some good highlights of the dependencies SW Testing teams can have on other teams:
https://www.itproportal.com/features/how-to-prevent-software-testing-congestion/

Quoting:
"When it comes to software testing, each individual team in the software development and delivery process shares the responsibility of making sure the final outcome is exactly what is required and that it is delivered on time. (..) In a recent survey by PractiTest and Tea-Time with Testers, 47 per cent of test teams said coping with timeframes for development could often be difficult. (...)

What’s affecting software testing productivity?

The first step to fixing testing bottlenecks is identifying exactly what is slowing down enterprise software testing productivity. The biggest areas that impact this are often weak test prioritisation, poor dependency management and bad communication.

Weak Test Prioritisation
How is testing prioritised? How is test prioritisation being updated based on feedback from development and testing?

It is human nature to go after the low-hanging fruit, to get a few quick wins to show progress, and testing is no different. Unfortunately, this just pushes off the hard tests to later when there is less time to address any issues that arise.

In any situation, it is beneficial to prioritise testing the highest value and most risky pieces first. Also, not relying on initial estimates of what the highest value and most risky things are is important. Feedback from development can help teams focus on the right things first, while feedback from testing can help guide the process as it moves forward. Remember, defects tend to cluster, so as defects start popping up, re-evaluate where the focus needs to be.

Poor Dependency Management
Part of what can make software testing unpredictable is the reliance on so many dependencies lining up at the same time. Of course, this often doesn’t happen and when dependencies are mismanaged it can result in unnecessary downtime and a significant slowdown of the whole process.

Some of the things that software tests can be dependent on include:

·         Requirements

·         Other tests

·         Test data

·         Test environments

·         Development completion

·         Quality of code

Productivity depends on knowing a team’s specific dependencies, understanding how those dependencies are progressing, and managing those dependencies.

Mapping out and managing dependencies is fairly straightforward to talk about, but in an enterprise where multiple teams are working on multiple projects, it’s often not that easy. "

Estimation: PERT / Three Point Estimation Technique

The estimation technique in Agile is Planning Poker.
As for Waterfall, a combination of decomposition (WBS), PERT / 3 point estimation (weights sum up 6) and iterative reevaluation could be used.
What is exactly the 3 point estimation and PERT? Google says:

• www.techrepublic.com/blog/it-consultant/use-pert-technique-for-more-accurate-estimates/
• http://www.pmdocuments.com/pert-three-point-estimation-technique/
• http://www.pmbypm.com/3-point-estimate-is-different-from-pert/
• https://en.wikipedia.org/wiki/Three-point_estimation

Quoting Tech Republic:
"PERT is an estimating technique that uses a weighted average of three numbers (see below) to come up with a final estimate.

• The most pessimistic (P) case when everything goes wrong [also called "Worst Case Scenario"]
• The most optimistic (O) case where everything goes right [also called "Best Case Scenario"]
• The most likely (M) case given normal problems and opportunities [also called "Expected Case Scenario"]

The resulting PERT estimate is calculated as (O + 4M + P)/6. This is called a "weighted average" since the most likely estimate is weighted four times as much as the other two values. You'll notice that the final PERT estimate is moved slightly toward either the optimistic or pessimistic value - depending on which one is furthest from the most likely. Generally this ends up moving the final estimate toward the worst case, since the worst case value tends to be further out from the most likely that the optimistic number."

Some tools (like MS Project) will allow you to do some What If scenarios, by defining different weights  (above the weights are 1, 4, 1, and all weights should sum up to 6). What if it is most probable that mostly everything goes wrong (could be tested with weights like 1, 1, 4). What if mostly everything goes right (4, 1, 1)?

Main Work Products for Software Development Projects (Waterfall)

The waterfall phases aim to define:
- What to do? - requirements catalogue (mapping to features; features could map to requested features)
- How to do? - high-level architecture and (if needed) detailed design
- Execute! - produce the code, unit tests and reverse engineer the detailed design (if possible)
- Validate! - validate the system against requirements (are we doing things right?)
- Accept! - prove that the system solves the customer problems (to the customer or a customer representative). Are we doing the right things? Deploy (transfer).
- Maintain! - keep the system up and running (and optionally evolve it - new features, changed features)

The core work products (outputs of applying QMS Engineering processes) are:
- SRS (Requirements Spec)
- SAS (Architecture Spec document - High Level Design; could include sections for ICD, DBS)
- SDD (Detailed design which could include a section for the DBS)
- The Code
- STS (and the STR, i.e. the test report)
- ATS (and the ATR)

In order to execute the project you'll also need plans (management documents, outputs of applying QMS Management or QMS Support processes) as well as present reports:
- PMP (Project Management Plan),
- CMP (Configuration Management Plan);
- CIL (identifies the review strategy for each item - and lists external CIs)
- QAP (Quality Assurance Plan)
- SVP (SVVP, and some similar docs could exist due to regulatory requirements)

- SPAR (SW Product Assurance Report - metrics per milestone)
- Milestone Reports
- Progress Reports (per milestone or per time period - e.g. monthly)
- etc.

PS. Remember that not all these documents need to be deliverables: A Deliverable requires extra peer review and formal customer acceptance (and all this takes you away precious time). Typically the project proposal identifies the minimum set of deliverables.

(@2016-10-13: rephrases, expanded acronyms)