Coffeehouse Thread

7 posts

Question for C9 team and others.

Back to Forum: Coffeehouse
  • MisterDonut

    I was wondering. Too often in building software, we say that we want to build better software. Usually, this can be measured with metrics like Scalability, Reliability, Response Times, etc.. But, Quality is one of those things that is difficult to measure. Yet, quality is essential to all of us developers in order to say our software is a success. It used to be that people (wrongly) measured lines of code. Defect counts used to be used, but in the absence of decent test cases, this number can be artificially low. When defined as "an absence of bugs in the code", no one can measure what hasn't been found.

    My Question, how does the Channel 9 team members measure quality? Anyone else care to elaborate on their experience with measuring quality?

    *Note* I'm not looking for trolls, or jabs at Microsoft or others about how their code has laughable quality. I'm looking for good metrics to take and learn from.


  • mot256

    From an article I read somewhere:

    "Inspection with the aim of finding the bad ones and throwing them out is too late, ineffective, costly. In the first place, you can't find the bad ones, not all of them. Second, it costs too much. Quality comes not from inspection but from improvement of the process." 
    [W. Edwards Deming, the American statistician who drove Japan's post-war quality revolution] 
     
    The late W. Edwards Deming, an American statistician, was widely credited with providing much of the know-how and philosophy that drove the Japanese recovery. Ironically, his know-how was developed in America where it was largely ignored. Deming was sent to Japan in 1947, but it wasn't until 1980 when NBC featured him in a television program "If Japan Can...Why Can't We?" that the West finally took notice. Deming's message slowly spread from Japanese manufacturing into western manufacturing, then into the western service industries. 
     
    Deming's key message was to build quality into the processes rather than inspect defects out of a product already built. Although his concepts are acceptable to software development where quality is job number one, the concepts have either not actually been applied or not well. The majority of software development projects still follow the waterfall lifecycle that relies on inspecting defects out, that is, testing the product to remove defects. 
     
    Late Inspection 
    Late inspection and testing are the simplest, most expensive, and least effective way to find bugs. Before manufacturers adopted modern quality techniques their main approach to removing defects was to inspect each product in the last stage of production once it had been built, but before it was shipped. If the product didn't meet specifications, then it was either reworked or scrapped -- both expensive options. Late inspection is prone to human error and rarely finds all defects. 
     
    Late inspection is also the model primarily used when doing waterfall software development. Rework (which the waterfall is meant to minimize) is pushed to a phase of the project when fixes are the most expensive; when the cost of change is forty to 100 times greater than if the defect was fixed when it was created (Boehm, 2004). This stage has the least wiggle room time to recover from show-stopper problems or unexpectedly large amounts of rework. This unpredictability becomes one of the main reasons why projects miss their schedules. 
     
    Early Inspection 
    Recognizing the high cost of late inspection, manufacturers moved back the inspection process by using self and successive inspection. Self inspection involves operators inspecting their output immediately after processing it. Successive inspection happens when operators inspect the inputs from prior processes. The equivalent processes in software development are walk-throughs, inspections, and unit testing. These approaches provide much earlier feedback when the correction costs are much lower, but still rely on human judgment. 
     
    Mistake Proofing – Building Quality into the Process 
    A more powerful approach is to design your processes to automatically prevent defects from happening. The Japanese name for such techniques is Poka Yoke, meaning mistake proofing. Poka Yoke has two aspects: prevention and detection. For instance, having different shaped plugs for each type of connection in a PC prevents novices and experts from making mistakes when installing a new PC, and the electric fuses and circuit breakers in your home detect when electrical circuits are overloaded which prevents electrical fires. 
     
    Likewise, one of the most common causes of defects – ambiguous requirements - can be prevented by writing comprehensive acceptance tests when each requirement is captured. Furthermore, automating these tests and running them as part of frequent integration-builds help detect defects when they happen. 
     
    The Missing Ingredient 
    Deming's core message was that we should stop inspecting defects out of products and start building quality in. Obviously preventing defects or finding them when they are cheapest to fix is preferable to finding them all at the end when they are many, many times more expensive to fix. Yet few software development projects write tests up front, do inspections, or frequent integration despite the benefits.  
     
    Why not? Because it is hard work in most environments. It's hard work trying to inspect hundreds of pages of documents. It's hard work trying to write tests for many pages of requirements at the beginning of a project. It is even harder to keep the tests up to date as the requirements change. It's harder still when you realize that you have to inspect the tests.  
     
    So what was different in manufacturing environment that allowed manufacturers to adopt an equivalent processes?  
     
    The primary difference is that when the Japanese manufacturers adopted Deming's quality practices and philosophy they also adopted what are now known as the lean manufacturing techniques. Lean factories have much less work-in-process compared to traditional factories and this allows them to spot defects much sooner. For instance, compare a factory where it takes three months from when material is released until the product is shipped with a factory where it takes only two days. If a systemic defect occurs in the first process, then it will take three months for the defect to be found in the first factory, but only two days in the second. At the point that the defect is found every product within the factory will have the defect and will need to be scrapped or reworked. The cost of rework and scrap is significantly lower in the factory with lower work-in-process (WIP).  
     
    Lean factories achieve much lower levels of WIP by working in smaller batches of work. The equivalent in software development is working in small increments as used by the Agile approaches. It is much easier write tests up front and to automate them when you are working on a much smaller set of functionality at any point in time. When working this way, defects are discovered much sooner, when they are easier to fix, the quality of the application is more transparent, and the odds of delivering on time much higher.”

  • scobleizer

    MisterDonut wrote:

    I was wondering. Too often in building software, we say that we want to build better software. Usually, this can be measured with metrics like Scalability, Reliability, Response Times, etc.. But, Quality is one of those things that is difficult to measure. Yet, quality is essential to all of us developers in order to say our software is a success. It used to be that people (wrongly) measured lines of code. Defect counts used to be used, but in the absence of decent test cases, this number can be artificially low. When defined as "an absence of bugs in the code", no one can measure what hasn't been found.

    My Question, how does the Channel 9 team members measure quality? Anyone else care to elaborate on their experience with measuring quality?

    *Note* I'm not looking for trolls, or jabs at Microsoft or others about how their code has laughable quality. I'm looking for good metrics to take and learn from.




    1) Do you like it? (you being collective you).
    2) Is it being used?

    Or, maybe it should be #2 first, then #1. Smiley

  • MisterDonut

    People,

    Thanks for your input. Both points are good.. But, not much of it (sans Scobles' "Is it being used") is measurable. I wholeheartedly agree Quality should be built into the process, fewer defects, etc...

    But, how does one measure "quality" in a contract? Everyone who has done software building as a consultant (internally or externally to their company) has requirements that often say something like "it should be "high quality" or "Quality should come first", etc. My problem is if I get assigned a task that has a contact in which Quality is of primary concern, how do I measure that? Some thoughts: 

          * I can't measure bugs, per se. I have to find them either during or after the software is being built. I can't have a "99% defect free" clause in a contract given to me. People could log erronously defects. And, what, exactly, is a defect? A bug found? Or a bug not found? (If a tree falls in the woods and no one hears it.. etc.)

           * Is Quality really a broad category in which reliability, scalability, usability, etc., reside? If I achieve everything under it, it's of "high quality"?

          * Use Cases completion? How many use cases can I work though with the software? (Ideally it's 100%, but sometimes they need to be curtailed or limited, due to time / cost contraints). 

    Has anyone else come up with measurable metrics? (That might be redundant.. but I wanted to stress the importants of measurement). 

    Robert, how do you and the C9 team measure, "Do you like it"?  If you go to Slashdot, I'm sure you'll find plenty of negative press on it (I seem to remember a few links from /. to C9). People who routinely use C9 probably like it (otherwise, why would you use it? We don't *need* to be here.. In fact, we all should be working.. Smiley ) What does C9 do to measure how much we like it?

    For the record, and the metric, I like C9. Smiley

  • mot256

    Got this from http://software.ericsink.com/ 

    Sorry, this is still not the “measurements” that you ask for, but I think it is quite amusing.

    P.S. I'm probably Group 1, but still would like too prevent it from happening  Smiley

    My life as a Code Economist (Friday, November 11, 2005)

    The six billion people of the world can be divided into two groups:

    1. People who know why every good software company ships products with known bugs.
    2. People who don't.

    Those of us in group 1 tend to forget what life was like before our youthful optimism was spoiled by reality.  Sometimes we encounter a person in group 2, perhaps a new hire on the team or even a customer.  They are shocked that any software company would ever ship a product before every last bug is fixed. 

    Every time Microsoft releases a new version of Windows, major news services carry a story about the fact that the open bug count is a five digit number.  People in group 2 find that sort of thing interesting.

    The surprising truth is that a person in group 2 can lead a productive life.  In fact, as with most any other kind of ignorance, they're probably happier that way.

    The exception of course is if you are a software developer.  That's an entirely different story.  If you earn your living building shrinkwrap software products, you need to get yourself in group 1. 

    Perhaps, dear reader, you are a software developer in group 2?  If so, I'm sorry I had to be the one to tell you this.  I feel like I'm telling the neighbor kid that Santa Claus isn't real.  Smiley

    Okay, seriously.  I apologize for the mock arrogance of the previous paragraphs, and for the Santa Claus remark.  I mean no condescension.  Just like everybody else, I've got lots to learn.  But I've also got lots of gray hair.  Some people read my stuff because they want to benefit from the lessons I learned from all the stupid mistakes I made which caused that gray hair.

    Which reminds me -- let me tell you a story...

  • blowdart

    scobleizer wrote:

    1) Do you like it? (you being collective you).
    2) Is it being used?

    Or, maybe it should be #2 first, then #1. Smiley



    Those are very very different. Often in corporate environments #2 is not a choice at all, it's delivered to the desktop and a user has to use it to do part of, or all of their job. In fact I would go as far as saying usage stats for software are a misleading guide to quality.

    Of course defining "quality" is a problem all on its own ...

    I worked on a bank project were sucess was measured by shipping on a date. Note that the software shipped did not have to actually work, and indeed didn't the release before I was invovled, they just had to have shipped *something* on the date they promised.

  • Flatliner

    I'll attempt an answer to your question, as i think it is an interesting one, if not a loaded question. software quality can be measured in many ways as 'quality' doesn't define an aspect or a requirement. Quality of what? usability, reliability? Our software is delivered as a service, so we have a quality of service contract which guarantees a certain level of uptime. But I get the impression you seek quality from a user acceptance perspective. In this respect, our software is measured in error reports, feedback and the number and nature of support calls we receive. When you ask how can you quantify qualitatively, remember its the opposite of quantitatively.

Comments closed

Comments have been closed since this content was published more than 30 days ago, but if you'd like to continue the conversation, please create a new thread in our Forums, or Contact Us and let us know.