Scaling Music and Software

I am working on a lecture for my economics class about production theory.  Production theory works nicely for software development, music and    information workers too.

Output is the result of labor and capital.  The amount of music produced is the result of musicians (labor) and instruments (capital).    As more and more musicians are added to the process of making music the process of making music becomes more formal and there are overhead costs.

If we start with one musician playing a piano, then there is not much overhead required.  Add  three musicians and you get a quartet and there is some communication that needs to take place.  Add about 96 more musicians and you get an orchestra.

The cost per output (the average product) begins to rise because you have overhead costs.  If you have about 100 or so musicians you need a conductor.   Sheet music is required too (formal documentation not necessarily required for a small group).  We see the marginal cost, the cost to produce additional music, begins to rise.  The reason are additional labor is required that does not produce music directly.

This same model works for software development too.  Output for software is the amount of functionality produced.  As the project scales upward formal processes are just required.   There project resources (labor) required such as a project manager.  Formal methods and documentation is required to help in communication.

So there you have it…. Production theory works for information workers.

What we have here is a failure to communicate

Why do bureaucracies emerge in organizations and often overwhelm it?

The real question in my mind is, “What is the root cause for an organizations to have too much bureaucracy?”  It is not like organizations or more correctly individuals within organizations wake up proclaiming we need more bureaucracy.   People do not say, “We need to burden teams with status reporting, forms, procedures until we reach a point in time where everything comes to a screeching halt.”

The idea of small autonomous teams that are unhampered by bureaucracy is not new.   During the 1990’s there was a software methodology called Rapid Application Development (RAD) and today the buzz is Agile. This idea of limiting bureaucracy is not limited to software development.  Going all the way back to the 1940’s there were teams called “Skunkworks” at Lockheed Martin.   The point being idea of self directed teams is not unique to software development and not a new idea.

Some of the clues for burdensome processes come from the word bureaucracy itself.  The term bureaucracy came into use shortly before the French Revolution of 1789, and from there rapidly spread to other countries. The Greek suffix – kratia or kratos – means “power” or “rule”. Does it turn out that bureaucracies are the result of power and rule?

My experience tells me it is not that simple. The root cause for bureaucracies is the result of poor communication.  The cause of bureaucratic burdensome processes is the result of a failure to communicate.   An organization can eliminate burdensome processes, but it does not resolve the problem of communication.  If root causes are not addressed, then problems will never be resolved.

The bottom line is  bureaucracies emerge because of poor communication.  Go though the two links below and look at the recoomendations.  We will begin to notic the opposite is done in software development.  As an example, it is recommended to avoid jargon.  How often is that done in software development.

99 Ways to Improve Communication

Improving Communication Skills


Software Librarians

There is a job title in the software domain called software librarian. I searched for the job title of software librarian on Monster.Com.   It does not seem that software librarian jobs require any background in library science.   Instead the job description indicated they wanted someone who could code.  When I searched for medical librarian or law librarian the following appeared in every job description.

You must possess the following requirements and experience to succeed in this position: Master’s Degree in Library Science from an ALA accredited program.

What is the motivation for the fields of medicine and law to require a Masters in Library Science to organize their documents while there is no requirement whatsoever for the software industry?   These other industries have realized how critical it is to organize knowledge, so it can be retrieved and reused.

I suggested to a client they hire a librarian or two to help them organize their software application.    He looked at me and asked me one of those obvious questions, “where do we find a librarian.” I suggested he look for a librarian at the library which they did.  They hired two librarians to consult and help establish a numbering system similar to the Dewey Decimal System for their software applications.  Like the DDC their internal numbering system was a faceted system.  This allowed a lot of cross-referencing of documents. In the end they had an online system where they could search for documents by a specific part of the application.

A librarian, with a Masters degree in Library Science, makes between $36,980 and $56,960 per year.[i] The average salary of a software developer is around $69,240.  You can get a librarian for less than a programmer and they are going to be twice as productive (maybe more) as a software developer when it comes to organizing.

Read more at Reboot! Rethinking and Restarting Software Development – Interesting ideas on design and organization of website and information. – interesting ideas on how information is shifting.  It is coming to us in a variety different ways now.

[i] Bureau of Labor Statistics

Benefits to documentation (yes there are some)

When application documentation is given meaningful names and numbers, it can be found easily. Software organizations need to establish a standard naming convention for documents and the application.  Documents need to be named using some agreed upon standard and a numbering system. 

By definition a complex software application is difficult to understand and needs to be documented well.  Yet, I have heard comments like, “our software is too complex to document.”   If your software application is too complex to document, then the software application is too complex to understand and your software needs to be simplified.  If your software application is too complex to document, then how will the complexity be communicated?

 No one wants to be a software archeologist.  One problem with unorganized information is it can’t be found, so there is a lot wasted time spent pecking around looking for the right document or trying to find the right person to talk to.   Another problem is if your documentation can’t be found, then redundant functionality is often created.  Of course creating redundant functionality is a waste of time.    The real benefit to the organization of documentation is reducing wasted time and spending more productive time on projects.  This, of course, improves profitability. 

 Another benefit to organized documents is the ability to reuse parts of the application documentation.   Reuse has been a hot topic in software development for sometime and it should not be limited to just the code.   Everything related to past projects can be re-used including requirements, test plans, test data, even project plans can be reused.  Reuse is a good idea and it does improve both productivity and quality.   Artifacts that can’t be found can’t be reused.  Those companies that have mastered reuse see a natural increase in their productivity and profitability.

 Read More at Reboot! Rethinking and Restarting Software Development

Welcome to Club Chaos!

The secret of all victory lies in the organization of the non-obvious.
– Marcus Aurelius (Roman Emperor AD 121-180)

Things Just Seem Complex

Software applications and documentation seem more complex when they are not organized. It does not matter if it is software applications, books, collections, or lives anything that is not organized seems more complex than it really is.   As John Maeda writes in The Laws of Simplicity, “Organization makes a system of many appear fewer.”   If applications and associated artifacts are not organized, then there is only a sense the application is complex.  By organizing application artifacts complexity of applications is reduced.

I have been to a lot of grocery stores around the world, including Rome, and it seems like they are all organized in a similar manner.  In any grocery store there is the fruits & vegetable section, meat & cheese, bread, canned goods, frozen foods, snack foods, liquor, so on and so forth.

Organized documentation and functionality is important for everyone especially the novice.  A grocery store is a complex system. Since the grocery store is organized, it is easy for the novice clerk to re-shelf the contents of the grocery store and the novice customer to find what they are looking for.   A novice is a person who is trying to understand something for the first time.  Think about all the different manufacturers of products or the number of countries where products originated to stock the grocery store shelves.   The goods only represent one layer of complexity.  Another layer is the scheduling of employees to clean the store, to shelve, and to run the cash register.  The beauty is all the complexity has been hidden from the customer.   The grocery store is an organized complex operation; yet, our interactions seem simple.

When software applications evolve without a comprehensive design, functions get scattered around and it becomes hard to navigate all the confusion.  Since there is no clear functional decomposition of the application, the user of the application and developer have a hard time finding a specific piece of functionality.   The application is difficult to navigate and users become frustrated.

Developers have a hard time finding functionality or understanding the functionality that already exists.   This causes redundant functionality to be created.  This extra functionality is not needed and it becomes additional expense to create and to maintain.

The perceived complexity is worse when multiple applications share information and communicate via undocumented interfaces.  If documentation is not organized which describes all the complex interrelationships it is going to be next to impossible to grasp what is actually happening.

Changes in Lattitudes, Changes in Attitudes

As Jimmy Buffet sings, “Reading departure signs in airports reminds me off all the places I have been.”  I am on my way to California today and whenever I travel it reminds me of all the places and people I have met before.  After 3 million frequent flyer miles, working in every possible industry, and on every contentent there are somethings that are very predicatble.  I can count on software developers saying or maintaining the following positions.

1. Measurement is not possible in the software industry because it is too complex and unpredictable.

Many in the statistics profession, including myself, do not believe in random events.  Some statisticians and mathematicians go mad trying to find patterns.   There is no need to go mad in the domain of software development because many of the patterns are blatantly obvious.   The process only appears to be random because it has not been studied.  It is common for a clinical psychologist to see patterns in an individual’s behavior while the individual is totally blind to his or her own patterns of behavior.   One of the functions of a clinical psychologist is to get people to recognize these patterns and behaviors.   The most common way for psychologists to get people to recognize their patterns is through measurement and keeping a journal.

Many software organizations write journals which are more commonly known as lessons learned documents.  I often review lessons learned documents, and it is painfully obvious the same patterns repeat over and over again. If all the past lessons learned documents are pointing to the same problems, then it is safe to conclude the next lessons learned document is going to point out the same problems. There are only three possible reasons for software organizations to keep repeating the same behaviors: the first is denial and ignoring the glaring fact that the same bad stuff happens in every project, the second is “not my fault syndrome” or nothing can be done about it, the third and final reason it is just too hard to change.

2.  My intuition works better than systematics study.  A wet finger in the air seems to work best for us.

The software developer prefers to use intuition instead of systematic study.  In a method of intuition, what makes logical sense must be true.   I have heard many software developers say, “I know what I know.”   It reminds me of the lyrics from the song “What I am” by The New Bohemians: “I’m not aware of too many things, but I know what I know if you know what I mean.”   If knowledge cannot be communicated via some type of, measurement it is difficult to understand what is meant.  If any process is “too complex” or “random” to be studied using the scientific method by applying measurements, then it is a safe bet that it cannot be studied using intuition either.

Read more at Reboot! Rethinking and Restarting Software Development

Unskilled and Unaware

Many software organizations are left with the mistaken belief they are doing fine when in actuality they are not.  Individuals in these  organizations actually believe they are doing a “quality” job .   These individuals  have this false belief because they do o not have the fundamental skills necessary to realize they are actually doing a poor job.

One of my favorite psychology papers is called Unskilled and Unaware.  The premise of the paper is when people are incompetent they suffer a dual burden.   First, they reach erroneous conclusions and make bad choices.  Second, their incompetence robs them of their ability to realize they are making bad choices and they are left with the mistaken belief they are doing fine.   Since they think they are doing fine, they will never learn from their mistakes.  They will never make an effort to improve until disaster strikes.

Those skills making one competent in a particular domain are often the very same skills necessary to evaluate competence in the domain.  Because of this, incompetent individuals lack what cognitive psychologists call “metacognition” or self-monitoring skills.  In other words they are just too ignorant to realize they have a problem.

Read more at Reboot! Rethinking and Restarting Software Development.

May the Force Be With You

I don’t know why I waste my time teaching statistics and quantitative methods when I could be teaching people how to use the force instead.  Do you remember how Obi-Wan Kenobi (Ben), teaches young Skywalker how to use the force?  Luke is training against some type of floating ball.  Of course Luke can’t hit the small beams coming at him from inside the floating ball.    Obi-Wan blindfolds young Luke with a helmet and says, “act on instincts.” Obi-Wan encourages Luke to ignore his eyes. Once Luke is blindfolded and uses the force he is able to hit all the small beams coming at him. In software development we call act on instincts expert opinion.  It seems like software developers use the force.

Too often when  someone  is asked, “How did you come up with your estimate?” they reply, “expert opinion.”  They are acting on instincts instead of using quantitative analysis.  Don’t get me wrong often instincts can give you insight into estimates and understanding projects status, but it should not be a substitute for quantitative methods.

The best part of the movie Star Wars Episode IV is when Luke is making his final approach to destroy the Death Star.   He is flying at a couple times the speed of sound and he has to hit a small target which is about one foot by one foot.   Time is running out and all the other attempts have failed to destroy the Death Star.   Luke hears the voice from the other side (Obi-Wan), the voice says, “Use the Force Luke.”  Luke looks around and thinks about it and the voice tells him, “Luke trust me,” so Luke turns off all the quantitative instruments.  Of course, Luke launches his rockets guided only by instinct and they are right on the mark and the Death Star is destroyed.  My advice is when creating an estimate, or when asked about project status it is best to let the force guide you.  When asked how you came up with your estimate simply respond, “The force was with me” and refer them to the Star War movies or this blog.

Just imagine how your next performance review could be worded.  In the Stars Wars movie Darth Vador says about Luke, “The force is strong with this one.”  Now wouldn’t it be great if your boss would write that about you.  That is something you could put on your resume.

Read more at Reboot! Restarting and Rethinking Software Development.

I want be an average software developer

Several years ago my sister called to tell me about my niece.  My niece, who was in 2nd grade, had taken some standardized tests and  she scored average.  My sister was distraught.  I tried to explain there is nothing wrong with average.  Average means there are 50 percent of the below.  Yes, my sister responded, “but it always means there are 50% above too.”

I am surprised at the number of people who write me asking what is the average productivity rate of software development.  The average productivity rate happens to be 42.  It is if they want to strive for the numerical average.  They are asking, “We know we are not very good, so we want to become average.”  These same organizations benchmark against average software organizations instead of best in class organizations.

The other day when I left my house to catch a flight my wife kissed me good bye and said, “Have an average trip.”   No she did not really say that.  She said, “Have a great trip!”  Software development organizations should strive to become great not average.   The leaders of software organizations need to understand what are the specific things best in class organizations do.  They need to be asking the question, “What are those specific things they do that are different from average and especially worst in class software development organizations?”

My niece recently graduated from Marquette University with honors and will be attending dental school.  It turns out she is above average after all.

Have an average weekend!

Read more at Reboot! Rethinking and Restarting Software Development

What is the distribution of effort of a software project

I received an email this morning asking, “What is the effort distribution across a software development life-cycle?”  This is a great question and it is important information for your organization to learn and understand.  I have been tracking this type of data since the 1980’s.

It turns out there is a relationship between how a software organization spends its time and productivity.  The more time it spends in coding and testing the lower the software productivity and quality rates.    Those organizations that spend the most time in requirements and analysis have the highest productivity rates.  The data is conclusive on this point.

In 1980 the best in class software organizations spent about 80 percent of it’s times in coding and testing.  Today the best in class organizations spend about 25% of it’s time in coding and testing.    Today those organizations with the lowest productivity and quality rates spend the bulk of time it’s coding and testing.

While the data does vary depending on the type of industry, the conclusion remains the same regardless of industry.  For example,  the Healthcare industry has the highest productivity and quality rates of any industry.   Those HealthCare IT organizations that spend the most time in requirements and analysis have much higher productivity rates than those HealthCare IT organizations that spend the bulk of its time in coding and testing.

The bottom 25 percentile have a breakdown as follows.

Requirements 15%
Design & Analysis 10%
Coding  35%
Testing  40%

The top 25 percentile have a breakdown as follows.

Requirements 35%
Design & Analysis 40%
Coding 10%
Testing 15%

Read more at Reboot! Rethinking and Restarting Software Development.