By David Lengacher

About a decade ago, researchers began to study the performance of e-government portals in order to identify best practices. Studies have also focused on identifying the factors that influence the information quality these portals offer to the public. Most of these studies consider only portal outputs, but ignore the resources (or inputs) used to develop and maintain the portal. This is a problem, because when you analyze only the outputs of a system you are only seeing half of the performance picture. In other words, you are only measuring effectiveness and ignoring efficiency. In an era of ever-tightening government budgets, we simply cannot afford to ignore efficiency in government services. Naturally then the question is: how do you measure the efficiency of e-government portals?

The answer lies is a little known type of analysis called Data Envelopment Analysis (DEA). Once viewed as tool only for serious economists, DEA has experienced an incredible surge in popularity in measuring the efficiency of systems, business units, and processes across dozens of business sectors. In fact, a quick search of Google Scholar yields thousands of published applications of DEA for this very purpose. While the details of DEA are beyond the scope of any blog post, I can provide you with the main idea of DEA and the challenges of trying to measure efficiency without it.

First, let’s review the definitions of efficiency and effectiveness. Efficiency is the degree to which inputs are utilized to produce outputs. Effectiveness is the degree to which stated goals and objectives are achieved. Thus, efficiency measures bang for the buck while effectiveness just measures the bang. Therefore, you will need both measures if you want to be able to see the big picture. In cases involving a single input and output, efficiency can simply be measured using one output/input ratio. However, things get much more complicated when you have multiple inputs and multiple outputs. Here we would have multiple ratios to evaluate. This is where DEA shines and become invaluable because it evaluates all of the ratios pertaining to a single entity (say Portal A) and determine its total efficiency relative to all other entities (say Portals B, C, and D). What DEA leaves you with is a single efficiency score for all of the portals under evaluation.

So what types of inputs and outputs would you feed into a DEA model? While every portal is unique, some common outputs might consist of information volume (breadth and/or depth), interaction level, integration level, and user participation level. Common inputs might consist of initial development cost, annual operating cost, and organizational capacity metrics.

I have used DEA to measure the efficiency of projects, strategies, and operations for over five years and can confidently say that it is an invaluable tool for understanding total organizational performance. For more technical information on DEA, please visit Google Scholar. If you would like more qualitative and illustrative examples, please see the links below or email me with your questions.

http://www.isixsigma.com/implementation/project-selection-tracking/lessons-crusoe-project-resource-allocation/
http://tdwi.org/research/2009/08/business-intelligence-journal-vol-14-no-3.aspx