Predicting Cost/Reliability/Maintainability of Advanced General Aviation Avionics Equipment

-6- and users. A complete list of information sources is included in Appendix A.

Using a portion of the information collected, we attempted to develop generalized estimating relationships for predicting purchase and operating costs of general aviation avionics. For reasons that are fully discussed in Section II, these attempts were unsuccessful.

REPORT OUTLINE Although cost, reliability, and maintainability are all closely interrelated subjects, they are covered in separate sections in this report. Past work in developing cost-estimating relationships, and an analysis of cost data collected for contemporary and new technologies, are covered in Section II. Conclusions on the development and use of generalized cost estimating relationships (CERs) for advanced avionics systems for general aviation are included at the end of the section.

Also included in Section II are recommendations as to how one should proceed when generalized methodologies are not available.

Section III contains a survey of the development of reliability prediction methods, a discussion of the accuracies and pitfalls in using the different methods, and ways to apply both prediction and assessment methodologies where applicable to avionics systems for gen- eral aviation aircraft that are projected for operation in the 1980s.

Section III ends with conclusions and recommendations on how to proceed with reliability prediction and assessment in the absence of parametric methodologies.

The subject of maintainability is covered in Section IV. Mili- tary and airline experiences are presented, and the applicability of -7- this experience to advanced avionics systems for general aviation is discussed. Conclusions and recommendations on maintenance problems and philosophy are included at the end of Section IV.

Some potential software problems that might be encountered in developing an Advanced Avionics System are described in Section V.

The causes of many of the problems are examined, and some generally accepted guidelines for avoiding them are discussed.

Appendix A contains a list of contacts made during the course of the study.

-8- U. COST ANALYSIS OF ADVANCED AVIONICS SYSTEMS INTRODUCTION The objectives of the cost analysis portion of this study were several. Primarily, methods were desired that could be used to esti- mate the costs* of the advanced systems. Originally, it was intended to develop estimating methods for both acquisition costs and mainte- nance costs. As the study progressed, it became evident that acquisi- tion costs could not be easily developed. Since maintenance costs are usually a function of acquisition costs, the explicit development of estimating relationships for them was dropped. The following discus- sion therefore only addresses acquisition costs.

The objective was to develop cost estimates that were compara- tive, in the sense that they could be used to compare the costs of one system with another system. They were also to be absolute and as accurate as the design of advanced systems would permit, so that the costs of any advanced system could be compared with the costs of existing avionics systems. As a further objective, the desired methods were to be parametric, so that conceptual avionics systems could be analyzed from broad descriptive inputs. Thus, the methods would express cost as a function of such things as output power, equipment weight, number of transceiver channels, etc. With reference to the system organization shown in Fig. 1-1, it was hoped that the generalized parametric relationships would represent the entire avion- *The word cost, as used in this discussion, refers to the cost of purchasing an avionics suite, or portions thereof. This "cost" to the purchaser is equivalent to the "price" of the equipment.

-9- ics suite; if this proved difficult, they would be developed to represent discrete portions of the suite. Last, the methodology was to be able to handle widely differing systems, ranging from those that are off-the-shelf to those using the most advanced sensors and elec- tronic techniques.

Past avionics cost studies, as will be seen later, have generally not been blessed with the same measures of success that accrued to cost studies of nonelectronic hardware. Because of this, and the very demanding nature of the study objectives, the cost analysis of advanced avionics systems posed special problems. It was hoped that some of the problems that had been incurred in previous avionics stu- dies resulted from the fact that these studies had dealt almost exclusively with military avionics, the special nature of which (e.g., military performance specifications and rapid technological evolution) narrowed the data base useful for generalized analysis. If this were the case, there was the possibility that the characteristics of gen- eral aviation avionics would be such that they would be more amenable to cost analysis.

The cost analysis task for the advanced avionics systems (AAS) posed some interesting problems. As stated above, the principal objective was to develop a generalized methodology by which the costs of candidate systems could be estimated and compared. However, a con- straint was imposed upon the methodology to the effect that the archi- tecture of the AAS and the technology to be used were undefined.

Therefore, the methodology had to accommodate systems ranging from the adaptation of existing off-the-shelf equipment (such as proposed by one subcontractor), to systems that were completely redesigned to use -10- integrated electronics and advanced sensors (such as proposed by another subcontractor), to systems that were open-ended regarding the technology used.

Costs can be estimated at several stages in the life of a system, including a stage prior to its actual design, but different kinds of data are required to develop the different kinds of relationships that are used. The development of cost estimating relationships (CERs) that would apply to avionics equipment of different technologies and different states of the art would require that the data base for the CERs be composed of these various technologies. Alternatively, separate CERs could be developed for each separate statistical popula- tion (i.e., each distinct technology and/or state of the art); The latter would, in this case, require a set of CERs for existing equip- ment, and sets of CERs based on future equipment that used new or presently unused technologies. This is possible, at least conceptu- ally, as long as the technologies or other major characteristics are well defined. To the extent that they are open-ended, the task cannot be done. Further, the development of CERs for future technologies may, of necessity, rely either upon smaller data bases or upon other techniques (e.g., engineering analogy) that cannot have the same degree of certainty as those used with sound data. This is simply a fact of life in the development of CERs, which tend to be good when a lot is known about the subject, and less satisfactory when less is known.

Since cost analysts occasionally face this problem, i.e., the question of comparing something about which a great deal is known with some future replacement about which fairly little is known, how do

-11-

they deal with it? In part, a situation such as this is not a com- fortable one. The ideal analysis is one that relies upon CERs that describe a continuum of equipment that includes all types to be com- pared. With such CERs, the analyst has a degree of confidence that the basic relationships between costs and the parameters contained in the CER are well quantified.

If CERs of this type are unavailable, other means of coping with the problem are needed. In the case of the AAS, some of the technol- ogy is not new, but is simply used in a new application, i.e., the application of LSI circuitry to an entire integrated avionics system.

For this case, estimates of cost may be made by using an approach that breaks the system down into components, estimates their costs, and sums them. The success of this method depends on the detail that is known about the system and on one's ability to understand the general relationships that exist between component part costs and the various other manufacturing and marketing steps that lead to a retail price.

Where the technology is new, and where there are no prior applica- tions, costs or prices cannot be estimated by any systematic method.

In this study, we attempted to develop CERs for the acquisition costs of existing equipment, so that AAS designs incorporating present equipment could be accommodated. We also examined the new technolo- gies expected to be used in other AAS systems and attempted to develop CERs for them. Where there were unknowns, due to unknown technolo- gies, we have tried to present some perspective on likely costs.

-12- SURVEY OF PAST STUDIES ON COSTING Any discussion of past cost studies on avionics equipment must be prefaced by the comment that virtually all such studies have con- sidered military avionics only. Because the cost of military avionics is usually substantially higher than the cost of nonmilitary avionics, such studies would be expected to be useful in a qualitative sense only. From these studies, it would be hoped to discern which equip- ment characteristics or descriptors relate to the cost of the equip- ment, and the general form of the relationship. Given that the rela- tionships are valid ones, it might then be expected that similar rela- tionships, with different cost coefficients, would exist for non- military avionics, and that the main task facing the cost analyst would be one of deriving the correct cost coefficients.

This hope of being able to "piggyback" on studies of military avionics costs is largely a vain one. Of the studies that were reviewed, few were found that could be directly related to general aviation avionics. In fact, remarkably little successful work has been done in the field of military avionics cost analysis, despite the fact that all of the cost data ought to be publicly available, and that there is an avid interest in the subject. Reasons for this will be discussed below, but the general implications of the lack of back- ground need to be understood in the context of this study.

The military studies that have been made largely involve data for existing avionic equipment, and were aimed at the creation of CERs for that equipment. Thus, the added dimension of attempting to anticipate new technologies did not exist to the same degree as it did for this study. Despite this possible easier task, few of the studies reviewed -13- were able to generate satisfactory CERs. The implications of this go beyond the inability to use military avionics studies as a jumping-off place—they say a great deal about the task itself and the prospects for successful results. This is especially true because relevant data for general aviation avionics is less extensive and detailed than for military electronics, and because the technologies that are to be expected for the AAS are liable to be new and different from those used by the military.

The major interest in developing military avionics CERs has come from the Department of Defense (DoD); but the individual services have also expressed an interest, usually through contractors. A descrip- tion of the studies that have been made follows. To give a more com- plete idea of the climate surrounding these studies, not only are the studies themselves described, but where no effort has been made, this is also discussed.

It will be seen that success has been rare in these endeavors.

One of the basic reasons is that electronic equipment is peculiar in that a desired function can be produced by a wide variety of elec- tronic means, each of which has its own circuitry and associated parts. Devices with similar, if not exact, performance capability, can have very different internal components, and therefore very dif- ferent manufacturing costs. They also can have different maintenance problems and costs. Because of the latitude that appears to exist in the design and construction of electronic equipment, one of the basic premises of developing useful CERs becomes tenuous. CERs are based on an assumed relationship between cost and physical and performance parameters. If this relationship does not exist, or cannot be -14- uncovered because the data include a wide variety of different basic designs, then attempts to develop CERs will fail. The failure will be due to the fact that costs are really related not to physical and per- formance parameters, which may be well documented, but to circuit, component, and construction parameters about which there may be little or nothing published.

Studies by the Department of Defense The interests of DoD have mainly been channeled through the Office of the Assistant Secretary of Defense, Planning and Evaluation (OASD/PA&E), and its predecessor organizations. One study, conducted by Resource Management Corporation (RMC), began in 1969, and a final report was issued in 1972. The study produced partially useful CERs for radar and fire control systems, but CERs for other equipment were less successful. For many pieces of equipment, the only significant independent variable was weight, which is not a very useful parameter for analysis, particularly of conceptual equipment. Basic problems in the study concerned data. At the study outset, it was decided that collecting the data from contractors involved an effort beyond the scope of the intended study. In lieu of this detailed information, budgetary and funding data were used which unfortunately contained an undifferentiated mixture of R&D and production funds. The poor results of parts of the study are partially due to the use of non- homogeneous data in developing CERs.

Rand became specifically involved in the collection of military avionics performance, physical characteristics, and cost data for OASD/PA&E in 1975, concentrating on the avionics suites of tactical -15- combat aircraft developed since 1965. Data were collected on indivi- dual pieces of equipment from manufacturers, the DoD data banks, and from the Electromagnetic Compatibility Analysis Center. It was hoped that these data could be aggregated in such a way that they could be cross-checked with budgetary data for the aircraft systems involved.

A great deal of data were collected, and some analyses were made. It was found that when an attempt was made to create homogeneous samples of equipment from which statistical analyses could be made, the sample size was reduced to the point where the analysis lost meaning. One of the study conclusions was that the addition of data from other types of military aircraft might expand the data base sufficiently for analysis to produce CERs.

Rand also performed an analysis of USAF aircraft avionics recov- erable spares, and developed a CER. The relationship is based on the number of aircraft and the avionics subsystem flyaway cost, and has an input that is a technology indicator.

General Research Corporation developed a complex parametric CER for ground based radars. The CER uses about 30 inputs, many of which require detailed knowledge of the device. This is not an avionics CER, but it is interesting from two points of view. It is electronic equipment, as are avionics, and its success is partly due to the large number of detailed inputs that are required. In this way, it approaches the concept of an engineering cost estimate, where detailed knowledge of the components is required.

Planning Research Corporation conducts military analyses that sometimes require avionics cost estimates. They generally make such estimates based on analogous systems. The extent to which estimates -16- of this type are successful depends on the type of analysis that is involved. Where the avionics costs themselves are not under analysis, they are probably satisfactory. However, if the object is to examine (for example) the cost effects of varying the parameters of the avion- ics themselves, these types of estimates are unsatisfactory.

Air Force and NASA Studies on Avionics CERs The Air Force Systems Command has been interested in avionics CERs in its Aeronautical System Division (ASD) and its Electronics System Division (ESD). Reliable, formal avionics CERs have not been developed in either division, however. Further, since most of their needs are for relatively near term equipment, they have been able to use the RCA PRICE model described more fully on p. 18 [18]. Typical applications of avionics costs include the life-cycle cost analysis of projects such as DAIS,* in which time phased acquisition and O&M costs are compared for competing pieces of hardware to determine minimum life cycle costs. DAIS uses a well developed life cycle cost metho- dology, but the equipment costs inputs to this methodology are sup- plied by the equipment contractors. These, in turn, develop their cost estimates by industrial engineering techniques using prototypes, or they use the PRICE model.

General Research Corporation (GRC) has also done work for the Air Force Avionics Laboratory on developing CERs for avionics. Their first effort resulted in suggested CERs for fire control radars, iner- tial navigators, computers, and doppler navigation radars [4].

^Digital Avionics Information System.

-17- The CER that was produced for computers in the GRC study attempted to take into account the rapid pace of technological change in this field by adjusting the data base not only to 1974 dollars, but to 1974 technology. The CER produced thus reflects only 1974 technol- ogy and is incapable of estimating the costs of future technologies without using a suitable technology change factor. With technology changing as rapidly as it is in the computer field, estimating this factor may be difficult. This point not withstanding, the CER itself is based largely on the weight of the computer, but it also includes inputs concerning the computational speed, memory, size, etc.

GRC, in a study for NASA Houston, published in 1975, examined 31 airborne and spaceborne computers, and developed two CERs, neither of which was developed from a data base applicable to the AAS. The first was for memory cost, and is stated as Ln(memory cost, in dollars per bit) = 8.119 - 0.I49(year-1900) The second concerned central processing units (CPUs) and related their costs to the date of development, word length, speed, number of bidirectional I/O channels, and whether or not the unit is a space application. Both of these CERs were developed from a data base of expensive units, the cost of which ranged from $32,000 to $683,000.

It was judged that this data base was out of the cost range of the equipment to be used in the AAS.

The Air Force is also an indirect sponsor of avionics cost- estimating activities through the companies that manufacture its air- craft. Lockheed is an example, especially with their series of antisubmarine warfare airplanes. However, CERs do not serve the kinds -18- of needs that Lockheed feels it has, and therefore it has neither developed nor used them.

Navy Studies on Avionics The Naval Air Development Center (NADC) has been engaged in the development of avionics CERs for some time. The Center mounted an extensive effort involving avionic computer systems in 1970 and 1971.

This effort resulted in a cost-by-function model that is based on a design-engineered approach. It works by using the input parameters to design and assemble a hypothetical computer to handle the functional load. Then, from a data bank, the model retrieves data on related existing equipment. The data that most closely approximate the system under evaluation are used as a baseline for cost extrapolation. This system depends on the data base, and the limitations in data have res- tricted the utility of the model. In concept, however, the model is similar to the RCA PRICE model.

NADC is continuing its avionics cost work at the present time, under joint Navy and NASA funding. Their objective is to determine top-level avionics costs for various categories of military aircraft, so that they can project future trends and assess probable avionics costs for different mission aircraft. They are presently almost fin- ished with data collection, and analysis will follow. The tools that they intend to use are stated to be: Experience and expertise Basic theoretical relationships Existing CERs -19- PRICE model Designer experience and projections At the present time there is little that can be used from this effort.

Other Defense Efforts The most notable effort has been by RCA Defense Avionics, where the PRICE model [18] was developed. PRICE is a proprietary model that requires the user to attend a special school prior to its use, and payment to RCA for its use. As mentioned above, the PRICE model is widely used for the estimation of military avionics costs.

Much of the model is based on data from RCA's own records and includes other "firms' equipment. (There are over 6000 items in the file.) There are 56 model inputs that are used to describe the equip- ment for which costs are to be estimated. The inputs are either com- plexity factors or physical characteristics. Some are mandatory, oth- ers can be calculated by the model. Some are calculated by the model regardless of whether or not they have been provided as an input, e.g., engineering performance schedule complexity factor, total com- ponent count, etc. This provides a check and balance to the accuracy of the system descriptors used as inputs.

Unfortunately for the novice user, over 75 percent of the inputs appear to require the user's judgment with regard to the complexity of such things as structure, electronic circuitry, and level of engineer- ing technological improvement. The model is apparently so general -20- that there is no requirement to specify the type or general class of equipment for which costs are desired. Little is known about the basis of PRICE, other than the statement by its developers that it is based on a dollar per pound relationship. Without being able to look at the data base used and the equations (standards) developed, it is impossible to determine the statistical validity of the model. How- ever, whether or not the model is statistically sound, results from an earlier NASA study, shown in Fig. II-l, indicate that in the hands of an experienced operator the model can provide estimates that are close to actual realized costs.

The subject of previous avionics studies can be summarized as follows: o Most studies of military avionics have not resulted in the development of broadly useful parametric CERs. None were found that could be used in this study.

o No previous CER studies of civilian avionics were located.

o The problems inherent in developing avionics CERS stem from the fact that a given function may be produced by a variety of electronic methods of widely varying cost.

This obviates the required relationship between functions and costs. The problem is generally manifested by the discovery that the published data on the equipment is often not related to its cost, and that conversely, the cost determinants of the equipment are often unpublished.

The conclusion of our search for existing tools to assist in the study is that they do not exist. What tools are required must be developed.

-21- Program PRICE Adjusted Net Actuals Equipment Computer Actuals Variance 13,096 12,076 ATCA 11,538 -4.5% 5,114 4,784 DECA 4,834 +1.1% 61,420 58,561 -4.6% RR Radar 55,966 20,244 19,300 -4.3% Transponder 18,553 35,776 33,776 Landing Radar 32,997 -2.3% Laser Altimeter 2,430 2,962 2,568 -5.4% 6,258 6,258 VHF Transceiver (LM) 6,570 +5.0% VHF Transceiver (CSM) 3,968 4,091 4,091 -3 . 0% VHF Transceiver (LM) 2,758 2,748 -4.1% Ranging Mod. 2,645 VHF Transceiver (CSM) 1,768 1,768 Ranging Mod. 1,728 -2.3% SOURCE: Frank Freiman, RCA Avionics Cost/Schedule Study, NAS 9-12356, 1972.

NOTE: Program Actuals Compared with PRICE Computer Runs—Flight Hardware ( $ 0 0 0 ) Fig. II-l—Example of estimation precision obtained by RCA PRICE model -22- THE DEVELOPMENT OF CERS Review of previous work revealed that significant problems existed in the development of avionics CERs. Also, a review of the two Preliminary Candidate Advanced Avionics Systems (PCAAS) developed for NASA by Southern Illinois University (SIU) and Systems Technology, Inc., (STI) [13,14] showed that two extremes were represented. One of the PCAAS was designed around existing components available off the shelf, and the other was a completely integrated system designed to function as a single unit. As explained in the Introduction, a cost- estimating method that treated the entire avionics suite as an entity was highly desirable. However, data on complete entities do not exist, and therefore a historical base from which to start is lacking.

Further, the exact problem that has been discussed concerning dif- ferent ways to achieve the same performance is presented by the two PCAAS systems. One resembles a grouping of familiar avionics com- ponents that have been hooked together to perform as a unit. The other resembles nothing that a general aviation pilot has heretofore seen, since it has been designed as a totally new unit. Yet both presumably serve the same function. A single cost-estimating method cannot accommodate both suites; they are simply too different.

Because of this, we approached the development of CERs in two ways.

First we attempted to develop CERs for off-the-shelf equipment, with the intention of using these CERs for those PCAAS that could be identified as being made up of existing components, in contrast to a completely integrated system. Then, for those PCAAS that were designed as integral units, we attempted to develop a different set of CERs. Both of these efforts will be described in detail below.

-23- With two different cost-estimating methods, consistency of esti- mates could be expected when like systems were being compared, i.e., when one system made up of off-the-shelf components was compared with a similar system, or when integrated design systems were compared with each other. This type of consistency disappears, though, when systems of different design, that each require a different cost method, are compared. The analyst must understand this, and must balance the problem of having no estimates to compare with the fact that different methods probably produce results that are only qualitatively compara- tive.

When one CER is used to estimate the costs of two or more sys- tems, the cost differences are consistent from one comparison to another because the CER translates the functional inputs into cost outputs according to the algorithms built into it. If one CER is used to estimate the costs of one piece of equipment and a different CER is used to estimate the costs of another, the cost differences observed may not be due to differences in functional inputs. This is usually caused by the fact that the CERs have different inputs, and the cost estimates are then based on different parameters. An example of this important point will illustrate it. Suppose that there are two kinds of commercially available transmitters, each using a different tech- nology. For one kind, a good CER might result from relating cost to power output. For the other, the best that can be done might be to relate cost to weight. Now suppose that we want to compare the costs of the two types based on their power output. Clearly if this is to be done, the analyst must be able to translate power output into weight for the second kind of transmitter. But if he could do that, -24- he probably could have developed a CER based on power in the first place. On the other hand, if all of the transmitters were of the first kind, the CER would be the embodiment of the change in cost as a function of a change in power. The use of single CERs is more mean- ingful in making consistent comparisons.

The above discussion presumes that the data bases for developing both kinds of CERs were equally good. Seldom is this the case.

Therefore the problems are compounded, even when the different CERs are more similar than those mentioned above. Poor data bases produce poor CERs; good data bases may produce good ones. Comparisons of costs that are estimated by using both good CERs and poor CERs will have associated low levels of confidence.

CERs--Existing Avionics Equipment The previous section mentioned that one PCAAS design already sub- mitted to NASA was based on off-the-shelf components. The use of a CER to estimate the prices of these components is clearly inappropri- ate, because the prices are known exactly. However, CERs based on existing equipment can serve an extremely useful purpose. They pro- vide the analyst with the tools necessary to perform a sensitivity analysis, and to answer questions of how costs may change as some of the performance features are altered. They allow the analyst to com- pare systems with different characteristics (such as power output) by equalizing them through the use of a CER to the same power output.

They also allow the conceptual design of a system that is composed of components that are similar to, but not exactly the same as, off-the- -25- shelf items. Based on this utility, we attempted to develop CERs based on commercially available avionics equipment.

We began the effort by obtaining data from the April issues of Business and Commercial Aviation, and from manufacturers' literature.

We used data for the years 1973, 1975, 1976, and 1977 (1974 data were not available in time). These data were segregated by class of equip- ment, i.e., transceivers, transponders, etc., and all listed attri^- butes were coded and entered into machine storage. Statistical ana- lyses of the data were then made by using a multiple regression analysis program to test preliminary hypotheses about cost/performance relationships.

Analyses were made of communications transceivers, transponders, navigation transceivers, and distance-measuring equipment. The results of these analyses were unsatisfactory in several respects, as follows: 1. No sensitivity to the model year of the equipment was indicated. Thus, cost trends could not be established, and the effects of changing designs, manufacturing techniques, and new technologies, which ordinarily would be expected to show up as a time trend, were unobserved. This might be implied to mean that manufacturers hold prices (in current dollars) relatively constant, and absorb the effects of infla- tion through economies resulting from the use of new technologies. Our interviews indicated that this process not only took place, but that greater -26- capability was continually being built into the equip- ment as well, as a result of changing technology that allowed this to be done without increasing prices.

2. The only explanatory variables that were consistently significant in the regression equations were power and weight, and these frequently explained only a small part of the data variance. There are some limited cases in which these two variables would be useful in a cost analysis, especially if there were other variables in the equation as well. But when one or the other of them is the only variable that has significance, the regression equation is almost useless. It does an analyst little good to know how the price of a navigation transceiver varies with its weight, when what he really wants to know is how its price varies with its performance.

3. Where equations were developed that included the limited variables discussed above, they frequently produced results that duplicated the data inputs with a margin that was judged to be too wide—plus or minus one-third was typical.

The poor results were attributed to the major problem of coupling price data to the listed physical and performance characteristics of the equipment. For communication transceivers, we had the following numerical data to work with: -27- Year model Number of channels Power output Number of boxes Weight Price Manufacturer But there were numerous other features that were available on some of the transceivers in the data sample. For example, included as listed variables were such features as: Remote mounted power amplifier Blue/white lighting Self test TSO categories Automatic squelch Speaker amplifier Storage and recall of additional frequencies Electronic frequency readout Portable battery pack option Dual frequency selector Self-contained cabin amplifier Automatic voice leveling Then there were other variations that were not listed, such as basic differences in circuit approaches, etc. Costs were judged to be more -28- a function of attributes of the equipment items that were not listed, and which are probably nonnumeric (circuit design is an example), than of the descriptors that are available. As a result, the statistical analysis of the data was terminated.

ANALYSIS OF NEW TECHNOLOGIES New technologies applicable to the AAS were identified in two ways. The more direct means of identification was through examination of the two specifications for the PCAAS that were submitted by the NASA subcontractors. The STI specification used more elements that can be described as "off-the-shelf" items than did the SIU specifica- tion. Even so, the individual pieces of equipment were tied together by a bus structure and fed into the central processor unit. Both the bus and the LSI circuitry of the CPU can be considered as relatively new technologies to general aviation, especially on the scale of the PCAAS. The SIU specification, on the other hand, uses far fewer "off-the-shelf" items. It incorporates the bus structure and relies heavily on LSI circuitry. In addition, it uses a flat plate area plasma display, and has provisions for advanced sensors.

The second means of identifying new technologies was through the manufacturers that were visited. In some cases, the visits were made specifically for the purpose of looking at new technologies. In oth- ers, the visits were for other purposes, but information about new technologies emerged during the discussions.

All technologies that were identified were considered applicable to the AAS. In general, the technologies that were considered were grouped as follows: -29- LSI and microprocessors Displays (mainly flat plate displays) Bus structure, with associated interfaces Sensors Software (this will be the subject of a separate later discussion) Information about LSI and microprocessor technology was taken from current literature, and from studies that have been performed for NASA on the subject. This information was supplemented by visits to pro- ducers and users of this type of equipment, as well as to avionics manufacturers. Intel Corporation, Hewlett-Packard, Delco, and Collins Radio are examples of organizations that provided helpful information.

Displays are the subject of great interest by avionics manufac- turers, and a variety of technologies are being pursued. Most of these however, are being pursued for use as alphanumeric displays.

Area displays are much less frequently encountered, and where they are, they are under investigation for military use. Most information on displays originated from Delco and Hughes, as well as from the literature.

Bus structures are analogous to LSI in many ways. They are in use at the present time, their parameters are well known, and they are subject to fairly dynamic forces that result from technological improvements, manufacturing changes, and market pressures. All of these indicate lower future costs. Hewlett Packard, who manufactures an IEEE 488 bus, provided most of the information.

-30- Sensors were found to be the most promising but least-well- defined area regarding technological change with associated cost changes. The situation was best described by one person who said that the manufacture of sensors was "moving into the 1C houses." The prob- lem is not simple though, because there is such a diversity of promis- ing technologies, and because knowledge of many of these technologies is so new and so small that it is difficult to say which are the most promising. In aggregate, all indications are that there are many changes on the horizon, and that these changes almost universally are harbingers of more accurate and lower-cost sensors that are free of mechanical maintenance problems.

LSI/Microprocessors The most common method of estimating costs in this fast-moving technology is by piece parts count. Attempts to use functional characteristics founder on the rate of change in the technology.

What required an entire printed circuit board yesterday, will require only one LSI chip tomorrow. While this concentration of processing power is occurring, the cost of circuit boards remains about constant.

It is for this reason that the piece parts count makes good sense, and gives good results. The method has other attractions as well. Piece parts are assembled into printed circuit (PC) boards of a given size and complexity. PC boards of like size and complexity are found to have similar piece parts costs, and therefore similar costs per board.

It is common for people "in the industry" to talk of PC board prices (which are a multiple of costs) of about $1000 per board, whether the boards are for one specific application or another. It is also common -31- for industry sources to say that the retail price of an item is equal to eight times the piece part cost. This multiple takes into account labor, overhead, G&A, profit, distribution and sales costs. The exact amount of each of these factors may vary from company to company, but the results appear to be surprisingly uniform in aggregate.

PC boards do vary in both size and complexity, and these vari- ables can be taken into account in the piece part count, while still using the rule of thumb that the finished PC board will sell for eight times the piece part cost (e.g., if the piece parts have a cost of $100, the PC board will sell for $800).

These relationships have been used in both the STI and STU specifications of the PCAAS. SIU derived a relationship of price = 10 times piece parts count, which is probably conservative, based on the industry criteria stated above, and used this relationship to estimate the cost of their PCAAS. Using a multiple of ten versus a multiple of eight can be viewed as a means of hedging on the accuracy of the piece parts count, which has some inherent uncertainties at the stage of design used by SIU. If their final design required 25 percent more piece parts, their cost estimate would probably still be valid.

STI used these relationships in a less rigorous manner, but still appropriately. Virtually all of their system was designed to use com- mercially available parts, including the PC boards for the central processing unit. However, there were a few PC boards that were not commercially available, and the price of these boards required estima- tion. This estimation was made by comparing the size, complexity, and piece parts count of these not commercially available boards with those that were commercially available, and then deciding that they -32- were sufficiently similar so that the prices would be similar. These PC boards, interestingly enough, are priced in the $200-$300 range, indicating that they are less complex than the boards generally used in the avionics industry. It was mentioned above that $1000 per PC board was a common number. The actual numbers obtained from each com- pany are as follows: Company A, MIL SPEC boards $1000 - $1500 Company B $ 800 - $1000 Company C $ 600 - $1000 The relationship between piece parts cost and retail price was given as follows: Company C - Retail price = 8 x piece parts cost Company D - Retail price = 8 x piece parts cost Company A - Government price = 4 x piece parts cost, and government price = 1/2 retail price The above relationships must be qualified in that they do not implicitly contain a measure of performance. It was mentioned earlier that the performance of LSI circuitry is increasing rapidly, and thus a task that requires two or more PC boards today may require one or less tomorrow, and the price of the board may be no different. This effect is automatically taken into account if the piece parts count is used; but if estimates are being made for the future, it is then necessary to estimate how the piece parts count will change in the intervening period, or how the performance of the PC boards will change. Projections of this type are difficult to make because the -33- present rate of change is very high, and the degree to which it will continue appears to be very uncertain. Figure II-2 was taken from a Honeywell study of computer technology that was performed for NASA.

This figure indicates these large rates of change, and could be inter- preted that there will be price changes of factors of two or more between now and 1980. But the figure is a straight-line projection on a logarithmic scale, and such extrapolations are often suspect.

Whether quantitatively correct or not, the indications are uncontested that whatever the cost estimate is for a given microcomputer today, it will cost less, and probably substantially less, in the 1980s.

Displays The only area displays that are presently in use in aircraft are CRT displays, which are used primarily for radar. Their costs are well established, and such displays can be considered an off-the-shelf item. Flat plate area displays are mostly in the research stage. The Honeywell report sums up the status of these displays by stating "none of the various flat-plate matrix displays under development are ready for application to general aviation." This opinion was shared by the equipment manufacturers. Delco, for example, is working with various types of alphanumeric flat plate display technologies, and much currently available general aviation avionic equipment uses alphanumeric displays. No one was found to be working with flat plate area displays for use in civilian aircraft, but Hughes Aircraft Com- pany is deeply committed to liquid crystal display (LCD) panels, which they perceive as having their first use in military aircraft.

-3.4- 0.010 O 8080 O MONOLITHIC Q LSIC, MSIC ASSEMBLIES © 8080 HDC-301 0 6800 O HDC-301A \\ © IMP 16 \ 0.001 \ \ \ IMS 9900 (990/4 CPU) PROJECTED P/P RATIO FOR \ QL A FAMILY OF HONEYWELL Q.

SINGLE CARD PROCESSORS \ (X \ \ \ 0.0001 HDP-5301 \ \ \ \ ^

\

COST ($) P/P RATIO = BITS X THROUGHPUT (KOPS) PROJECTED P/P RATIO SLOPE DUE TO PRICE REDUCTION OF SEMI- CONDUCTORS ONLY.

0.00001 -f- 1974 1975 1976 1977 1978 1979 1980 1981 1982 1983 1984 YEAR Fig. II-2—Projected price/performance ratio for a family of Honeywell processors compared with past and present devices -35- Some cost information is available for alphanumeric displays, which is presented in Fig. II-3. No cost information is available for flat plat area displays suitable for aircraft use. These displays are still under development, and are not expected to be used by military aircraft until the early 1980's. Subsequent to this use, they may be expected to be available for civilian avionics application.

Bus Structure As mentioned above, busses are in common use, and prices are known for both the cables and the interface circuits. These prices were stated by one manufacturer to be as follows: Cable lengths: 1 meter $60 2 meters $65 4 meters $70 Interface circuits: $150 to $1000 What is more interesting to speculate about is the future price of the interface circuits, since they consist of LSI, MSI, and hybrid circuits, which are subject to the same cost reduction forces that have been identified above. In this respect, the manufacturer hazarded a guess that the retail price of the least expensive inter- faces would reach as low as about $50 in the 1980's, and that their manufacturing cost would be well below $10. If the relationship described earlier of 8 x piece parts cost = retail price is applied, then a $50 retail price implies piece parts costs of about $6, and total manufacturing costs would probably be as they described—well below $10. Also, they are essentially projecting a retail price reduction from $150 to $50 over the next 5 years or so (in the -36- to O ce: CQ o (0 CO •H Q CQ •H Csl — * Of

^2 S

O o O -37- 1980's). This projection would appear to be consistent with experi- ence with new, solid state electronics circuits.

Sensors The prices of present sensors are well known, and if these sen- sors are used in a PCAAS, as was done by STI, there is no problem in estimating them. If, on the other hand, advanced sensors are to be used, there is presently no cost information available. But our interviews made it clear that sensor technology is on the threshold of dynamic changes that will probably reduce acquisition costs substan- tially, improve quality, and reduce maintenance costs.

Attitude sensors will probably undergo the most profound changes.

There are a number of different technologies that offer promise.

Prominent among them are ring lasers and resonant quartz "bells." In the latter, strap-down quartz bells are excited to their resonant fre- quency. When the attitude of the bell changes, it causes a displace- ment between the standing wave on the bell and the position of the bell. This displacement can be measured as a minute capacitance change, and reported as the appropriate change in attitude. No moving parts are used, and the entire sensor is electronic. Development of this device by Delco is in its infancy, but it was suggested that the device would reduce gyro costs substantially. Personnel at Collins Radio also concurred that by the 1980s, mechanical attitude sensors would be a thing of the past, and that whatever nonmechanical system (or systems) would be in use would be much less expensive than present systems.

-38- There are many pressure sensors on an aircraft, and there are presently two nonmechanical devices that are likely contenders. One is a resonant quartz crystal that pressure acts on to change the resonant frequency. A digital frequency counter completes the system.

Collins Radio feels that this device would work far better than mechanical devices, and that it has the potential of eventually cost- ing the same.

Honeywell has a piezoelectric pressure sensor that relies on a silicon diaphram with resistors, similar to a strain gage. There is a problem with this device, since it is temperature sensitive, and the temperature-induced output is substantially larger than the pressure- induced output. This problem has resulted in the use of sophisticated bridge circuits, plus software "calibration" for the individual unit.

However, Honeywell feels that this sensor has the potential of being very cheap.

J-Tech, of Cedar Rapids, has developed an airspeed indicator that is based on a pipe with a post in it. A vortex forms behind the post, and then moves away from it and dies out. The number of vorticies formed per unit time is a function of the airspeed. An electro-optic vortex counter completes the instrument.

The sensors used in the STI specification are off-the-shelf items, and their prices are known. SIU uses off-the-shelf attitude sensors (gyros) but specifies no particular kind of pressure sensors.

Instead they advocate voltage-controlled oscillator devices for which they have allowed $55 per sensor in their estimate. At the moment, there are no cost data on advanced sensors. The development of cost- estimating relationships for advanced sensors must wait until the technologies evolve further, and data become available. Until that -39- time, the only guide to sensor costs is industry's belief that they will probably be lower than the cost of today's equipment.

CONCLUSIONS AND RECOMMENDATIONS The investigation into the development of generalized cost- estimating methods for general aviation avionics found that: o Previous attempts to develop avionics CERs had only limited success.

o Attempts to develop a single methodology were precluded because of lack of data and the problem of dealing with as yet undefined future technologies.

o The development of different methodologies for the two PCAAS systems already proposed to NASA was unsuccessful in the context of the creation of parametric relation- ships that expressed costs as a function of physical and performance parameters. This failure was due to the lack of significant statistical relationships among the variables tested.

o A number of estimating methods, rules of thumb, and industry practices were collected which can be of assistance in analyzing equipment costs.

Because parametric relationships could not be developed, esti- mates of the acquisition and maintenance cost of advanced avionics suites must rely on techniques that are based on piece parts counts.

The use of these techniques will allow the cost data for one system to -40- be compared with the cost data of other systems. The ability to per- form sensitivity analysis will be limited, or nonexistent.

NASA cannot predict the types of designs that may be submitted, or what technologies might be incorporated into the designs. There- fore, NASA should require that each submission be accompanied by a contractor's prepared estimate of users' purchase price, maintenance philosophy, and maintenance cost estimates. These estimates should be backed up by a description of the cost-estimating methodology used, and a listing of all factors and analogous and corroborative data.

The description should be sufficiently complete to duplicate and evaluate the estimate. Estimates prepared in this way will most likely be engineering estimates that rely upon piece parts count, piece parts prices, and manufacturing labor cost estimates.

The inputs to these cost estimates can be reviewed for their rea- sonableness and completeness, or, in the case of technologies that are not yet commercialized, for evidence that the estimate made is defen- sible. Estimates made in this manner can be used to make comparisons among the various designs that are submitted, provided it is under- stood that the estimating methods used for different levels of design will probably have widely differing (and unknown) estimating accuracy.

Therefore, comparisons that are made should stress general cost ranges rather than exact differences among estimates; they should concen- trate, wherever possible, on uncertainties in the estimates. These uncertainties could be of critical importance where the technology is new, or is noncommercial, or where either the piece parts count or prices appear to be subject to doubt. For some systems, it may be possible for the contractor to submit the estimate in disaggregated -41- forra, so that cost comparisons may be made by functional subdivisions, i.e., transceivers, transponder, etc. For integrated designs, this disaggregation may be impractical, but at least the hardware and software may be separated.

System integration, particularly for systems that are made up of off-the-shelf components, is likely to be one of the driving costs, and NASA should request that this subject and the subject of software be addressed explicitly and in detail.

Using the above techniques, certain assessment of costs is possi- ble. However, the ability to perform some types of analysis will be constrained by the fact that these estimates will rely upon engineer- ing cost estimates. That will be particularly true in the case of sensitivity analysis, where functional relationships are more useful than parts counts. As an example, consider two designs, one of which is clearly superior to the other, but which has a higher estimated initial cost. Suppose that the superior design has a higher tran- sceiver output than the other design, and that the additional power is judged, unnecessary. A reasonable question to ask is how the cost of the better unit might change if the transceiver power were reduced.

With the right kind of parametric relationship, this exercise would be simple. Without such a relationship, the exercise cannot be done unless the unit is redesigned and a reestimate is made based on the new piece parts count and description. This example serves to illus- trate that each estimate that is considered represents one point on a curve that describes cost as a function of various parameters. Having the functional relationship gives the analyst an understanding of cause and effect, and the ability to test the cost sensitivity to -42- changes in the values of the parameters. When it is not possible to develop these functions, as for the MS, simpler comparisons are pos- sible, and often sufficient.

-43- III. RELIABILITY OF ADVANCED AVIONICS SYSTEMS This section will briefly describe the state of the art of relia- bility prediction and reliability assessment for avionics equipment.

It will be shown that there is currently no comprehensive and trustworthy methodology for achieving the desired predictive capabil- ity discussed in Section I of this report. Furthermore, the lack of any significant amount of data on reliability and maintainability of current avionics systems used in general aviation makes it impossible to develop new prediction methods responsive to the objectives of this study. It will be seen, however, that even within these recognized and substantial limitations, reliability prediction and assessment can be accomplished to a worthwhile degree, particularly in the important aspect of comparing either the projected or achieved reliability of two or more competing designs, and also in identifying those portions of any single design which have a disproportionate likelihood of being troublesome.

This section will address these several facets of reliability, including reliability prediction, design reliability, quality control, environmental factors, and maintenance (though most aspects of the last-mentioned subject are in Section IV). In order to accomplish as much as possible in these matters, we have chosen not to restrict our sources of information to general aviation per se, but to incorporate commercial and Air Force experience wherever it seems appropriate. In the case of the Air Force experience, some specific reasons for including this information will be given later.

-44- WHAT IS RELIABILITY?

Reliability has been defined as the probability of failure-free operation within the design performance limits for a given period of time under specified operating conditions. Reliability is usually expressed as a decimal fraction or as a percentage of uptime to total time (this latter quantity should more correctly be labeled availabil- ity). Conversely, unreliability is the probability of a failure dur- ing a given period of time. Many times it is useful to express the reliability of a device in terms of mean time between failures (MTBF), which is the average hours of operation between failures as measured on a sample group of units for an extended period of time. It should be emphasized that MTBF and reliability are not synonymous terms.

However, a precise mathematical relationship does exist between them.

For a fair variety of situations, including most involving electronic equipment, a reliability equation can be written which illustrates the most general parameter relationships: m R = e where R is the reliability, less than or equal to 1.0, t is the required time of operation, usually hours, and m is the mean time between failures.

The equation shows that only two routes to improved reliability are possible: reduce the operating time, or increase the MTBF. Since the former is usually not possible, the importance of the latter is made clear. This and other terms for quantifying reliability and the -45- relationships between them are discussed in detail later in this sec- tion.

Although reliability engineering as a discipline has been esta- blished for over 20 years, there are still many areas that are not well understood, and many problem areas lack formal approaches to solution. The major areas of reliability science are reliability prediction, design for reliability, reliability measurement, and reli- ability improvement. The most common misconception in this regard is that a given level of reliability is built into a piece of equipment when it leaves the factory, and that built-in reliability is what the user, on the average, can expect. However, this is not at all what happens. Reliability depends not only on design, but also on quality control during manufacture, parts screening prior to installation, failure mode analysis, quality of maintenance, user training, environ- ment, scheduled maintenance, etc.

SURVEY OF THE DEVELOPMENT OF RELIABILITY PREDICTION METHODS During the early days of reliability prediction efforts, it was universally recognized that the reliability of a device was severely affected by its complexity, giving rise to the admonition to "keep it simple." Thus, the early substantive efforts concentrated on a meas- urement of the complexity of a device, for which the active element group (AEG) became the usable proxy in electronics. This type of prediction is exemplified in the so-called Bird diagram,* which gives a failure rate per operating hour as a function of the analog func- *George T. Bird, ARINC Research Corporation.

-46- tional complexity [Fig. III-l]. In addition, different relationships are associated with different environmental operating conditions, the least severe being storage, whether in depot or on ship, and the most stressful being the missile flight environment. For the less stress- ful of these environments, the failure rate was believed to be linear with complexity (slope = 45 degrees). For the more stressful environ- ments , the failure rate depended on the complexity to a power greater than 1 (slope > 45 degrees).

The Bird diagram was satisfactory during the days when electron- ics equipment consisted of vacuum tube circuits. Since nearly all vacuum tubes had an electrically heated cathode and other conducting elements, their failure characteristics in circuits were reasonably alike. There was a known degradation phenomenon associated with the heated cathode emitter, which was certain to make itself felt at some stage in the life of the device. When transistors came into wide use in electronics, the usefulness of the Bird diagram declined, since transistors did not have the same mechanism of degradation associated with the heated filament in a vacuum tube. Furthermore, the Bird diagram implied that no significant improvement in reliability could be made through time, without a decrease in the complexity of the dev- ice. During the 1960s, it was clear that improvements in the relia- bility of certain high-quality equipment were taking it out of the realm of the Bird diagram prediction.

At this stage, it became necessary to account not only for the vacuum tube, which had been the dominant contributor to unreliability of the active element group in the tube-type equipment, but rather to account for all the piece parts in a given electronic circuit. Moving -47- o D) & M 0) o u.

0> "o Q£ 0> U_ 10 100 1,000 10,000 Equivalent Analog Functional Complexity, N Active Elements Fig. Ill-1—Bird diagram -48- in this direction, the most capable researchers in this field developed failure rates for individual piece parts; these rates were to be tabulated during the design phase of a device and then added up to provide an estimate of the failure rate of the assembly. One focus of this activity was at the Air Force's Rome Air Development Center, which developed the first handbook for estimating electronic reliabil- ity. This publication, MIL Handbook 217, was published in the early 1960s. It tabulated hundreds, if not thousands, of failure rates for individual piece parts, and had a few reliability failure models for different types of equipment.

The first version of the handbook was followed by two others in 1965 and in 1974. The latter, MIL Handbook 217B [77], contained much larger tabulations of part failure rates, including derating parame- ters, environmental stress parameters, and a larger number of relia- bility failure models. Included in the tabulation were the first results for modern electronic devices such as monolithic circuits, and bipolar digital devices, bipolar and MOS linear devices.

MIL Handbook 217B has had two updates. The first added microwave transistors, high-power tubes, lasers, and cooling blowers and fans to the original tabulations. The second update, originally scheduled to be completed late in 1977, will add opto-electronic semiconductors, including light emitting diodes (LEDs), displays of LEDs, and optical isolators. It will also include further information on high-power microwave tubes and some new information on failure models for new hybrid integrated circuits.

It should be noted that the second update involves devices which are already in wide use, both in military and commercial systems.

-49- This means that MIL Handbook 217B can be considered to be at least 1 and probably 2 years behind the technology of these devices and their wide application. Inevitably, any reliability prediction methodology based on failure rates of individual piece parts must lag the technol- ogy it covers, at least until considerable experience has been gained with the type of circuitry in question. Therefore, one shortcoming of any reliability prediction methodology must be expected to be the lag behind current technology.

Mil Handbook 217B contrasts in another way with the Bird diagram.

The earlier method was directly usable in the conceptual phase of sys- tem design, since so little detail was required to apply it. The more recent method requires detail design knowledge in order to account for all piece parts. Thus, one useful facet of a prediction method was lost in the transition.

The most recent significant development in the field of elec- tronic reliability prediction concerns computer-aided evaluations of reliability. Some years ago, Kenneth Blemel, formerly of R/M Systems in Albuquerque, New Mexico, developed a computer reliability predic- tion technique for the U.S. Air Force. This method is currently being used by the Air Force's Logistics Command to estimate spare parts demand during electronic system lifetimes. The method is primarily aimed at logistics considerations, and has limited usefulness for direct reliability predictions [62].

Another computer prediction model was built by Gaertner Associ- ates for the U.S. Army [81]. Although not directly usable for elec- tronics reliability prediction outside of military specification parts, this method was adopted by Rome Air Development Center as the -50- foundation for a prediction model to be used by Air Force electronics development sources. Once again, the method uses military standard parts nomenclature in evaluating likely reliabilities.

The most recent effort in this field, which has a much better chance of being used in the commercial environment, occurred at the Naval Weapons Center, Ridgecrest, California, in collaboration with Systems Consultants Incorporated, also of Ridgecrest. This Navy- sponsored development uses commercial descriptions of piece parts for input parameters to an evaluation model. The model, called 217B Predict, uses roughly the methodology of the 217B handbook, without requiring the MIL standard designation of parts [75]. In addition, the characteristics of standard commercial parts can be input by the user so that a complete reliability prediction can be made without requiring a relationship to published 217B reliability figures.

The objective of these computer evaluation models has clearly been to reduce the amount of effort and time required to- evaluate the likely reliability of a given electronic design. It is probable, how- ever, that this effort will have some additional benefits, involving the elimination of errors of omission or commission which are very frequent in this type of evaluation. The computer is unlikely to overlook individual elements in a circuit, or to fail to require the input of certain environmental parameters. Computer evaluation should improve the inclusiveness of reliability predictions, if not other aspects of their quality.

-51- THE ACCURACY OF RELIABILITY PREDICTIONS A key element in an understanding of the state of art of relia- bility prediction methods today is the fact that such methods yield very poor prediction precision. A review of how good such predictions have been over the past 25 years will set the stage for the subsequent discussion of current analysis methods. This review will, of neces- sity, draw almost entirely on military experience.

About halfway through that 25 year interval, researchers from Rome Air Development Center reported on the quality of reliability predictions made during the first 10 years of their intensive activity. Table III-l shows a comparison of gross field MTBF measure- ments with predictions for several ground electronic equipments [82].

Table III-l GROSS FIELD MTBF MEASUREMENTS (NORMALIZED TO PREDICTED VALUE) 90% Confidence Predicted Field Interval MTBF Point Lower Upper (Mrs) Equipment Estimate Limit Limit 320 Computer 1.09 .99 1.21 850 I/O Controller 2.31 1.64 3.35 490 Memory Element 1.35 1.21 1.51 580 Message Processor 2.11 1.61 2.80 835 Drum Controller 1.12 1.60 1.33 Radar (1) 1.27 1.12 1.45 Radar (2) 1.80 1.54 2.15 513 Data Display Console 0.097 0.094 0.099 4131 Status Display Console 0.25 0.23 0.27 179 425-L Display Console 0.88 0.79 0.99 65.2 473-L Display Console 1.03 (Not Available) The agreement between prediction and field results is relatively good in this comparison. However, one would have to read the original report carefully to note that the equipment being tested "in the -52- field" was in air conditioned laboratory environments. Furthermore, numerous corrections to observed data had been made to eliminate cer- tain failures which were deemed not appropriate to count against the inherent equipment reliability. The table shows that experience data reflects, with reasonable faithfulness, the predictions made for the equipment in question. Unfortunately, this tabulation of achieved reliability relative to predicted reliability is not at all typical of most prediction experience, particularly for airborne equipment.

Indeed, a report by RADC researchers 1 year later showed that achieved reliability of a larger sample of equipments ranged downward to one- fifth of predictions [63].

Needless to say, some care is essential to the evaluation of such data. One can infer directly from the Bird diagram that from a relia- bility point of view the laboratory is most benign, with other ground-based environments somewhat more hazardous, and airborne vehi- cles still more stressful. Thus, evaluation in a laboratory, while probably necessary at one stage of development, must not be inter- preted as representative of end use. By the same token, testing and maintenance by the manufacturer's skilled technicians may not ade- quately represent treatment by owners or fixed base operator (FBO) maintenance mechanics. What counts is reliability in the hands of the user, warts and all.

In line with that notion, a more representative tabulation of achieved versus predicted reliabilities was the subject of a response from the Comptroller General of the United States to an inquiry from the United States Senate a few years ago.* The following table shows a *The letter, from Elmer B. Staats to Senator Gravel, was printed in the Congressional Record for December 9, 1974, pp. S20775-S20776.

This letter has been widely used and misused, as in the Reactor Safety Study, Wash-1400 (NUREG-75/014), Nuclear Regulatory Commission, Appen- dix XI, pp. 3-15 to 3-21.

-53- comparison of aircraft radar subsystem reliabilities achieved in operation compared with their specified MTBFs which were the result of reliability predictions. Typically, the achieved reliability was one-fourth to one-third of that specified. In one case, the achieved reliability was less than 1 percent of the specified MTBF.

Specified Achieved Aircraft MTBF* MTBF* F-4B 10 4 A-6A 75 8 F-4C 10 9 F-lll A/E 140 35 F-4D 10 10 A- 7 A/B 90 30 A- 7 D/E 250 12 F-4E 18 10 F-111D 193 less than 1 F-4J 20 ^'Approximate figures.

One paragraph from the Comptroller's letter is particularly worth noting: NASA experts believed that "absolute" reliability numbers are misleading and that the time required to develop them is better spent on critical-component reliability analyses. [NASA] does make predictions during development to compare design alternatives and to evaluate components.

A somewhat more recent examination of the quality of the relia- bility estimates was done by Hughes Aircraft Company for the Rome Air Development Center. That report, titled "Operational Influences on Reliability," found, for example, that the field reliability of a par- ticular piece of avionic equipment depends heavily on the type of air- craft in which it is used [64]. A more pertinent observation for our purposes is that field reliability is significantly less than -54- contractor-predicted values. In this connection, the contractor- predicted reliability of the equipment surveyed, when compared with the reassessed relevant failure rate, showed a ratio of nearly three to one. The researchers concluded that contractor estimates exhibited "some degree of optimism." Also noted was the fact that equipment used on long-duration flights showed significantly higher reliability, in terms of MTBF, than similar equipment used on short flights, a phenomenon which we will examine again somewhat later.

Perhaps the most significant finding of the Hughes study is that field reliability ranges from one-third to one-eighth of the predicted reliability for the equipment studied, depending mainly on how one defines a failure. The Hughes study found that of the failures which were deemed to be "relevant," 45 percent were due to operational fac- tors and 55 percent to environmental and other factors. These, in turn, were outweighed by deficiencies that were attributed to defini- tional factors, which constituted about four times as many "failures" as those remaining when the so-called non-relevant ones had been removed.

Attempts have been made to develop methods for estimating relia- bility in the preliminary design phase, when good information is not yet available concerning exactly how many piece parts will actually be used and other features of the design that involve such things as derating, redundancy, etc. A study of such a method was done by Hughes Aircraft Company in 1974 for Rome Air Development Center.* "Hughes Aircraft Company, "Study of Reliability Prediction Tech- niques for Conceptual Phases of Development," DDC No. A001919.

-55- Hughes proposed models for both parts count and failure rate or relia- bility according to a regression study of the data from a number of avionics systems. The reader should note the strong similarity to CER procedures and results. For the failure rate of communications avion- ics equipment, Hughes proposed the following equation: Ln(F) = 879.816 - .4436DY + .0029476TL + .04224RL + .1364PP - .0004324RBP(RBW) - .0002051TL(PP) where F is the failure rate per million hours, DY is the design year, TL is the transmitter level in watts, RL is the receiver level in dB below 1 mW, PP is the prime power in kilowatts, RBP is the receiver band pass and RBW is the receiver bandwidth. The fallacy of this equation is that it implies an annual growth of 56 percent in the reliability of the device, something which we know from experience simply does not happen. A similar examination was made for the equa- tion proposed for parts count, and this was also found to have a tremendous annual improvement factor, which is not reflected in current technology of avionics. The similarity between this modeling failure and CER shortcomings described in a previous section are real and significant.

Where do reliability predictions go wrong? Deterioration from otherwise justifiable predictions of reliability of electronic equip- ment can occur in two general areas: the first area is in the factory environment where the manufacturer of the equipment has control of it; the so-called, in-house region. The second place where reliability -56- deterioration can occur is in the field or in customer use. A brief discussion of this type of reliability deterioration will be given in a later subsection.

The manufacturer's responsibility is further broken down into » design, manufacturing or vendor control, and workmanship topics. The effect of design on reliability has been studied intensively. Each avionics manufacturer typically has his own set of rules which his designers must follow in designing avionics equipment, such as rules for derating, parts utilization, design review, etc. Great care is also required in the manufacturing environment, as for example, in the control of vendor parts which are assembled into an equipment. Mass screening techniques have been developed along with burn-in pro- cedures, testing procedures, etc., all aimed at catching existing or incipient defects before a piece part is assembled into an end item.

However, another area demands attention: the workmanship of assemblers in the factory. Once again, each manufacturer has his own set of rules to be followed by the assembly personnel in the factory: rules for handling, soldering, testing, etc.

One factor degrading the inherent reliability of a device is deficiencies in the design of that device. Permissible tolerances, derating of components, and redundancy in signal paths all can have an effect on the end reliability of the device.

The importance of design inadequacies in failure to meet relia- bility expectations is usually underestimated and sometimes overlooked altogether. An intuitively appealing explanation is that a designer will find it impossible to believe that his creation will behave in a manner other than the one he had intended.* In spite of these high *See Squires, Frank H., "Safety Engineering and the Murphy Ef- fect," Quality, April 1978, p. 57.

-57- expectations, many failures are traced to design defects. A fairly recent example involves about seven million passenger cars recalled during 1973 for potential safety defects. In more than 200 different recall programs, the largest single cause was improper design, not random failure. When all inherent causes were summed, design inade- quacies of various types accounted for nearly half of all the recalls [65,66]. This phenomenon is by no means limited to low technology applications. Many years ago, it was found that a large fraction of the test failures (actually, malfunctions) in the Atlas missile pro- gram did not involve real failures of any kind, but were inherent in the design of the missile or its support equipment [67].

During the process of manufacture, other factors enter the equa- tion. TSO* specifications are not a guarantee of end reliability.

All they require is that certain specified procedures be followed dur- ing the design and manufacturing process. The component quality level also has a demonstrable effect on end reliability. Commercial part reliabilities are not as good as the reliabilities of "controlled" parts, which are seldom used in general aviation avionics. The last step in the factory process involves inspection of the product before it is shipped, a topic we will treat in some detail.

One of the more important contributors to an increase of relia- bility of given equipment over the years is the manufacturer's inspec- tion procedures within his own plant. As an example of how such pro- grams work, consider the quality control program for preshipment pur- *TSO (Technical Standards Order). These requirements are aimed primarily at performance and accuracy objectives.

-58- poses conducted by one avionics manufacturer. The old production test program, applied to all units before shipment, consists of 48 hours of operation at 70 degrees Centigrade, following which a test is per- formed on the unit at ambient temperature. Under these circumstances, 30 percent of the units fail, and are recycled for repair. In order to improve the quality of goods shipped, the manufacturer is testing a variety of modified final inspections. The first of them substitutes a test at 70 degrees Centigrade for the current test at ambient tem- perature following the 48-hour run. Under these circumstances, it is found that 45 percent of the units instead of 30 percent will fail this test and therefore get recycled. A second candidate of this nature is to test at -46 degrees Centigrade. Results are not yet available for this modification.

An even more dramatic change in final inspection procedures involves a sampling program in which some units to be shipped are operated for 248 hours at ambient temperature in a pre-burn-in environment. This operation is then followed by a burn-in process involving both thermal cycling and vibration for a period of 248 addi- tional hours. In this type of operation, it has been found that 30 percent of all the failures experienced in the 248 hours of cycling are experienced in the first cycle of the process. This means that the first thermal cycle has a better ability to ferret out failures in the units than do subsequent cycles."' *For some additional examples of this phenomenon, see William W.

Provett, Jr., and Richard S. Ullroan, "Effective Reliability Planning and Implementation," Proceedings, 1976 Annual Reliability and Maintai- nability Symposium, IEEE Cat. No. 76 CHO-1044-7RQC.

-59- The units tested by all four of these procedures are being fol- lowed in the field for a period of 1 year to determine which of the four methods is the most cost effective in achieving a desired improvement in field reliability of these units.

A number of factors are now believed to contribute to the typical overestimation of reliability which characterizes reliability predic- tion. One of the strongest contributors is the pressure to win a con- tract. Obviously, a high reliability estimate tends to indicate that a particular piece of equipment may be superior to other candidates for a particular application.* Another contributor is known to be non-electronic portions of otherwise electronic systems. The relia- bility of mechanical systems is usually significantly lower than that of modern electronics, and considerable efforts are underway to sub- stitute electronic for mechanical components of avionics systems.

Another factor is believed to be the fact that state-of-the-art tech- o nology, that is, technology which is at the forefront of research, is typically used to gain a sales advantage, even in the commercial avionics market. Still another factor is the uncontrollable environ- ment which may be encountered in aircraft. Operating temperatures may not meet the standard of 40 degrees Centigrade which is typical of the specified operating environment for a fair number of avionics systems.

Of somewhat lesser importance is the fact that a particular electron- '•'•"The previously referenced Comptroller's letter to Senator Gravel quotes an unidentified Air Force Source thus: "...where a manufacturer is interested in having his equipment look good he can, and will, select some of the more optimistic data he can find or generate, to use in his reliability predictions. Thus reliability predictions, for several reasons, tend to be generally optimistic by a factor of two to six, but sometimes for substantially greater factors."

-60- ics manufacturer may have no prior experience in building the type of equipment which is to be designed for a particular application.

Finally, a number of manufacturers apparently enter the design and manufacturing stages with no reliability plan, a procedure which is virtually guaranteed to cause consternation later [68].

Given that the first demonstration of the reliability of an elec- tronic device is likely to be disappointing, what can be done about it? The two favored approaches are to make appropriate design changes to enhance reliability, and to eliminate early or incipient failures due either to design or manufacturing-caused weaknesses. While other approaches are also used, these two procedures seem to show the most progress in the improvement of reliability through the early life of a system. A General Electric study of several years ago showed an interesting relationship between the achieved level of reliability and the effort expended in the improvement and design of manufacturing techniques for a fair variety of both electronic and mechanical dev- ices : a Cumulative MTBF = kh , where h is the total operating hour experience, k is indicative of the initial (first hour) MTBF, and a is a growth descriptive parameter less than 1.

Examination of a number of studies which used this so-called Duane model sheds some light on how quickly reliability can be expected to grow in a development program. The following table shows the exponents of the Duane growth model as a function of the reliabil- ity effort for three studies, the first by General Electric [69] (the -61- original Duane study) in 1970, Hughes Aircraft [70] (for RADC) in 1975, and studies by the B-l system program office (SPO) in 1976.

Table III-2 GROWTH RATE DATA Source Reliability Effort Low Medium High General Electric (1970) 0.1 -- 0.6 Hughes Aircraft (1975) 0.3 0.37 0.45 B-l SPO (1976-1977) 0.1 0.3 0.5 Each of these sources shows that the slope (i.e., the exponent a) of the reliability growth curve can be expected to be low when reliabil- ity effort expended is also low, and high when reliability effort is intensive. Typical values for low reliability growth are an exponent of .1, while for a high growth the exponent is between .45 and .6.

In-house measures, which are applicable to both original design and reliability improvement efforts, are known to have an impact on reliability. For example, RCA Avionics has 20 rules to be observed in the design and manufacture of their avionics products. One category of these consists of design rules, such as a requirement for minimum derating in the use of particular types of parts. Another group is concerned with design practices, such as the use of metal film instead of carbon resistors in the design. A third category is concerned with design review; that is, consultation with engineers who are not involved with the particular design to take advantage of their -62- knowledge and experience. Finally, design proof is used in the form of a testing program to verify that certain design features are indeed as good as they are hoped to be.

An example of a particular avionics system developed for the U.S.

Air Force (a horizontal situation display, involving complex elec- tronic and mechanical functions) is illustrative of a number of the problems and procedures associated with reliability prediction and reliability improvement efforts. The candidate system for which ini- tial requirements were specified in 1966 had an original MTBF require- ment of 580 flying hours. In 1970, this requirement was revised down- ward by nearly 50 percent, to 300 flying hours, when the first bread- board systems were available. Needless to say, this was a complicated system, requiring considerable design effort and ingenuity. The first prototypes, tested during 1969 and 1970, actually exceeded the required 300 hours MTBF, giving an average of 360 hours MTBF. How- ever, when production items became available in 1971, the earliest test results showed that only 7 hours MTBF was being achieved. A large product improvement effort was begun; it was projected to achieve an average of 50 hours MTBF during its accomplishment and to result in 125 hours MTBF at its conclusion. While the final results for this system are not available because of security restrictions, it could be shown, through the use of the Duane model, that the reliabil- ity improvement programmed into the effort was of a very high order, requiring considerable effort and expense.

One of the less ambiguous things about avionics reliability is that it is growing. Figure III-2 is an example of the type of evi- dence available for commercial aviation equipment. There also seems -63- 1966 1967 1968 1963 1964 1965 Individual system hours I I 4000 8000 12000 20000 Fig. 111-2 — Reliability trend —727 flight control system -64- to be an understanding among the reliability oriented professionals that when an acceptable level has been reached or surpassed, effort will be turned in the direction of gaining increased performance, even at the cost of a portion of the achieved reliability.

A seemingly opposite extreme is exemplified by the Air Force, which has some reliability growth, but mostly at a much lower level of 1 to 2 percent per year. To be sure, the Air Force occasionally makes significant breakthroughs in reliability improvement. The reliability of a redesigned TACAN unit for a new Air Force aircraft increased by a factor of approximately 10 to 1 over the old unit, even while costs were decreasing significantly. However, this doesn't always happen.

The reliability of a redesigned UHF receiver-transmitter procured by the Air Force in fairly recent years actually decreased while the pro- ject cost increased, although the redesign involved only a translation from vacuum tube technology to solid-state technology. Part of the Air Force's problem (not shared significantly by general aviation) is that their equipment is becoming progressively more complex, heavy, and expensive. A Grumman study showed that MTBF per thousand parts in Air Force avionics equipment actually increased by 10 to 15 percent per year from 1960 to 1970 (remarkably similar to the commercial equipment), but that this improvement was almost completely offset by equally dramatic increases in complexity (and thus cost) during the same time period [71], resulting in the apparent low growth cited.

For a variety of reasons, we believe that Air Force reliability data are relevant to the general aviation avionics reliability pic- ture, and not just from a reliability growth point of view. For one thing, the Air Force operates in a less than optimal environment as -65- far as temperatures, weather, etc., are concerned. The Air Force also has low usage hours, typically 1 hour a day or less for most of the aircraft in the inventory. This is far closer to the general aviation situation than is the commercial airline experience of 10 or 12 hours a day. The Air Force, as just mentioned, experiences slow reliability progress, probably more akin to general aviation than commercial experience would be. The Air Force also has a considerable variabil- ity in the available maintenance skills and practices, due to a rapid turnover in the work force. Finally, the Air Force is deeply involved in work with digital multiplexed (integrated) avionics systems like DAIS, the F-15, the F-16, the former B-l, and the E-3A. These activi- ties must eventually generate experience useful in general aviation integrated systems.

The general aviation community seems to fall between these two apparent extremes of rapid growth in commercial aviation and slow growth in the Air Force. The relatively few references we have been able to find indicate that the reliability of general aviation avion- ics is still growing, but at a somewhat slower pace than for commer- cial aviation avionics. In the case of general aviation, the growth seems to be between 5 and 10 percent per year for the same or similar equipments [72,73,55,56], compared with the apparent 25 percent or more exemplified in Fig. III-2.

One possible significance of this observation is that the consu- mer is becoming progressively better informed regarding his own self- interest in the achieved reliability of specific devices, and he is willing to forego increased reliability only if additional valued functions are substituted for the improved reliability which would -66- otherwise be expected. The practical impact of this observation is that the consumer will seldom accept a system, however impressive its performance, if he believes it to be unsupportable from a reliability and maintainability point of view.

RELIABILITY ANALYSIS .

This discussion is concerned with what we know about current avionics reliability. Reliability is an elusive characteristic in avionics. It's hard to quantify, for a variety of reasons. The peo- ple whom we would expect to be the best informed, the manufacturers of avionics, tend in the main to have poor feedback from the field. The big users of avionics seem to be the best source of information con- cerning its reliability, and the character of those users seriously affects such data. Another problem concerns the fact that the usual criterion for reliability, the mean time between failure, is an inap- propriate measure for many aviation purposes.

Any assessment of reliability must logically proceed from experi- ence data. Compared with the amount of such data needed to develop a predictive model by conventional methods, data applicable to the owner-pilot portion of general aviation is essentially nonexistent.

To be sure, attempts have been made to develop experience data, most notably by AOPA and AEA, but all such efforts have fallen short of even the most minimal requirements for reliability projection. One of very few successful efforts on our part to get such information came in a decidedly unconventional form, which does seem to have some prom- ise, and will be discussed immediately before the Conclusions and Recommendations subsection.

-67- Most people, when asked where the best reliability information might be in avionics, will suggest the manufacturer of the equipment.

But warranty failure data are notoriously unreliable. The manufac- turer has relatively poor feedback from field users. For example, a number of unfranchised (usually untried, one-of-a-kind) installations of this equipment are made in the field, and their results are seldom reported to the manufacturer. Another source of confusion is the fact that original equipment manufacturers (OEM's) of aircraft will fre- quently exchange avionics units at the factory when they are being installed and will return them to the manufacturer as field failures instead of factory failures or vice versa. The manufacturer will not usually know how long one of his units sat on the shelf until it was either sold by a fixed base operator (FBO) or installed by an OEM. In any warranty summary, heavy users of equipment will be over- represented, and this is important because their maintenance tends to be better than that available to the individual aircraft owner.

Finally, a major contributor to the poor quality of warranty informa- tion is the fact that the number of hours that the equipment is operated is unknown. The manufacturer customarily assumes that the equipment is used 200 hours per year in general aviation. All of these factors conspire to make warranty data very poor for the purpose of determining inherent avionics reliability.

Even if one could get good information about field failures and field operating hours, the extraction of mean time between failures from these data might still be inappropriate for some purposes. For example, an emergency locator transmitter (ELT), similar to the one proposed in the STI avionics suite, shows a purported MTBF of 4500 -68- hours according to warranty data.* Meanwhile, it is commonly believed that the reliability of these emergency locator transmitters approxi- mates only 0.5, a figure based on accident statistics. The low relia- bility in service is caused not by a short MTBF, but rather by inherent design inadequacies: the fact that a crash is either not serious enough to set off the g sensor, or is so severe that the unit is destroyed, and therefore cannot transmit. Neither of these phenomena come into play in the calculation of MTBF from warranty data, nor from any other source.

The mean time between failure may also be an inappropriate meas- ure when considering flight safety. It has been observed that most failures occur (or at least are discovered) at the turn-on of an elec- tronic device. Once operating, an electronic device is much more likely to work during an entire flight than an MTBF specification would indicate. This means that an MTBF correctly translated to the reliability of any particular length of flight will give a much too pessimistic prediction of the likelihood of failure, particularly for items with a relatively low MTBF.

An even more fundamental problem with reliability assessment of avionics is that the inherent reliability is seldom achieved, cus- tomarily being degraded by a variety of factors.

The end use of an item is probably the biggest contributor to variability in achieved reliability. The environment in which the device is used has a substantial effect on the achieved reliability.

*The most common failure mode is a weak battery. Since the dev- ice is normally push-button tested prior to flight, this mode of failure is probably not relevant to reliability in use.

-69- It is well known that higher operating temperatures result in lower reliability. Another factor contributing to the observed reliability is the amount of use. Heavy users, as mentioned earlier, get more hours between failures from comparable equipment than those who use it a few hours a month. The length of a flight has an unexpected effect on reliability. Longer flights result in higher observed MTBF, other things being equal. This is verification of the phenomenon usually called cycling stress, or turn-on stress, which has been demonstrated in electronics equipment for more than 25 years. Quality of mainte- nance is also a contributor to or a detractor from reliability. A demonstration of this will be given later. Finally, other factors are known to have an effect on observed reliability in the field, and some of these will be brought out in the later discussion.

All avionics manufacturers seem to agree that the quality of maintenance in the field environment has a significant effect on the observed reliability of their equipment. As an example of this phenomenon, consider the information in the following table showing removals and failures, with resultant MTBF calculations, for a VKF receiver-transmitter in a cargo airline operation. This receiver- transmitter, which is widely used in commercial and even in general t aviation, had shown a fairly steady reliability in this application over a period somewhat longer than 1 year. During October, November, and December of 1976, however, a significant decrease in the observed number of verified failures was noticed. At the same time, the number of removals seemed normal. Apparent MTBF jumped by a factor of at least two. This was followed by a period of 3 months in the beginning of 1977 when removals approximately doubled and failures were verified -70- ih nearly all of those removal cases. MTBF dropped to half its normal level.

REMOVALS FAILURES MTBF PRIOR 14 MONTHS 188 149 1250 JULY 1976 11 11 AUG 9 5 1591 SEP 21 13 OCT 14 3 NOV 16 7 3142 DEC 21 8 JAN 1977 40 34 FEB 22 22 742 MAR 16 9 What had happened, and was vaguely known by the people involved, was that there was heavy pressure during October, November, and December of 1976 to get aircraft back into the air when failures had occurred. This resulted in poor maintenance of these communications transceivers, with the result that they later failed again. But even more information is available on this particular situation, which attracted considerable attention at the organization involved. The number of removals per aircraft was tabulated for the first quarter of 1977. The expected value was 2.4375. Seven or more removals were observed in four aircraft, which is far above the expectation for this number of removals. This is an indicator that something might be wrong on those particular aircraft.

The same type of analysis can be applied to the individual radio chassis. An expectation of 0.84 removal during the 3 months was the appropriate measure. Four or more removals of a particular radio -71- would be expected to occur on only one radio during the time period, but five such radios were observed, suggesting that individual serial number radios also had a problem.

An even further examination of this particular field problem involves the individual failure modes. The tabulation below shows that two adjustment jobs constitute a significant percentage of all of the problems encountered in these sets during the first quarter of 1977. The other principal contributor, a transistor, is now known to be the result of a deficient specification in the original manufac- ture; a campaign is now being conducted in the field to replace all of these transistors. The last two entries in the table are probably not epidemic problems.

REPAIRS INDICATING PREVALENT FAILURE MODES ADJUST AUDIO 9 REPLACE Q208 9 ADJUST SIDETONE 8 REPLACE Q604 4 REPLACE Q410 3 Total 33 = 50 percent of all repairs Another of the clear-cut issues in this instance is the effect of ancillary equipment on total system reliability. Many reliability analyses do not include the ancillary equipment on the basis that it is presumed to be an unimportant factor. However, experience with current avionics equipment indicates that this is simply not the case.* For example, during the first quarter of 1977, the cargo car- '"It was not the case more than 10 years ago, as already shown in Table III-l.

-72- rier mentioned previously had the following experience with the com- munications subsystem of their aircraft: MAINTENANCE EXPERIENCE: (FIRST QUARTER 1977) QPA» ITEM REMOVALS PERCENT MTBF 3 HEADSETS 60 28% 2 AUDIO PANEL 53 24% (600 HRS) 3 RECEIVER-TRANSMITTER 78 36% 2 MICROPHONES 25 12% -QUANTITY PER AIRCRAFT In the table above, failure information for the voice recorder was omitted, since it is not an essential part of the system. Note that the audio panels accounted for 24 percent of the removals during the 3-month time period. Their MTBF was approximately 600 hours, the lowest MTBF of any item in the assembly, even during a time when the receiver-transmitters were an epidemic problem. Note also that head- sets and microphones — clearly ancillary equipment--made significant contributions to removal rates during the same time period.

In summary, then, general aviation avionics reliability does not seem to differ much from the reliability characteristics derived for other classes of electronic equipment, nor for that matter from avion- ics reliability in military systems.

Given that reliability estimates based on actual records from general aviation avionics usage are hard to come by, one can reason- ably ask what other tools might be brought to bear. One obvious -73- answer is the previously described reliability prediction work, par- ticularly MIL Handbook 217B. In that specific case, some precautions should be observed. We know, for example, that MIL Handbook 217B has a bias toward a higher reliability than is usually justified in estimating. Also, commercial parts are normally used in general avia- tion avionics equipment, rather than MIL spec parts as customarily assumed in a 217B analysis. Furthermore, the 217B approach assumes a relatively benign environment in general, even though an allowance is made for being airborne in a manned environment as an example.

Finally, less than optimal installations frequently occur in the gen- eral aviation avionics business, and these are known to have a ten- dency to reduce inherent reliability [72]. If the above factors are not considered in a reliability analysis, the outcome will indicate an unreasonably optimistic reliability prediction.

If a system design is predominantly the assembly and interconnec- tion of sub-units that are in an initial production phase or in a mature design and usage stage, then estimation of reliability of the final system offers far fewer challenges, at least for an initial evaluation. Under these circumstances, field reliability estimates for individual components are entirely appropriate, though somewhat difficult to obtain. Assuming that such estimates can be obtained somehow, there are still some limitations which should be considered as applicable to the reliability figures thus derived. The assumption is inherent in most field reliability estimates that the highest qual- ity of maintenance will be available, and this frequently is not the case. Another assumption will be that no untested installations will be represented, and this also is seldom true in general aviation -74- equipment. Long flights are likely to make disproportionate contribu- tions to the reliability data bank available to anyone looking for such information, and these are not typical of general aviation avion- ics. Likewise, heavy use will be a characteristic of those data sources which will constitute the bulk of the information. Always, an allowance must be made for interface problems, which are highly likely to be a significant factor, and may even dominate the reliability pic- ture for such an avionics assembly. Finally, to be fair, we should note that flight reliability should be better than the customary MTBF calculations show, because of the turn-on stress mentioned earlier.

One rather novel approach has been developed for estimating avionics reliability, with at least some degree of success. Many researchers had previously found that support costs derived from maintenance activities can be approximated for a fair variety of equipment as a few percent of original equipment cost per year [45].

This can be translated into a method for estimating the reliability of prototype or early production systems, and all such methods seem to have flowed from a study done at The Rand Corporation 25 years ago [78]. As an example, an ARPA study done a few years ago noted that for nearly fifty Air Force avionics units comprising communication, navigation, and computer functions, a fair approximation of field reliability was provided by the single measure of unit cost, in a sim- ple inverse relationship: 1.3 x 10 MTBF = DOLLAR COST -75- Obviously, very cheap or very expensive equipments may deviate consid- erably from this relation. The study also examined techniques by which certain special equipments had mean times between failure one to two orders of magnitude (i.e., 10 to 100 times) higher than more representative units [74].

CONCLUSIONS AND RECOMMENDATIONS In this section, we have tried to illustrate that both the pred- iction of the reliability of future systems and assessment of the reliability of existing systems pose significant problems for the unwary.

Reliability predictions tend to be severely biased toward the high side, mainly by various factors associated with competition, but also by typically novel aspects in new designs: new and untried cir- cuits, new parts instead of standard, etc. All of these make the prediction of future numerical reliability extremely hazardous. How- ever, the prospects for a more limited objective in this area—the comparison of two (or more) candidate designs for their inherent rela- tive reliability—are really quite favorable, given only few easily understood precautions. First, use of the standard MIL Handbook 217B methodology ensures that many factors known to affect reliability are assessed equally between or among candidates. The use of a newly available digital computer program for this purpose increases the assurance of completeness and lack of bias (both essential to produc- ing the best of what may still be flawed predictions), and at the same time reduces the effort and expense required to provide this vital comparative measure.

-76- The specific procedures consist of: 1. Listing the individual electronic parts of the system; this means that a detail design must exist.

2. Associating all relevant parameters with each part, such as operating conditions, quality level, etc.

3. Identifying special characteristics of parts (number of gates in an 1C, voltage rating of capacitors, etc.).

4. Applying the above information to tables in MIL handbook 217B.

This detailed procedure does not mean that these most modern evaluation methods cannot be misused to provide extravagant reliabil- ity estimates. They can, by the relatively simple expedient of minim- izing initial design complexity to the point where the later addition of circuits and components will become necessary in order to meet per- formance requirements. Even higher reliability estimates may be had through optimistic parts selection and derating policies. It should be clear that such unrealistic assumptions are less likely to be accepted now, given today's standardized estimating tools.* A second appropriate course of action should be a continuing sur- veillance of the current state of avionics reliability in general avi- ation. The assessment of current avionics reliability is made diffi- *A related problem arises with regard to comparing systems in different stages of development, as for example the SIU and STI/MILCO PCAAS proposals, the former a prelimnary design and the latter mainly an assembly of off-the-shelf units.

-77- cult primarily by the substantial number of factors known to affect it, and the lack of representative data on GAA usage, the latter being caused by the understandable reticence of those holding even imperfect information on the subject to share it with the world. This situation is relieved somewhat by the growing existence of experience data banks on reliability among users of equipmment as opposed to manufacturers, but because these users tend to be large, they represent the more favorable side of environment, installation, and maintenance, and even of length of flight and intensity (hours/day) of use.

A third appropriate reaction is to watch for relevant experience with other integrated (multiplexed) systems, particularly commercial and Air Force avionics. Expanded activities in these areas should make more reliability information available on that subject.

Needless to say, sufficient problems remain to cause even profes- sional reliability engineers to have a conservative view of what can be accomplished. Two of the more outstanding areas in which informa- tion is inadequate are those relating to interface problems, which have yielded only somewhat to accepted standards in (for example) mul- tiplex systems, and software reliability, which becomes a questionable area in virtually every new system.* If any one recommendation is worth making about this subject, it is to beware of false prophets, who have been around since reliability emerged as a discipline. There are no panaceas, and only a relatively *The second edition of the well known "Reliability: Management, Methods and Mathematics" by David Lloyd and Myron Lipow has a long chapter devoted to this subject [79].

-78- limited (though increasing) number of viable tools that, when used with appropriate care, can often deliver enlightening results.

Finally, we have observed that quality of maintenance has demonstrably beneficial effects on reliability as reflected in mean times between failures. Some implications of this fact are explored in the next section.

-79- IV. MAINTAINABILITY OF ADVANCED AVIONICS SYSTEMS INTRODUCTION The subject of maintainability for general aviation avionics equipment has been of deep concern to manufacturers, service shops, and professional organizations (for example, the Aviation Electronics Association (AEA)) for many years. Through the efforts of these organizations maintainability of general aviation avionics equipment has been improving, but avionics maintenance is still a significant part of the cost of ownership.

In promoting the development of highly integrated avionics sys- tems for general aviation aircraft, the problem of maintainability is understandably of deep concern to NASA. This concern stems partly from the known problems and costs of maintaining contemporary equip- ment, and partly from the fact that in the few instances in which com- ponents have been interconnected in an architecture similar to that shown in Fig. I, the problems of fault isolation have been found to be very different from those encountered in more conventional systems.

The capability of incorporating a range of self-test programs in these systems is appealing, in the expectation that it could greatly sim- plify fault isolation. However, a significant incorporation of these features in an operating system is yet to be completely successful.

The initial objective of the maintainability part of this study was to develop a generalized methodology that would allow the predic- tion of maintenance costs (man-hours/flight hours) as a function of design/performance parameters. The development of this methodology depended on the availability of field experience data on avionics from -80- the general aviation community, and our ability to identify the per- tinent design/performance parameters.

During the course of the study, we attempted to locate data and identify design/functional parameters through interviews throughout the avionics industry. We found that field data on maintenance are equally as scarce as cost and reliability data. Although some FBO's keep records on maintenance action, the cost of assembling these data is prohibitive. It was also difficult to ascertain if there were enough data in the records to permit reasonable statistical analysis.

Although the study was unable to produce generalized methodolo- gies, the interviews with the many organizations and individuals pro- duced considerable information that can be useful to anyone designing avionics equipment for maintainability, and for NASA to judge the maintainability of a proposed design.

In this section, we have assembled the results of our industry interviews. We have tried to identify and, to the extent possible, analyze the many factors that might be useful in evaluating or design- ing an advanced avionics system. In trying to piece together the parameters affecting the cost of ownership, one is quickly made aware that maintainability is a complicated concept involving such varied factors as quality of service people, service organizations, equipment design, documentation, usage environment, test equipment, and test procedures.

In discussing maintenance with a number of users, it is clear that they have more than a passing concern about avionics maintenance.

On the average, it has been estimated that the maintenance bill for contemporary solid-state general aviation avionics equipment runs -81- about 3.5 percent of the original cost per year [45]. This does not, however, represent the total cost picture, as the user experiences it.

In many cases it requires 2 or 3 days to troubleshoot and repair a complicated unit, and the aircraft is usually unavailable for use dur- ing this period. Consequently, the knowledgeable purchaser of a new avionics suite is not only looking for advanced capability, but also, as a concomitant goal, for the reduction of the maintenance costs and the mean time to repair (MTTR).

Most of the documented experience in designing for maintainabil- ity is either from the military or the commercial airlines. While much can be learned from looking at this experience, it should be recognized at the outset that the maintenance structure for supporting the general aviation user is very different from that of either the military or the airlines. In addition to organizational differences, the skill levels of the people, the complexity of the equipment, and the economics are also different. In an airline operation, economics dictate that a quick return of the aircraft to service is the number one objective, even at considerably higher maintenance cost. This requirement for a short MTTR also exists for military aircraft in time of crisis operations or combat; consequently, the military maintenance philosophy must also be geared to this type of operation.

AIR FORCE EXPERIENCE In the Air Force operation, the user seemingly has substantial control over his destiny. Maintainability requirements can be written into procurement contracts to ensure that newly acquired equipment can be supported. Also, the avionics maintenance personnel are under the -82- direct control of the user. The training level of the people is set by the user, and if additional training is believed to be needed, it is provided at the user's expense. The user also controls who repairs what, and where repairs will be made. Inventories of line replacement units (LRUs) are maintained at the expense of the user, so that an aircraft can quickly be returned to service with simple replacement of failed units. Malfunctioning units that have been removed from ser- vice are then repaired in the shop and returned to inventory. In the case of the Air Force, different levels of shop maintenance are authorized for different organizational entities, e.g., minor repairs (printed circuit board replacement, etc.) are performed at the base shop, and units requiring major repairs are shipped to a depot. In either event, each LRU is repaired by someone who has specialized on that particular unit. Both the Air Force and the airlines have con- cluded that today's avionics are so complex, and have such a multipli- city of functions, that it is not reasonable to provide in-depth training in each system for all line and shop personnel.

In spite of having complete control over its maintenance operat- ing equipment, philosophy, and personnel, the Air Force is still hav- ing serious problems in maintaining the highly integrated advanced avionics systems that have entered the inventory since 1968. This has caused much concern, and considerable effort is being expended on understanding this problem.

A 1974 Rand study [8] took a broad comprehensive look at the Air Force problem in developing and maintaining these highly integrated systems. Some of the observations and conclusions of this study may provide insight into approaches to designing for maintainability in -83- integrated avionics systems for general aviation. A few of the more important ones are presented here.

The following conclusions were held hy the 1974 Rand study team members at the beginning of the study, and formed the basic premises for their research: 1. In the development of a new component, there cannot be complete a priori knowledge of that component's failure modes.

2. In the integration of known components into new arrays, new failure modes are introduced which cannot be completely predicted.

3. The only way we know of thus far to identify these failure modes prior to operational deployment, and to provide reliable verified diagnostic procedures for subsequent field identification, is by long and expensive test programs.

4. To perform technical improvements repeatedly in a new weapons acquisition program is expensive, time-consuming, and operationally imprudent.

It was found that the interaction of the units of a highly integrated system sometimes produced results that were unexpected.

For example, the report stated the following: Modern avionics systems are integrated through a digital computer; or, as in the FB-111, a group of digital computers receive information from the various items of equipment, process it, and provide corrections back to the various pieces of equipment plus output indications to the crew. Through the corrections provided to individual pieces of equipment, interactions occur between the pieces.

An output from one item of equipment in an integrated system reflects not only its own performance but also that of others. Moreover, a malfunction in one item of equip- ment may be detected more clearly in the output from another. Such a coupling effect from one piece of equipment to another is a new phenomenon in avionics, and is one characteristic of highly integrated systems.

-84- It was observed that the performance of an integrated system w;is determined by how closely the equations that constitute the mathemati- cal model of the system represented the real world and the desired performance. Any failure in the model produced a faulty operation, the same as a malfunctioning piece of hardware would cause faulty operation. Much of the problem experienced by the Air Force in util- izing integrated systems was identified as the failure of the model to accurately describe the functional requirement and the real world in all operational situations.

Another serious problem identified by the Rand study was inade- quate fault-isolation capability. This aspect of the problem was characterized in the following way: The nature of. fault isolation has changed under the impact of integrated, complex avionics systems and highly automated test equipment for inflight diagnosis, shop repair, and depot repair. No longer does a pilot describe a malfunc- tioning piece of equipment to ground personnel, who proceed to trouble-shoot the equipment. Instead the functioning of a built-in test equipment light signals a malfunction to the ground crew, who remove the indicated item of equipment and take it to the avionics maintenance shop. There the equipment is subjected to a complete standard test on automated test equipment. If it fails the test, the indi- cated plug-in modules are replaced and the test is repeated.

If the equipment cannot be repaired, it is returned to the depot where automatic test equipment is used in a similar fashion by more experienced people.

This growth of automatic test equipment at all echelons of repair standardizes the testing and reduces the training required to trouble-shoot the equipment; however, as the above description illustrates, it almost completely cuts off communication between the levels of maintenance and it places total reliance on the fault-isolation capability of the automatic test equipment.

Unfortunately, the fault-isolation capability of built-in tests and shop test equipment has not been adequately tested. Current procedures only show that the test equip- -85- raent will pass good units and will fail completely faulty units. Since the actual distribution of failure modes is not known when the test equipment, its associated computer programs, and its operating procedures are developed, it is not surprising that the effectiveness of the test equip- ment in fault-isolating field failure modes is completely untested.

In spite of this uncertainty, the capability of the built- in test equipment and shop aerospace ground equipment to isolate faults is taken for granted in the maintenance philosophy and in spare provisioning. This, and the lack of communication between levels, leads to frequent dis- agreement between test results at different levels of maintenance. A fault indication will appear on a flight, but subsequent shop test shows no fault with the equipment.

Components are removed in a shop test as faulty, but in depot testing they show no fault. In the absence of special tests, these results appear only in the aggregate results of higher-level testing, since the lack of communi- cation between levels obscures results in specific cases.

The pertinent point is that deficiencies in fault-isolation underlie a large part of avionics problems. To correct these deficiencies will require improved validation of the capability. In practice, extensive testing of the equip- ment and computer programming affecting fault-isolation is extremely lengthy and expensive. A possible compromise is to reduce the scope of the problem through the use of already developed and tested building-block equipment components. With these, the principal failure modes and symptoms should be well understood before the equipment is used in a particular system. Testing of fault-isolation can concentrate on the limited set of new failure modes induced by the particular application or environment of the new system. Divide and conquer, so to speak.

The maintenance problem of integrated avionics systems in the Air Force has been further complicated by the inability to retain mainte- nance personnel. Data collected in a T976 Rand study [6] indicated that in 1975, the probabiity that a person who initially entered the avionics flight-line maintenance career field would remain in the Air Force to his fifth year of service was 14 percent.

-86- AIRLINE EXPERIENCE The air carriers do not face the same complexity level as the Air Force. Their requirement is to get from point A to point B safely in all kinds of weather. Military avionics, on the other hand, has the additional complication of sophisticated weapons systems. The air- lines also maintain a more conservative approach to avionics systems.

This conservative posture is exemplified in a number of ARINC stan- dards; for example, the recent standard for DITS (MARK 33 Digital Information Transfer System). The standard excludes multiplexed use of transfer buses, allowing only one information source on each bus.

All of this is not to say the airlines do not have complex maintenance problems—only less so than those faced by the Air Force.

One requirement in airline maintenance is fast turnaround time.

The average time between arrival and departure is 1/2 hour for through flights, and 1 hour for turnaround. During a recent seminar on air- line avionics maintenance [5], one airline official summed up the problem as follows: In the amount of time listed, we and notably the manufacturers expect a line mechanic to look in an aircraft log book, analyze a hurriedly written set of symptoms about a system or systems that transcend three or more ATA chapters in the maintenance manual, may involve 50 or more LRUs, and make the right decisions with at least 85 percent probability of success!! With the present troubleshooting aids, or lack of aids, they do well to achieve their present success rate.

-87- The principal maintenance aids currently used by line technicians to isolate problems are fault balls," BITE (Built In Test Equipment), inline monitors, and ground test equipment. The following evaluation of the quality and utility of these different diagnostic aids was presented by Mr. Jim Takeuchi of United Airlines at the Avionics Maintenance Conference: Fault Balls The airlines are strongly of the opinion that fault in most of today's so equipped LRUs are ineffective and even misleading. Specifically, they are classified as "oversensitive" and "unreliable." A very few LRUs are referred to as having reliable and effective fault ball operation. A survey conducted some time ago at United illustrates the point: of 940 aggregate fault balls on 24 LRUs installed in wide-bodied aircraft, 408 were tripped, eight of which with related write-ups and 400 without any write-up in the log book.** Built In Test Equipment (BITE) The airlines are almost unanimous that BITE as it exists today is inadequate. It is expensive. It is insufficiently reliable within itself, and has a very low confidence level. Some airlines seriously question the cost effectiveness of BITE. Some time ago an engineer at United made a study of the 727 A/P BITE reliability and came up with the following interesting observations: a. Following a gripe, if a unit self tests good, is the system good? - 53 percent yes.

b. Following a gripe, if a unit self tests bad, is it bad? - 60 percent yes.

*Fault balls are electromechanical indicators used to indicate the occurrence of an event. They can be triggered by a voltage level or a pulse, and once triggered, they remain triggered until they are reset mechanically.

**Airline responses to a questionnaire indicated that the concept of fault balls itself was not in question. It is the circuit that should be refined to be practical.

-88- c. Following a gripe, if a unit is replaced straight off, without self test, is the plane fixed? - 63 percent yes.

Historically, BITE or self test was given up as something extra or a nice thing to have. It seldom came as a result of a solid maintenance management plan. It needs a systematic approach. Any self test, for which we pay plenty, must be considered as an integral part of the basic system design, not as an added on feature. ...We the airlines, need on board line maintenance aids and testing methods that are at least 85 percent effective in localizing a defective LRU, transducer relay, etc. This is a must for multi box systems.

In-Line Monitor As the purpose of the in-line monitor is to inform flight crews of failure(s) detected in flight, and not necessarily to be used as a primary line maintenance tool, it provides little help for the line mechanic. Nevertheless, the in-flight monitor has been a source of many unconfirmed removals. An important point is that the monitor threshold levels must carefully be determined to minimize unnecessary removals. It is one thing to establish design tolerances; it is quite another to see that these tolerances closely relate to the level at which pilots begin to recognize performance degradation.

Line-Test Equipment Eighty percent of the airlines feel that today test boxes are so complex or inconvenient that they cannot practically be used except on layovers, and that simple inexpensive function no/go go test boxes would help isolate quickly to the LRU level.

The airlines are in total agreement that improvements must be made in fault-isolation methods. The current shotgun approach to mak- ing rapid repairs (replace everything that could possibly cause the problem) is too costly. The capital cost of purchasing enough addi- tional spares is tremendous, and the shop costs to constantly test and -89- return these spares to stock further increase the cost. In addition, the wear and tear on plugs, connectors, and LRUs under constant remo- val and reinstallation ultimately cause further problems. The number of nonconfirmed removals (NCRs) now runs as high as 50 or 60 percent, and promises to become worse with the addition of more complicated equipment, unless better methods of fault isolation are developed.

Unfortunately, if an airplane must be back in the air in a half hour or so, the shotgun approach is currently the only effective solution.

If repeated squawks followed by a maintenance action are required to ultimately find a_ single failure in an LRU, then it becomes clear that reliability moves down on the importance scale, and maintainability becomes the dominant issue.

In summary, airline experience in avionics maintenance shows the number one issue to be fault isolation in the aircraft system.

Present methods are woefully inadequate. The next level of importance is one of LRU maintainability. The 1977 ARINC Avionics Maintenance Seminar [5] identified, and discussed in detail, a number of areas that need improvement: piece parts, printed circuit boards, connec- tors, thermal design, component accessibility, testing methods, etc., and, finally, the need for better documentation. Unfortunately, docu- mentation has not kept up with the sophistication of the new equip- ment. A maintainability questionnaire, consistin of 24 detailed ques- tions, was answered by 40 different airlines, a summary of which is included in the Proceedings of the ARINC Maintenance Seminar. Since many of the issues are equally relevant to general aviation avionics, the summary of this airline response to the maintainability question- naire should be referenced by anyone designing advanced general avia- tion avionics equipment.

-90- GENERAL AVIATION There is very little documented experience on maintainability of general aviation avionics, and even if there were, it is doubtful that it would give useful insights into the problem of maintenance in highly integrated systems. The best source of information is the mil- itary and airline experience previously discussed; however, the differences in the requirements and the organization must be kept in mind at all times.

The general aviation user has no standard procedure for maintain- ing equipment. Most users rely on the independent avionics repair shop for service; however, some larger FBOs do maintain and control their own independent facilities and personnel.

All shops performing service on installation must be certified by the FAA. The rules governing the shop operation and certification are set down in Civil Aeronautics Manual 52. CAM 52 requires that a shop have (a) appropriate service and instruction manuals issued by the manufacturer, and (b) minimal prescribed electronic test equipmment.

Also, the individual in charge of inspection, maintenance, and overhaul must have had 18 months experience in the work he is supervising. The FAA currently divides avionics equipment into three classes: communications equipmment, navigation equipment, and radar equipment. A shop must have certified personnel and equipment for each class of equipment that it will install or service.

On the surface, these minimal safeguards seem enough; but in practice, they afford very little help to the user of the service. It -91- is very difficult for a general aviation user to make an a priori evaluation of a particular independent shop, or of the individual capabilities within a shop. One shop may be capitalized at $30,000, which probably represents the bare minimum of equipment and material, whereas another shop may have $200,000 or $300,000 in sophisticated test equipment. Also, the FAA requires only that the individual in charge be certified; the qualification of the remaining service people is the responsibility of the management. A requirement of 18 months of practical experience for a supervisor is very little assurance of competence. Also, the required management certification of competence adds very little to the user's confidence level.

Competency in service personnel is one of the most serious prob- lems faced by the general aviation community. Many of the service technicians stay with general aviation only long enough to build up experience; then they move into airline jobs where the pay is higher, the fringe benefits are better, and the working hours and conditions are better. There are some good avionics shops throughout the coun- try, and there are some highly qualified competent people in the busi- ness, but discussions with a number of FBOs who depend on independent avionics shops point to the general inadequacy of this service for the general aviation community.

Another area where general aviation is different is in the con- cept of an LRU. In an Air Force or airline operation, all replacement units are owned by the user, and replacing an LRU is simply a matter of exchanging something the user owns for something else he owns.

This is seldom the case in general aviation. A typical aircraft owner would be reluctant to accept the exchange of functional black boxes as -92- a means of repair unless the units he received were identical and in as good condition as those he originally bought. Also, the service shop could not possibly afford to stock serviceable units of all the myriad different makes and models of equipment he is likely to encounter. The maximum LRU level of complexity that seems acceptable to the average owner, and that would be reasonable for most avionics shops, is the printed circuit (PC) board level. The PC board as a field replacement unit has gained wide acceptance in repairing home television sets, computers, computer terminals, and other electronic equipment.

In designing new systems, it is the responsibility of the manufacturer to decide on the maintenance philosophy to be followed.

Once that philosophy is established, equipment design and factory sup- port must conform to that philosophy. For example, if the principal LRU is to be the printed circuit board, a number of things must be done. The system design must incorporate pluggable PC modules, the modules should be accessible without removal of major chassis, and the modules should conform to system function in such a way that they will support fault diagnosis. System partitioning into LRUs can be devised in such a way that it will minimize the time required for fault diag- nosis (and thus, support costs), provided this aspect of system design is considered simultaneously with other system requirements and maintenance planning.

It is highly unlikely that avionics manufacturers can provide, or influence to any significant degree, the incentives needed to attract good, high-level people to the general aviation avionic maintenance career field. Also, any pressures applied by the FAA in the form of -93- more rigid certification requirements will only serve to reduce the number of people who choose avionics maintenance as a career, assuming that salaries remain the same. This is unfortunate, but it is a fact of life that must be reckoned with. Another problem is that contem- porary avionics maintenance personnel are trained and experienced predominately in analog devices, with little or no knowledge of digi- tal circuits and systems. Although training and exposure over time will change this, there is no way to quickly implant many years of experience into the profession.

The ability of maintenance personnel is an additional factor that must be considered by the manufacturer in establishing maintenance philosophy. A reasonable approach is to conduct research and experi- mentation on job performance aids and training methods to enable the utilization of persons of lower levels of ability. This approach has been used by the television service industry and to some extent, the computer industry, with reasonable success.

Another part of the maintenance philosophy relates to the kind of support required from the manufacturer. If the intended philosophy is that PC boards are not normally repaired by the independent avionics repair shop, then exchange serviceable boards should be available to the shops within 24 hours or less. Since shipping time is an impor- tant factor, regional resupply or repair stations may be required.

Regional factory repair stations could provide complete maintenance service to the user; however, the impact of this competitive posture with the independent shops should be examined carefully. Advanced avionics systems that are highly integrated, such as the Demonstration Advanced Avionics System (DAAS), offer some unique challenges in the -94- area of maintainability. There is some military experience, and there is some experience from other industries; however, the applicability of this experience to general aviation is not completely clear. For this reason, the maintenance philosophy and the means for carrying it out must remain flexible to accommodate what is learned about the real problems and the effectiveness of the system to deal with them. This means that information feedback to the manufacturer must be planned and provided for. One means of keeping the manufacturer in the infor- mation feedback loop longer is by providing extended warranty services (beyond 1 year). This would discourage unfranchised installations, and encourage factory-controlled maintenance. Extended warranty ser- vice contracts have been used successfully by RCA in television war- ranties, and by Sears Roebuck & Company in the maintenance of home appliances.

CONCLUSIONS AND RECOMMENDATIONS The Air Force and airline experience with advanced avionics equipment shows fault diagnosis and isolation to be the number one maintenance problem. One of the problems that frustrates the design of tests (i.e., diagnostic program development) for detection and location of faults is that when systems are highly integrated, failure modes are introduced that are very difficult to predict. For this reason it is essential to emphasize fault isolation early in the design stages. To ensure that maintainability is considered early in the design, a requirement must be placed on the designer to establish a maintenance philosophy and plan as part of the preliminary design.

An evaluation of the philosophy and plan should be conducted at each -95- design review to ascertain if they will work, given the technical capabilities of existing maintenance personnel and the organizational structure.

A part of the early design should include a model describing the equipment performance; however, since real world performance cannot be predicted precisely, it is important to continue the testing and updating of this model throughout the design, development, prototype testing, and early operational phases of the program.

Although the contemporary approach is toward automatic fault diagnosis, there is still a requirement for some degree of human interaction in the operation of tests, and in the interpretation of test results. This human interaction in the fault diagnosis process must also be tested during the development process.

In testing of human interaction with the fault diagnostics, the technical capabilities of the individuals involved should be carefully considered. To utilize design engineers with a deep familiarity with the intricacies of the system and the automatic test procedures (ATP) will reveal very little about how a line service technician can use the ATP for fault isolation. Testing must be representative of the real world.

The opinion of people who have responsibilities for the mainte- nance of avionics is that the PC board level is probably the best level for a line replaceable unit (LRU) in the kind of electronics that will be used in advanced avionics systems. There are strong indications from other industries (e.g., television, computer, etc.)

that for much of the electronic equipment, the PC board is a good modular level to establish as an LRU. It may be that a larger unit -96- will have to be removed from the airplane and shop tested, but even then, the actual replacement part probably should be a PC board. It must be recognized that these other parts of the avionics structure that are not PC boards are still contributors to equipment failures.

These components must each be analyzed and provided for in the overall fault diagnosis procedure and maintenance plan.

-97- V. SOME SOFTWARE CONSIDERATIONS We have chosen to discuss software separately in this study in order to give it special emphasis; however, this is not to imply that software should be considered separately. Software and hardware are closely interrelated components in a system, and are not individual entities to be developed separately, using different management and control techniques. The computer and its associated software are but one element of the system and thus should be treated as a holistic subsystem during most phases of the design process.

Any of the proposed advanced avionics systems (AAS) will clearly be increasingly reliant on larger and more sophisticated computer pro- grams to perform many of the basic functions. These may be automatic functions, such as the monitoring of aircraft systems, or they may be operations requested by the pilot. In addition to performing in- flight operations, computer programs may play an important role in fault diagnosis and in the maintenance of the avionics system itself.

Because of the safety-critical nature of many of the in-flight opera- tions, as well as the potential for reducing system maintenance costs through fault isolation, it is important that avionics system software reliability and maintainability factors be carefully considered throughout the entire system development process. This section discusses some of the problems that have been experienced in develop- ing software, and some generally accepted guidelines for minimizing these problems.

Traditionally, software has been viewed as completely reliable once it becomes operational, simply because its performance does not -98- degrade over time or through repeated use. However, experience has shown [79] that software can perform to an acceptable level most of the time, even though it contains coding or design errors in seldom- exercised logic paths. This is especially true of complex systems.

Performance of this kind might be adequate if one were willing to accept degraded performance or could tolerate occasional errors. But in applications involving flight safety, the loss of some key function or faulty operation is intolerable. Experience has also shown that software flaws may not show up in the testing phase, and a failure may first appear months or even years after the system is placed in opera- tion.

Lloyd and Lipow [79] present twenty-four definitions of software quality characteristics. Two that are critical to AAS software are included here to clarify the goals of good design.

Reliability: Code possesses the characteristic reliability to the extent that it can be expected to perform its intended functions in a satisfactory.manner.

This implies that the program will compile, load, and execute, producing answers of the requisite accuracy; and that the program will continue to operate correctly, except for a tolerably small number of instances, while in operational use. It also implies that it is complete and externally consistent.

Maintainability: Code possesses the characteristic maintainability to the extent that it facilitates updating to satisfy new requirements or to correct deficiencies.

This implies that the code is under- standable, testable, and modifiable, e.g., comments are used to locate sub- routine calls and entry points, visual -99- search for location of branching statements and their targets is facilitated by special formats, or the program is designed to fit into avail- able resources (memory) with plenty of margin to avoid major redesign.

There are differences between software and hardware reliability and maintainability, and these need to be well understood in assessing a particular design effort. For example: 1. Software, unlike hardware, does not wear out or degrade over time or with use. However, software may "fail" (i.e., not perform according to specifications) because of a hidden flaw in the program design or a mistake in coding.

2. New imperfections are not introduced in making copies of a computer program, but existing errors are reproduced faithfully.

3. Computer programs are unconstrained by physical laws and therefore are not susceptible to a priori proof that design objectives are impractical.

4. There are many more distinct configurations and paths to check in software than in hardware.

5. Software failure modes are usually different from hardware failure modes. Software will fail without warning and may leave no indication that a failure has occurred, or of the time and source of the failure.

-100- Another difference is that people, including those who manage system development, have a much better intuitive understanding of hardware than they do of software. Thus, software problems might sometimes be overlooked until it is too late to take reasonable corrective action. System managers and developers of software must be aware of the characteristics of good software design, and how to achieve them during the different phases of development.

It is clear that the way to produce reliable software is to remove all errors from the software package prior to placing it in operation; however, this is extremely difficult to do in actual prac- tice. It means producing software that has minimal errors in the first phase, and then testing and debugging it until all remaining errors are eliminated. In large software systems, several years of use may be required to find all the mistakes and bugs.

One study [90] showed that over 60 percent of software errors occur during the requirements formulation, preliminary design, and detailed program design phases, and that less than 40 percent of them arise in the programming and coding phase. From this, one can see that the basic fundamentals of error prevention are simple: a func- tional structure that represents realistic requirements, and care in producing the system. It is extremely difficult, however, to state valid, complete, and unambiguous requirements for a new system.

Therefore, any requirements that are developed must be subjected to vigorous scrubbing by some adversary process involving experienced system users to test their validity and completeness. Once a good set of requirements has been achieved, the design and coding of the system must proceed with great care. Some generally accepted techniques for minimizing errors in the design and coding are: -101- 1. The use of a hierarchical approach (e.g., top-down structured programming) with parallel refinement of functional requirements.

2. The use of higher order languages (HOLs) unless assembly language is necessary to minimize storage and for timing problems.

3. The use of organizational structures conducive to effective software production, e.g., "Chief Programmer" concept [91].

AAS software will require attention throughout the lifetime of the system for a number of reasons. First, there will be errors to correct, even after a program reaches the field, and more effective methods will be developed for performing certain functions. Second, as the system matures in the field, enhancements and new functions will be added to meet specific new operational requirements. These additions will be especially important to commercially produced sys- tems that must be competitive with improved products from other manufacturers. For this reason, a plan must be devised early in the design phase to allow for a software maintenance activity.

To support program changes, software documentation must be main- tained throughout the life of the system and a strict procedure for updating it must be established. This procedure should (1) ensure that corrections made to one copy of the documentation are reflected in all copies; (2) allow evolution from one system .release to another to occur without affecting system operations; and (3) ensure that all changes to a system are thoroughly tested before the system is sent to • the field.

-102- Maintainability is a feature that must be planned for in the sys- tem design. Some general guidelines that have been found effective are: 1. Use a modular design approach.

2. Limit the size of each module and limit the functional responsibility of each module.

3. Use a coding structure that is easily understood.

4. Include complete and consistent comments in the code.

5. Prepare documentation consistent with internal code comments.

CONCLUSIONS AND RECOMMENDATIONS In this section, we have discussed, briefly, some important but often overlooked factors in the development of systems with a substan- tial software component. Software problems are frequently traceable to the early conceptual stage, and often result from poor and unreal- istic system requirement specifications. Failure to employ accepted software engineering techniques in producing the system design can also cause problems. Finally, errors frequently go undetected during the coding and integration phase because of inadequate testing and poor documentation.

To prevent these problems, management and designers should assess the software system and its components throughout the development pro- cess in terms of the following critical questions: -103- 1. Have the requirements been scrubbed and subjected to an adversary process?

2. Have hardware and software tradeoffs been made?

3. Are good software engineering procedures being included in the design process?

4. Does the design allow for easy modification and for update of the software?

5,. Does the design facilitate testing, and is there an integrated test plan?

6. Is documentation and documentation maintenance being planned for?

It is not only important to ask these questions, but it is neces- sary that someone be able to accurately interpret the answers. For this reason, some software development expertise should be available in all design reviews. Although structured programming and Chief Pro- grammer organization are recommended, it should be emphasized that they will not offset the effect of poor programmer motivation and talent.

-105- PAGE BtAtJK NOT Appendix A ORGANIZATIONS CONTACTED DURING STUDY I. Manufacturers of General Aviation Equipment o Bendix Avionics, Ft. Lauderdale, Florida o Collins Avionics, Cedar Rapids, Iowa o RCA Avionics, Van Nuys, California o King Radio, Olathe, Kansas o Narco Avionics, Ft. Washington, Pennsylvania o Delco Avionics, Santa Barbara, California II. Manufacturers of General Aviation Aircraft o Cessna Aircraft, Wichita, Kansas o Beech Aircraft, Wichita, Kansas III. Users of Avionics Equipment o Federal Express Corporation, Memphis, Tennessee o Krueger Aviation, Santa Monica, California o IFR Electronics, Van Nuys, California o United Airlines, San Francisco, California o ARINC Incorporated, Annapolis, Maryland o ARINC Research, Annapolis, Maryland IV. Air Force Agencies o Rome Air Development Center, Rome, New York o Air Force Avionics Lab, Wright-Patterson AFB, Ohio o F-16 Program Office, Wright-Patterson AFB, Ohio o Advanced Airborne Command Post Office, L.G. Hanscom Field, Massachusetts V. Government Contractors o McDonnell Douglas, St. Louis, Missouri o Dynamics Research Corporation, Wilmington, Massachusetts o Hughes Aircraft Company, Culver City, California o Charles Stark Draper Labs, Cambridge, Massachusetts VI. Commercial Companies o INTEL Corporation, Sunnyvale, California o Delco Corporation, Milwaukee, Wisconsin o Hewlett Packard Corporation, Palo Alto, California -106- VII. Professional Organizations o Aircraft Owners and Pilots Association (AOPA) o National Business Aircraft Association (NBAA) o General Aviation Manufacturers Association (GAMA) o Aircraft Electronics Association (AEA) -107- REFERENCES 1. Seacord, C.L., and Barrel Vaughn, "Computer Technology Forecast Study for General Aviation," NASA CR-137889, Honeywell, Incorporated, St Louis Park, Minnesota, June 1976.

2. "Transcription of the Workshop on General Aviation Advanced Avionics Systems," NASA CR-137861, Stanford University, November 1975.

3. "Determination of Flight Equipment Maintenance Costs of Commuter Airlines," NASA CR-152069, Summerfield Associates, November 1977.

4. "Advanced Cost Estimating and Synthesis Techniques for Avionics," NASA CR-2-461, General Research Corporation, September 1975.

5. "Future Equipment Maintenance Characteristics," ARINC Avionics Maintenance Conference, May 1977.

6. Carpenter-Huffman, Polly, and Bernard Rostker, "The Relevance of Training for Maintenance of Advanced Avionics," The Rand Corporation, R-1894-AF, December 1976.

7. Carpenter-Huffman, Polly, J.E. Neufer, and Bernard Rostker, "Analysis of the Content of Advanced Avionics Maintenance Jobs," The Rand Corporation, R-2017-AF, December 1976.

8. Mclver, D. W., Robinson, A. I., and H. L. Shulman, "A Proposed Strategy for Acquisition of Avionics Equip- ment," The Rand Corporation, R-1499-PR, December 1974.

9. "Proceedings of the Eighth Annual Department of Defense Cost Research Symposium," Compiled by Directorate of Cost Review and Analysis, Office of the Comptroller of the Navy, Washington D.C., November 1973.

10. Ratner, R. S., et al., "Design of Fault Tolerant Airborne Digital Computer," Vols. I and II, NASA CR-132253, Stanford Research Institute, October 1973.

11. Smith, T. B., et al., "A Fault-Tolerant Multiprocessor Architecture for Aircraft," Vols. I and II, The Charles Stack Draper Labs, Inc., April 1977.

12. Base, K. W., "Aviation Electronics," Sams publications, Third Edition, 1977.

13. Smyth, R. K., and G. L. Teper, "System Design for Preliminary Advanced Avionics Systems (PCAAS)," Systems Technology, Inc., April 1977.

-108- 14. McCalla, T. M. , et al., "Preliminary Advanced Avionics Systems for General Aviation," Southern Illinois University, June 1977.

15. "Avionics Equipment Cost-of-Ownership Seminar," Aeronautical Radio, Inc., April 28, 1976.

16. "Project Plan for the General Aviation Advanced Avionics System," National Aeronautics and Space Administration, June 1974.

17. "Aircraft Avionics (Digital Avionics Study Report), Vol. IV (Appendix C) Cost Trade Analysis," United States Air Force, April 1973.

18. "PRICE Methodology," RCA Government and Commercial Systems, January 1973.

19. "DAIS Briefing to Industry," United States Air Force, September 1976.

20. "On the Applicability of Integrated Circuit Technology, to General Aviation Orientation Estimation," Stanford University, September 1976.

21. Putnam, Lawrence H., "A Macro-Estimating Methodology for Software Development," IEEE Computer Society, September 1976.

22. "Hewlett-Packard Interface Bus, A Compendium of Technical Articles," Hewlett-Packard, Palo Alto California, undated.

23. "IEEE Standard Digital Interface for Programmable Instrumentation," The Institute of Electrical and Electronics Engineers, Inc., IEEE Std 488-1975, 1975.

24. "Vectors," Vol. XVIII, Hughes Aircraft Company, Fall 1976.

25. Ernstoff, M.N., "Liquid Crystal Pictoral Display," presented at the 1975 Digital Avionics Systems Conference, sponsored by AIAA, April 4, 1975.

26. Loughry, Donald C., "Interface Standard 488 in Action: Concepts and Capabilities," presented at Electro '76, Institute of Electrical and Electronics Engineers, Inc., May 1976.

27. Armstrong, Bruce E., "Avionics Data for Cost Estimating," The Rand Corporation, P-5745, November 1976.

-109- 28. Hoffman, William C., and Walter M. Hillister, "Forecast of the General Aviation Air Traffic Control Environment for the 1980's," NASA CR 137909, Aerospace Systems, Inc., Burlington, Mass., June 1976.

29. "An Initial Cost-by-Function Model for Avionic Computer Systems," NADC-SD-6923, Naval Air Development Center, March 1969.

30. "A Cost-by-Function Model for Avionic Computer Systems," NADC-SD-7088, Vol. I, Naval Air Development Center, March 1971.

31. "A Cost-by-Function Model for Avionic Computer Systems, (Model Validation and Data)," NADC-SD-7088, Vol. II, March 1971.

32. "Cost Analysis of Avionics Equipment," AFAL-TR-73-441, Vol. I, Air Force Systems Command, U.S. Air Force, February 1974.

33. "Life Cycle Cost Task Group of the Joint Services Data Exchange for Inertial Systems, Proceedings of Quarterly Meeting," Northrup Aviation Company, March 1977.

34. "Mathematical Methods of Resource Allocation for Optimizing Reliability," RADC-TR-76-38, Rome Air Development Center, AFSC, February 1976.

35. Davis, Ruth M., Evolution of Computers and Computing, SCIENCE, 18 March 1977.

36. "Sputtering Thin Films for Pressure Transducers," VACUUM TECHNOLOGY, April 1977.

37. Allan, Roger, "LEDs: Digital Domination," IEEE SPECTRUM, February 1977.

38. Jargen, R.K., "Competing Display Technologies Struggle for Superiority," IEEE SPECTRUM, October 1974.

39. Strandt, E.R., "Design of a Low Cost Compact Versatile Airborne Plasma Alphanumeric Control and Display Unit," Delco Electronics Division, General Motors Corp., Milwaukee, Wisconsin.

40. Elson, Benjamin M., "Fuel Saving Computer System Studies," AW&ST, March 7, 1977.

41. Klass, Philip J., "Chip Carriers Offer Microcircuit Gains," AW&ST, January 24, 1977.

-110- 42. Trifari, John, "Buss Standard Brings New Power to Benchtop Instrumentation," TEST AND MEASUREMENT, July 1976.

43. Larson, George C., "Way Out Wonders," FLYING, February 1977.

44. Klass, Philip J., "Swing Votes Key to Landing Aid Choice," AW&ST, February 21, 1977.

45. Lewis, Arnold, "Avionics - The Cost To Keep Them Working," BUSINESS AND COMMERCIAL AVIATION, July 1972.

46. Morozowich, Merle, "Are Serai-Conductors Reliable?" CIRCUIT COMPONENTS, July 1976.

47. Rice, L.R., "Circuit Reliability Is Not Semi-Conductor Reliability," Tech Tips, Westinghouse, Youngwood, Penn.

48. "Equipment Reliability Factors Studied," AW&ST, March 28, 1977.

49. Abbott, R., et al., "Get Ready for the New Generation of Dynamic RAMs," EVALUATION ENGINEERING, January/February 1977.

50. Elson, B.M., "Inexpensive Advanced Avionics Concepts," AW&ST, August 1, 1977.

51. Harding, William, "Microprocessors: Beyond the Price Tag," MACHINE DESIGN, May 12, 1977.

52. Belson, John, "The 21st Century Flight Deck," FLIGHT INTERNATIONAL, April 1977.

53. 1976 Avionics Bonus Supplement, AOPA PILOT, June 1976.

54. Klass, Philip J., "New Data Yields Clues to Reliability," AW&ST, February 13, 1967.

55. Weeghman, Richard, "Keep It Simple To Keep It On," FLYING, April 1975.

56. Bail, John W. , "General Aviation Avionics Reliability," SAE Business Aircraft Meeting, Wichita, Kansas, April 1968.

57. Gillette, Jim, "Reliability," Hewlett Packard Co., Doc.

No. 5950-3735, May 1977.

58. "Head-up Display Concept Under Development," Hughes Aircraft Company, AW&ST, 16 May 1977.

59. Grimm, Richard W. (AFAL), "Fire Control Radar and Airborne Computers Cost Prediction Based on Technical Parameters," Eighth Annual DOD Cost Research Symposium, November 1963.

-111-

60. Jackson, C.T. Jr., et al., "Flight Evaluation of Advanced Navigation Techniques for General Aviation Using Frequency Scanning," American Institute of Aeronautics and Astro- nautics Guidance and Control Specialist Conference, San Diego, California, August 1976.

61. Berkstresser, Betty K., "Advanced Computer Technology: An Aspect of the Terminal Configured Vehicle Program," AIAA Digital Avionics System Conference, Boston, Massachusetts, April 1975.

62. Blemel, Kenneth G., "The AFLC IDEALS Prototype," Proceed- ings of the Annual Symposium on Reliability and Maintain- ability, IEEE Publications, New York, 1975.

63. Feduccia, Anthony J., and Jerome Klion, "How Accurate Are Reliability Predictions?" Proceedings of the Annual Symposium on Reliability, IEEE, Cat. No. 68-C 33-R, 1968.

64. "Operational Influences on Reliability," Hughes Aircraft Company, RADC-TR-76-366, December 1976.

65. "Motor Vehicle Safety Defect Recall Campaign," Department of Transportation, January 1, 1973 to December 31, 1973.

66. Quality Management Engineering, September 1974, p. 31.

67. Madansky, A., and M. P. Peisakoff, "Thor and Atlas In- Flight Reliability Growth," The Rand Corporation, RM-2598, June I960.

68. Ashendorf, George, "Pitfalls in Reliability Predictions," Society of Automotive Engineers, Inc., Proceedings of the Sixth Reliability and Maintainability Conference, 1967.

69. Codier, Ernest 0., "Reliability Growth in Real Life," Proceedings of the Annual Symposium on Reliability, IEEE Cat. No. 68C 33R, 1968.

70. "Reliability Growth Study," Hughes Aircraft Company, RADC-TR-75-253, October 1975.

71. Pelehach, Michael, "The Influence of Avionics on Aircraft Design," Grumman Aerospace Corporation, 35th Flight Mechanics Panel of AGARD, Amsterdam, 2-5 September 1969.

72. Hauter, A. J., "Environmental Considerations for Avionics Installations," SAE Paper 770447, 1977.

73. Olcott, John W., "Inside Navcoms," (2 Parts), FLYING, June and August 1973.

-112- 74. Gates, H.P. Jr., et al., "Electronics-X, A Study of Military Electronics with Particular Reference to Cost and Reliability," Institute for Defense Analysis, NTIS No. AD-783007, January 1974.

75. "217B Predict System Reliability Prediction Computer Program," Naval Weapons Center, China Lake, California, Reg. 3686-127-77, March 1977. For a comparison of this method with Ref. 81, below, see comments by W. A. Marsh in IEEE TRANSACTIONS ON RELIABILITY, December 1977, p. 372.

76. Von Alven, W. H., "Reliability Engineering," ARINC Research Corp., Prentice Hall, 1965.

77. "Reliability Prediction of Electronic Equipment," Military Standardization Handbook 217B, Department of Defense, September 1974.

78. Kirkwood, R. L., and L. W. Sepme'yer, "The Cost of Unreli- ability of Air Force Airborne Electronic Equipment as Represented by the Cost of Maintenance," The Rand Corporation, RM-1002, December 1952.

79. Lloyd, David, and Myron Lipow, "Reliability: Management, Methods and Mathematics," Second Edition, Lloyd and Lipow Publishers, Redondo Beach, California, 1977.

80. Nowlan, F.S., "Review of On-Condition Philosophies from a Planning and Operational Viewpoint, Past-Present-Future," Proceedings of the 15th Meeting of International Air Transport Association (IATA), Belgrade, Yugoslavia, October 7-9, 1969.

81. Gaertner, W. W., D. S. Elders, D. B. Ellingham, J. A.

Kasting, W. M. Schreyer, "Automatic Electronic Component Failure Rate Prediction with MIL-HDBK-217B," IEEE TRANSACTIONS ON RELIABILITY, August 1977, pp. 214-219.

82. Coppola, Anthony, and John E. Daveau, "Reliability and Maintainability Case Histories," ANNALS OF RELIABILITY AND MAINTAINABILITY-1967, pp. 582-586.

83. Drezner, S.M., et al, "The Computer Resources Management Study," R-1855/1-PR, The Rand Corporation, Santa Monica CA, April 1976.

84. Daly, E.B., "Management of Software Development," IEEE Transaction on Software Engineering, May 1977.

-113- 85. Morgan, D.E., and D.J. Taylor, "A Survey of Methods for Achieving Reliable Software," COMPUTER, February 1977.

86. Boehm, B.W., "Software Engineering," TRW-SS-76-08, TRW Systems, Redondo Beach CA, October 1976.

87. Dreyfus, J.M., and P.J. Karacsony, "The Preliminary Design as a Key to Successful Software Development," TRW-SS-76-09, TRW Systems, Redondo Beach CA, October 1976.

88. Hecht, et al, "Reliability Measurement During Software Development," NASA Contractor Report 145205, 1977.

89. Williams, R.D., "Managing the Development of Reliable Software, TRW-SS-75-06, TRW Systems, Redondo Beach CA, April 1975.

90. Thayer, T.A., M. Lipow, and E.G. Nelson, "Software Reli- ability Study," TRW Systems Report to Rome Air Development Center, Contract F3060274-C-006, March 1976.

91. Weinberg, G.M., "The Psychology of Computer Programming," Van Nostrand Reinhold Company, New York, 1971.