Trade studies help us make wise decisions

A trade study is a procedure that provides a framework for making wise decisions.

To make any sort of wise decision you must:

• Consider the point of view of everyone affected by the decision. Understand what they need and what they want. Determine what is most important to them. In a trade study, you distill this information into a set of criteria to consider when making the decision.
• Think through the relative importance of each of the criteria. In a trade study, you assign a weight to each criterium that reflects its importance: more important criteria receive a larger weight than less important criteria.
• Think of several possible solutions. Describe each solution in detail. In a trade study, you rate each solution on how well it meets each of the criteria and then calculate a score for each solution.

The more thought you put into each of these steps, the more likely you will come up with a wise decision.

Systems Engineers use trade studies to evaluate proposed designs and select the best one. But trade studies can be useful whenever it is important to make a wise decision and difficult to undo a poor decision.

A trade study is documented in a table

To prepare a trade study table, list the proposed solutions in the columns.  List the decision criteria in the rows. Add a column to display the weight of each of the criteria. Add another column for notes.

After the table is prepared, add a rating for how well each proposed solution satisfies each of the criteria.  Ratings can be displayed in many ways: one to five stars, one to three stars, a number between 1 and 10, or a percentage are common ways to display a rating. Add a note that explains the reason for the rating if the reason is not obvious.

After you have filled in all of the ratings, calculate a score for each proposed solution. To calculate a score, multiply each rating by the weight assigned to that criterium and then sum up all the weighted ratings. Display the score in the bottom row.

For a more detailed overview of trade studies, see Survey of Trade Study Methods posted on the NASA website.

Trade studies are subjective 

Selecting the criteria is subjective.  Everybody with an interest in the decision will have a different opinion about what criteria are important and what ones aren’t. What matters to some people doesn’t matter to others.  Criteria can conflict with each other or even be mutually exclusive. The criteria chosen will affect the decision.

Weighting the criteria is subjective.  Some criteria are more important to some users than to other users. Changing the weight of a criterion to make it more (or less) important may cause a different proposed solution to be selected.

Assigning ratings is a subjective process.  Even criteria that appear objective are subjective on closer examination.  For example, when evaluating different vehicles, one criterion might be the capacity of the gas tank.  While the capacity is an objective measurable number, the meaning of the rating is subjective.  Is a 15-gallon tank excellent, good, fair, poor or terrible?  It depends on the vehicle and the needs of the driver.  So yes, ratings are subjective.

Subjectivity can be managed 

Subjectivity makes engineers uncomfortable. We want to judge which solution is best overall based on objective evidence; however the essence of good judgement is the interpretation of objective evidence within an appropriate context.

Managing subjectivity is a group activity. Back-and-forth discussion allows everyone to explain their opinions and maybe modify them upon hearing what other people think. The group can then work individual opinions into a decision that avoids individual biases. These group discussions are often fun and invigorating.

At least, this is what happens if the discussion is led by a skilled moderator and everyone is acting in good faith. An unskilled moderator can fail to keep the group focused and the discussion can become tedious and unproductive. A participant that is acting in bad faith will try to game the process by manipulating the criteria and weights so that the solution they prefer comes out on top. In a well-led group, other people quickly call out this sort of manipulation. 

Another way to manage the subjectivity is to use a simple rating system.  It takes much less time for a group to choose between a rating of 2 vs 3 stars than to choose between a rating of 64% vs 65%.  

Design synthesis 

What if one proposed solution meets several criteria especially well and another proposed solution meets a different set of criteria especially well?  Is is possible to combine (synthesize) the two proposals into a new proposed solution that would meet both sets of criteria well?  This process is called design synthesis. It is one of the most important skills for a systems engineer to cultivate.  Trade studies help identify solutions that could be synthesized into something better. 

Trade studies have many virtues 

Trade studies are simple and intuitive.  Nearly everyone, regardless of technical background, understands trade studies. They are an excellent way to explain design decisions to non-technical people. 

A trade study guides the decision making process in ways that make it more likely to come up with a wise solution. It encourages the consideration of many points of view. It emphasizes the distillation of these points of view into a set of criteria by which to judge proposed solutions. And it requires the careful description of several potential solutions.

EXAMPLES

Example #1: Mom Loves the ER

Let’s take a look at the trade study documented in the post Mom Loves the ER: 

This trade study has some good points: 

• The use case is clearly defined.   
• The 5-star rating system is simple. 
• Notes explain why the ratings were given. 

It also has some weak points: 

• I was the only person that participated in the evaluation.  Other points of view would have added to the depth of the analysis. 
• I did not assign weights to the criteria.  Clearly a quick and accurate diagnosis is more important than other criteria. 
• I gamed the criteria to favor the ER solution.  I don’t value the ability to select a specific doctor so I left it out. However, if for example you don’t speak English well you might want a doctor that speaks the language you are more comfortable with. Or if you have an unusual medical condition you might want to see a doctor that has treated you before. 

This example is very simple and somewhat flawed, but even so it can be useful.  If you are interested in improving the ability of doctors to diagnose unknown ailments, it hints at several issues to address: 

• Can we make it easier for doctors to collaborate with each other? 
• How can we encourage collaboration between doctors? 
• Can we integrate our electronic medical records systems?
• Can we reduce the time it takes to get an appointment? 

Example #2: Sick Child at Night

What about this situation? You are a parent with a seven year old child who wakes up in the middle of the night and is sick. Your insurance covers telehealth visits. And like nearly everyone else your age, you are comfortable with technology. Now the trade study might look like this:

In this situation, telehealth clearly beats out taking your child to an urgent care clinic. And if your child’s condition doesn’t seem life-threatening, telehealth would be a better choice than the ER.

When the company I worked for first offered telehealth back in 2017 or so, they presented it to employees as a way of saving the company money. Given that saving the company money isn’t even listed as one of the criteria that a worried parent might use to decide how to get care for a sick child, it is unsurprising that telehealth wasn’t used much. If the person charged with promoting telehealth to employees had done a trade study, they might have chosen to emphasize convenience and 24/7 availability instead!

Example #3: Criteria change!

In 2020, the pandemic changed everything, especially in healthcare. A revisited trade study would include an important new criterium: Limiting exposure to Covid 19.

With this added criterium, telehealth becomes even more attractive.

Example #4: Consumer Reports

The magazine Consumer Reports is doing a type of trade study when it evaluates products. It rates a selection of competing products according to a set of criteria and calculates a score for each product. But readers can weigh the criteria differently according to their own needs and wants and make an informed decision as to which product is best for them.

In the example below, persons A and B are both looking for a new car. Person A is looking for a car with great performance. Person B is looking for a very reliable car. They weight the criteria differently according to what they value in a car and come up with different recommendations.

Source: Consumer Reports April 2021 and September 2021.