Student Feedback - Chapter 11

[jasonjabbour]

Chapter 11 - Benchmarking AI

• We felt like we walked away with all learning objectives except for the seventh.
• This chapter was interesting because it tried to broach a large topic (benchmarking) by tackling specific cases. While this was helpful, it resulted in a lot of the text being repetitive, as the same ideas were being re-introduced with only slightly different contexts. Also, benchmarks have been mentioned/defined several times up until this point, normally to help explain benchmarks for the topic of each chapter. Because of these two things, we feel like this chapter might be unnecessary. Rather, you can build up the explanation of benchmarks in each chapter, giving relevant information on benchmarks while still sticking to the topic of the chapter. Readers would still walk away with understanding of benchmarks in specific capacities, altogether giving them a general idea of what benchmarks are.
• Some repetition in explanation that we noticed were across 11.4.4 Training Benchmarks, Metrics section and 11.4.5 Inference Benchmarks, Metrics section. These ideas were generally the same, and explanations of the each metric in both places did not feel necessary. Also, in 11.4.4 Training Benchmarks, MLPerf Training Benchmark is explained twice (and each explanation has a paragraph dedicated to it).
• This chapter was also long; a potential place to cut down on text could be 11.5.1 Historical Context. The background information is nice, but not strictly necessary.
• We didn't understand Figure 11.5 (sorry!)
• The 11.5.3 Lessons Learned section felt like more of a conclusion---perhaps it should go towards the end of the chapter, rather than the middle.
• Figure 11.7 is right under the 11.7 The Trifecta heading, but does not demonstrate the trifecta. Perhaps switch out this image with something more relevant?
  Machine Learning Systems - 11 Benchmarking AI.pdf

Originally posted by @sgiannuzzi39 in #256 (comment)