Who is familiar with Toyota's challenges back in 2013 with unintended acceleration event. Guys are too young for that. That's all right.' We're going to talk about it. So, in Toyota's vehicles, they had an ECM, the engine control model. The company experienced a number of their cars within what they deemed as unintentional acceleration events resulting in the deaths of 89 people. This was a big deal, it was in the news every night or at least once a week, back then. I remember Toyota representatives denying, there's no problem with our car, everything's fine, we're not sure what's happening, but there's no design issue. Well, when everything was said and done, the company ended up paying $1.1 billion, warranty and recall costs. To make matters worse. It paid, I don't know what they paid elsewhere around the world but in the US, they paid $1.2 billion in fines. They were out $2.3 billion. Painful, very costly. The bar group performed an investigation and Michael Barr was the primary expert witness for the plaintiffs, there were lawsuits and all the stuff that ensued. He did a tear-down analysis, but a very detailed tear-down analysis. It does assembled all the code, looked at all the code and made a professional as an expert witness, he made a professional call and subsequently has written up a summary, this information is available now. So their report illustrates a cautionary tale to all involved in safety critical systems. So, this is just a summary. So, the software is defective and contain bugs that can cause an annex unintentional acceleration event. The code quality metrics predict that the presence of additional bugs would be in there. So, they took the code and disassembled it and then they ran it through code quality tools, and these tools predicted that there were additional bugs in the code. He coined Toyota's sale phase measures as defective and inadequate calling them 'a house of cards' safety architecture which is pretty scathing assessment. Yeah, nobody wants to hear their product as a house of cards because the house of cards fall down easy or hard. On the hardware side, Toyota claimed that the CPU memory had error detecting and correcting codes commonly known as EDAC, it didn't. Oops! That was on the hardware side. On the software side, there's a whole long litany of things, but these are some of the issues that Barr group found. Mirroring was not always done of storing multiple copies of critical variables, this is a common thing in safety critical systems that you have, you build in a lot of redundancy and you don't make decisions unless multiple copies of certain critical variables say that," Oh, yes. These are all in agreement". If you just got one copy of a variable, and alarm gets corrupt, your if statement, you might not want to take that branch or you might want to take that branch. So, mirroring was not always done, was done sometimes but not always. Stack overflow. So, we know that that is? Stack overflow? So, the stack grows down in memory, in the heap, and then application does it allocates memory grows up, and if they cross, stack overflow. So, the stack just overwrote some critical variables or potentially instructions. Toyota claimed that only 41 percent of the heap was used, but the Barr investigation showed that it was more like 94 percent which is getting dangerously close to overflowing. The hardware had no memory protection unit also. So, there was another failure if you will, on the hardware side to guard against the Stack Overflow. So, the MISRA-C rule violating recursion was found, so the MISRA as the Motor Industry Software Reliability Association, they publish standards that for C code, whole bunch of rules that are supposed to be followed for all in the automotive industry and apparently they broke a bunch of rules, one of which was the recursion. Whatever that rule is, I didn't go look at the details, but they found. They were clearly able to identify that this recursion rule was being broken. Here's a quote," So, although Toyota had performance stacked analysis of our concluded that the automaker had completely botched it, Toyota missed some of the calls made via pointers. They missed a stack usage by library and assembly functions that were linked in, about 350 in total. They missed the RTOS use during task switching and they also failed to perform any runtime stack checking for Stack overflow". It can't be done in software, you just have a timer that periodically goes and looks where at the top of the heap is and where the stack is. That's not a 100 percent guaranteed because it can go like this and between the timer, and you had Stack Overflow and you don't detect. It would be much better to have a memory protection unit or hardware monitoring that than rely on software. Then, they had a litany of other faults were found in the code including buffer overflow, unsafe casting and race conditions between tasks. So, lessons are to be learned there. Tells ever the easy one, they should follow the MSIRA coding guidelines, that'd be a big good for start. What else could they have done? Code validators. Are you sure? Extensive coverage or writing tests having separate extensive coverage in depth in your code basically. Yeah. Run code coverage analysis tool. What about a rigorous design process? Where the design is done first, and the code is written second. If there's one thing you take away from this class. Take away that one way. All right. I am not kidding. Do the design first and do the coding second, and life is just so much better. You don't end up paying $2.3 billion in damages. So, I think they had a number of issues part of which we probably process, some of which might have been cultured under the gun to get it done. Get it out the door. Oh! We don't need to do this, reviews somebody who was, there was a manager probably that got fired as a result of this or many people were probably let go. Design reviews are very useful to get everyone in a room and step through the logic of a piece of code, and tie a poke holes in it. You can find all kinds of bugs that way. I am a huge believer in designer reviews. You're not going to find everything. She can find a lot, and then the coder didn't think about this corner case or that corner case or what happens when this happens and that happens. A lot of bugs are temporal in nature and they're very hard to find. " What happens if this happens and this happens and this happens and this happens, oh, I never thought of that. I have to cover that case in my code", that kind of thinking come up in a code review, and you want sharp people in those code reviews. Review your code or review another person's code but get it like a game. That's how I, I used to be nervous about code reviews and didn't like them very much because I felt somehow like my honor was being tarnished because, here's my RTL code, Verilog for this module. I very quickly came to see the benefit of having an independent third person looking at my code because they found bugs. Why are we there? To build the best product we can. We don't want to build products with bugs. So, set your pride aside and encourage the review process. Engage it and embrace it. Challenge that your audience to find as many bugs in your code as they can. Tell them you'll buy everyone dinner that finds a bug in your code or something. I mean you can be creative like that, right? They had some serious systemic issues at Toyota in my opinion
