During autumn 2022, I had the chance to attend the great FlowCon conference hosted in Paris, France, whose core topic targets stream delivery in software engineering. Lean, Agility and Continuous delivery are thus highly represented in the conference. One of the talks caught my attention, and I was far from disappointed when I saw it; The talk was realized by Woody Rousseau and Flavian Hautbois (The replay is in French here, and there’s a talk in English on the same topic made at the Craft Conference 2022). I can translate its title to “Radical Quality – from Toyota to IT.” In this post, I share some insights from this talk, especially the Dantotsu method, which was new to me.
The talk relates the story of Sadao Nomura, who worked at Toyota between 2006 and 2015 to solve a pain point on quality issues. Before he joined Toyota, the company had a non-satisfying number of internal defects in its factory chains. Because of these bugs, it was common that the requirements to ship cars outside the factory weren’t met. Toyota really wanted to work on that to make sure the expected levels of quality were satisfied more frequently. And so Sadao Nomura came in.
Sadao Nomura aimed to generate a 50% reduction in the number of defects on the factory chains for three years (meaning an overall ~88% of the global reduction for the full period). He told the full story in his book The Toyota Way of Dantotsu Radical Quality Improvement.
Dantotsu means “Better than the best.” This radical approach challenges what companies usually settle to tackle quality issues. What are these ideas?
Why use the radical term when referring to Dantotsu? Because the philosophy isn’t about quickly fixing a defect, it’s also about deeply understanding why this defect happened, what’s been done to resolve it, and how to avoid this occurring again in the future. These are the core principles of Dantotsu.
In the book, Nomura explains different aspects of the approach, particularly the importance of visual management and the training programs (Dojos), where workers can practice the right gestures to avoid defects. He also provides an interesting classification of the defects. In Tech, we often use the priority (Low/Medium/High…). He suggests using the four following types of bugs:
Now, what happens when a defect is identified? In the context of Toyota, when a single unit has an issue:
(The process has 8 steps for the record, which I’ve summarized for simplicity).
And all this process happens in 24 hours. Radical right? Speed is key. When we see a defect, don’t just fix it. We don’t want to see him again. This approach turned out to be successful in practice at Toyota.
The second part of the talk was about how the Dantotsu method was implemented in two tech companies. They recall that, in software engineering, there’s still work to do to convince people that doing things right costs less than non-quality (surprising, right?). If you’re skeptical about this statement, we recommend you to read The Economics of Software Quality which precisely demonstrates the cost of bad software quality. The Accelerate book closes the discussion on that question if it’s still needed.
The two speakers in the talk explain that to reach the zero-defect (utopian) ambition, the key part is to train developers to produce source code with no defects. They also transpose the bug categorization into the software engineering world, so, for instance, Type A is a defect caught by developers locally or during the continuous integration step. In their respective company, they’ve both implemented a new process where defects are documented, especially the investigation and the countermeasure. Here’s a capture of one slide that encompasses all the information:
Slide extracted from the conference.
Here, we can see the deep level of details where they go to document what’s been wrong, and indicate that the countermeasure was to consolidate the tests. (Disclaimer: not surprisingly, consolidating the tests is a common countermeasure). NB: the two quoted companies in the conference used wiki-like systems to document these defects.
Also, in their companies, they’ve set regular meetings dedicated to bugs. In these sessions, someone can present to the other participants a bug report and goes deep into the root cause and how to prevent it again. Similar to what happened in the Toyota factories, someone is showing the right gestures to avoid a problem happening again. If the sessions can be opened to anyone, it can also gather the Tech leaders, who will ensure the message is broadcasted to their respective teams.
What were their conclusions after the implementation of this methodology? I won’t mention all the details, but here are some interesting insights:
So yes, the Dantotsu philosophy can be transferred to the Software industry. It’s easier to fix a defect in our code when others developers have met it before. The overall method won’t be smooth and easy initially, but it’s worth a shot!
We saw that when a bug is found in the source code and fixed, this should be turned into shared knowledge for the team (so that it won’t happen again, remember?). That’s what’s shown in the screenshot above. An easy way to perform this operation is to use the Promyze platform, which aims to ease best practices sharing in engineering teams with deep integration in developers’ tools (IDE & Code Reviews). Developers are keen on sharing best practices in many situations, such as performance improvement, security fixes, better readability, or bug fix.
Let’s see an example of how to proceed with a concrete example of a defect fixed in late 2022 internally. In our staging environment, before a final release, we spotted an error when our application started and performed patches on the MongoDB database. In our code, a method from the Mongoose framework was called to regenerate the indexes of a collection. We observed runtime issues and found this method should rather not be called in our context (this last point is an important precision; I’m not saying you should never call this method, but in short, it wasn’t compatible with our system).
Let’s see how we propose to adapt the Dantotsu process to Promyze.
The defect has been spotted in the code, and we removed the call to reIndex(). Before that, we created a best practice from VSCode using the Promyze extension: Selection of code → Right Click → Promyze → Identify a new best practice. Just put a name, and this is sent to the Promyze application.
Then, with Promyze, our team (a project team, feature team, or even a community of practice) runs regular workshops to review best practices proposals. Each contributor explains the intent behind the suggested practice. In our case, Arthur explained to the whole team the issue met with the reIndex function call, and everyone could indicate if this was clear to them.
Once the best practice has been validated, it can be shared with others teams in the organization, or within a community of practice. The knowledge is spread beyond teams.
Once a best practice has been validated, you can set regular expressions that match the pattern you want to avoid to push notifications in developers’ IDE and Code reviews (a linter-like behavior). This will be possible if your pattern can be matched by the syntax of the code. It will reduce the risk that this method will be used in the future.
This is what it looks like when configuring a regex in Promyze, and then getting suggestions in VSCode:
When new developers join your organization, they need to get familiar with your best coding practices, and also the patterns to avoid defects. In Promyze, there is a feature called onboarding workshop for new hires, where they can train on your knowledge base. As each best practice has associated code files, each recruit will be invited to locate the area where the best practice has been followed (or, inversely, not applied). In the example below, they’ll need to highlight line number 6:
What happened here is that we caught a bug and fixed it, and in addition, we capitalized on our work and generated a new best coding practice from it. We tried to follow the Dantotsu philosophy!
If you have never heard about it before, I hope this short introduction helped you to get a better overview of Dantotsu. I’d be happy to have your feedback. Have you implemented the Dantotsu in your context? Please share!
Promyze, the collaborative platform dedicated to improve developers’ skills through best practices sharing and definition.
Crafted from Bordeaux, France.