Threat modeling is time-consuming, especially the first iteration of a threat model, tied into the first time the team has created a threat model. In the development world that time equates to costs in which there is no direct return on investment. Developers are under tight timelines to get software out the door and many teams are not given the time to include threat modeling. The time factor all depends on your experience, size of the application, development process and it will get quicker as you start doing the threat modeling on a regular basis. The cost of doing the threat modeling, and adhering to the mitigations during development, will be less than the time spent going back and fixing the vulnerabilities later.
2. What are the keys to building a good threat model? What tools should you use, what practices should you follow? How do you know when you have built a good threat model - when are you done?
Motivation is the biggest key to building a good threat model. If you are forced into threat modeling you are not going to take the time to do a good job. In addition, an understanding of what you want to accomplish with your threat model. Some threat models use data flow diagrams, while others do not. You need to identify what works for your situation. As long as the threat model is helping identify the potential risk areas, threats, and mitigations then it is doing its job. It is important to set aside some time for threat modeling to focus on it. Many times, I see it is something that is a task that is buried with other tasks handed off to one developer. This should be a collaborative effort with a specific focus.
There are many different tools available to help with threat modeling. Microsoft has the SDL Threat Modeling Tool which is a great tool to use if you don't have any threat modeling experience. Microsoft also has the Escalation of Privileges card game which is meant to help developers become familiar with threat modeling in a fun way. You could just use Microsoft Excel, or some other spreadsheet software. It is all up to what works for you to get the results that you need.
Threat models are living documents. They should not really ever be finished because each update to the application could affect the threat model. You can usually tell if you are done with the current threat model when you are out of threats and mitigations. Think of making popcorn, when the time between pops gets longer, the popcorn is probably done. The same thing with threat modeling, when you haven't identified a threat or mitigation in a while, it is probably finished.
3. Where should you start with threat modeling? Where's your best pay-back, are there any quick wins?
If your company has a bunch of applications that are somewhat similar, you can create a template for your company threat models that can help fill in some generic stuff. For example, protecting data between the client-server, or server-server. This should be in all of your apps, so you can have a base template where you don't need to do all the legwork, but just verify it is correct for the current application. This can help increase the speed of threat modeling going forward.