Subscribe
Posted on August 15th, 2010
Oh the joys of not being in the grind. From this position I get to see all sorts of stuff about doing “Agile” right. Primarily from a Project Management perspective. But, I’ve got a secret. It’s one of them there plain-sight secrets that turns out to be how successful teams get things done. To us plain language folk, we call it “Talking to each other”. But, since that will never work in consultant speak, we’ve got one of them there fancyish names for it. Wanna hear it?
Explicit Policies
Wooo Doggie! Doesn’t it sound grand?! But those two words, that’s the secret to making your team agile! How do I know? It’s right in front of our noses.
Let’s look at Scrum. What is Scrum? Well, we stick all our work in a Product Backlog. The Product Owner manages it. The team meets every sprint (usually 2 weeks) to take the next highest items off the backlog. We meet every day to answer three questions. We track our work in a burndown chart. We demo what we’ve completed at the end of the sprint. We reflect on what we’ve learned before our next planning meeting.
Do you do those? This isn’t one of those trick questions. What I mean is that, if your team adopted Scrum, it’s because you all looked at a book – or online – saw that as a blueprint, talked to each other and agreed to do that stuff. In other words, your team has an explicit policy to do those things. What things?
- We agree work is managed by a role called a Product Owner
- We agree to meet every two weeks to determine what the next highest priority items to be worked on are
- We agree to demo every thing we’ve completed every two weeks to the customer
- We agree to meet every day to get a sense of where we are and where we are going
- We agree to use a burndown chart to estimate out if we’ll be done by the end of the sprint
Those, folks, are explicit policies. You’ve agreed to them as a team. Turns out, explicit policies are a key part of Kanban as well:
- We agree that we will measure the cycle time of work through the system
- We agree to limit the work-in-progress at any one time
- We agree to meet every week to let the business reprioritize and replenish the backlog queue
- We agree to demo as much work as we’ve completed every two weeks
- We agree to meet every day as a team
And then some team-specific ones:
- We agree that when an item is blocked we can go one over our WIP limit for that queue
- We agree to swarm on test items when the WIP limit has been reached
- We agree to write acceptance tests before writing code to increase flow
- We agree that bugs can be fasttracked through the system, and can violate existing WIP limits by 2
Explicit Policies – the agreements of the team are actually what makes any agile method successful. Yes, Lean thinking helps. Yes, XP practices are vital. But the team owns the process. And if you aren’t talking as a team – and yes, as an organization – it ain’t gonna work. Like the one time by cousin Billy and my other cousin Billy were wiring up a moonshine still and Billy said he was gonna pour the grain alcohol and the other Billy lit a match so they could see in the woods. It was fun for a brief second, but then you end up without any eyebrows and people just know.
So there, now you know. You want to know if your manager can add stuff mid-sprint? Talk about it. Want to know what happens if you test-last? Talk about it. Try it out. Run experiments, and know what results you are looking for. It is the heart of Individuals and Interactions over Processes and Tools.
And if you see your Aunt – I was never here.
Posted on August 14th, 2010
This past week I’ve been attending the Agile 2010 conference here in Orlando, FL. For those who don’t know, the Agile 20xx series of conferences came because of the combination of two separate conferences in 2005 – XPUniverse and Agile Development Conference. Since that time, the conference has grown quite a bit – this year we had 1400 attendees, over 1000 submissions, and around 200 sessions. But…something felt like it was missing:
#agile2010 Programmers started the agile movement to get closer to customers not project mangers. – @unclebobmartin
< 10% of the talks at #agile2010 are about programming. Is programming really < 10% of Agile? – @unclebobmartin
@unclebobmartin …the conference was stolen a very long time ago. you were busy keynoting, i guess. #agile2010 – @GeePawHill
But it wasn’t just the conference attendees. I was the stage producer for the Team-Room Agile stage, and my co-producer Corey Haines and I talked a bit about the amount of technical / development content in the program. Mike on the blog A Software Craft said it very well with this blog post:
What do I see dominating the Agile2010 program? Lots of good sessions on learning, communicating, coaching, creativity, adoption strategies. Sessions that I would probably come out of saying "interesting" and "thought-provoking". But ultimately, sessions that were easy to digest. Not sessions where I encountered concepts that are fundamentally difficult and require hard work to master. Not sessions where I struggled to write good tests and emerged with a determination to rework and discuss the examples over the coming weeks until I finally felt I understood it.
Perhaps I’m a minority. @HackerChick tweeted about a tutorial on TDD where only a quarter of the attendees showed up with laptops, prepared to get their hands dirty. Perhaps Agile2010 isn’t the conference for me. A conference where the technical track is only 1/15 of the program. And the technical track includes sessions on "The Butterfly Effect" and how to "Walk and Code". But I worry about another crop of agile converts, filled with all the soft skills and strategies they need to succeed, headed for failure because they don’t know about the hard work and dedication needed to build that essential ingredient: the agile developer.
That second paragraph says it all. The Monday tutorial he’s mentioning was Brad Wilson’s session called Evolutionary development for the web with ASP.NET MVC and TDD. Even though it was said it was hands on, and that it would be using Visual Studio, only around 25% of the attendees brought a laptop. And in Uncle Bob Martin’s Clojure session, I was surprised at how few people had their laptops.
So when Corey said to me, “Two Words: XPUniverse 2011”, I thought it was funny. And then I realized it was right. We have a niche inbetween the Agile 20xx series of conferences at the Software Craftsmanship conferences such as SCNA. A conference focused back on the delivery of software – the XP Practices and Principles. What are they? Here’s the practices from Ron Jeffries’ site. Whole Team. Customer Tests. Planning Game. Small Releases. Collective Ownership. Coding Standards. Continuous Integration. Metaphor. Sustainable Pace. Refactoring. Simple Design. Pair Programming. Refactoring. Test-Driven Development. The hands-on practices that enable us to deliver software which provides incremental value in an iterative fashion. And a hands-on conference where you are expected to have your laptop, where you will be taking back practical exercises and practices to your development team. A conference focused on the developers, testers and the customers they love and who love them. A conference focusing on the fundamental principles of XP: Rapid Feedback, Assume Simplicity, Incremental Change, Embracing Change, Quality Work. And a conference small enough (~300 people) to be able to connect, explore and learn.
Over the next few weeks we’ll have much more information. We know it’s going to be in the US, likely in Chicago given its central nature and good base of people (though we’ve considered Portland as well). We know it’s going to be in 2011, and we know that it will be Corey Haines and myself organizing. Outside of that, we’re going to be working to bring you even more information.
For now, please be sure to follow @XPUniverse2011 for the latest information. We’ll be adding a website and more information as it becomes available. And if you need any specific information or have questions, please feel free to leave a comment below, or email me at foyc at cory foy dot com. Otherwise, we hope to see you soon!
Posted on August 12th, 2010
This afternoon at Agile 2010 I ran a session called Technically Distributed: Tips and Techniques for Distributed Teams (slides are at the end of this post). While I gave information about some of the tools I’ve used for teams, I think the key revelations came from two exercises we did.
In the first, the teams had to create a picture. Their “product owner” had the picture, but wasn’t allowed to show it to the teams. This was to simulate that most product owners have a great vision in their head of what they want – the tricky part is communicating that to the team. We ran the session with a 5 minute backlog planning session, followed by two iterations of 2 minutes of planning followed by 10 minutes of working. Half of the team had a collocated product owner, while the others had to send a “Business Analyst” to the front of the room to ask the PO questions when they had them. At the second sprint I swapped so the collocated teams were now distributed, and vice versa.
What was interesting were the insights the teams uncovered. The biggest frustrations from the teams were having to wait for the BA to get more information, and several tried just sending everyone up. On the Product Owner side, the biggest frustration was how to communicate the vision – there was lots of spatial information to communicate, and strange objects. But by the end everyone had finished (and come pretty close to!) the picture. We identified three key areas that attributed to them being able to do that:
- Prototyping. Words alone were not good enough to communicate the vision. It was only through rapid prototyping and iterations that the teams were able to move towards a solution
- Framing the vision. On the PO side, they weren’t allowed to just draw the picture, so they had to find creative ways to express their vision. During the retrospective between the two sprints, I recommended to the POs that instead of doing something to the team, that they do something to their vision to communicate it to the team. I then took a sheet of paper and folded it into quadrants – simulating a common language for the teams to talk to the PO
- Collocation. Even for the distributed teams, the Product Owner was “onsite” during the planning period. The teams took heavy advantage of this to do prototyping to understand the vision.
After the exercise was complete, we played a round of Speed Boat. I asked people to write down the anchors that held them back as distributed teams. As you can see from the picture, we had quite a few anchors. We didn’t have a chance to group them during the session, but I took them back to my room and found out that the spread across categories wasn’t quite what I was expecting. We had five main categories – Trust and Team, Communication, Time Zone, Org Issues and Tech Issues. Maybe it shouldn’t have been so surprising that very few of the issues were technical in nature – reinforcing that tools are not going to solve the distributed team problem. It ended up like this:
Another interesting aspect was that there weren’t a lot of duplicates. Here are the cards, with duplicates removed:
- Organizational Issues
- Lack of a company wide commitment to agile and education
- Travel budget
- Length of iterations
- Lack of shared space for the team
- Cost savings in offshoring (using junior developers, no travel budget)
- Not enough ‘gas’ (vision/agile) to get to the ‘island’ (goal)
- Technology Issues
- Availability of videoconferencing units
- Not Visual
- Lack of tools for efficiency
- Latency
- Lack of “remote whiteboard”
- Low fidelity communications
- Not enough bandwidth
- Network Connectivity
- Frustration with tools
- Time Zone
- Time Zone Differences
- Delayed Communications
- One location bears the burden of timezone differences
- Difficult to schedule meetings all can attend
- If someone doesn’t respond, you lose a whole day
- Delivery team doesn’t have access to stakeholders due to different timezone
- Trust and Team
- Other people’s problems not my problem – made worse by lack of contact
- Product Owners uncomfortable working with offshore resources
- No common language between product owner and the dev team
- Lack of Trust / No Trust
- Trust building difficult (don’t know what others are doing)
- Visa process for various countries
- Feeling Invisible
- Lack of knowledge sharing
- Developers isolated from BAs and testers
- Lack of dedicated team members
- Professionalism / Proactivity hindered by “invisibility”
- Too many chiefs
- Product Owner Availability
- Language Barriers
- Culture Clash
- Too many unrelated staff at the table
- Communication
- Communication
- Unclear expectations
- Vision, Business Knowledge, how what they’re doing is (or is not) adding business value
- Unclear / Complex requirements/stories – epic sizes
- Change in Priority
- No Priority
- Weak Information Radiators
- Poor “bandwidth” of written word
- Relying primarily on written requirements
- Lack of business knowledge
- Team vision not distributed
- No sense of urgency to get things done / delays are OK
- Fallback reliance on more extensive written documentation rather than invest effort and money to achieve more direct team communication
- Conversation takes more energy
- Email instead of phone
- Keeping focus during virtual meetings
I had a great time, and most of the feedback seemed to echo that the participants did as well. I promised that I would list out the tools I mentioned or use, so here they are:
- Wacom Bamboo tablet – This awesome device costs about $60 and plugs right into your USB port. I use it when I’m doing remote coaching by just opening Microsoft Paint and sharing it using LiveMeeting or GoToMeeting. It’s not as smooth as a whiteboard, but it is a great help
- Google Docs – Google Docs has a decent Word Processor and Spreadsheet that work well for collaboration, but they’ve recently added a drawing capability. It’s pretty close to a digital whiteboard, and I find it works better that the whiteboards from the meeting services like LiveMeeting.
- PlanningPoker.com – Mike Cohn has a great resource for teams wanting to do Planning Poker online. Within about 5 minutes you can have a game setup for collaborative play with Planning Poker.
- Innovation Games ® Online – Luke Hohmann’s awesome Innovation Games ® have an online piece which can be very helpful for product planning and prioritization. In fact I did a session at XP2010 on using the games for planning if you want more information.
- FitNesse – FitNesse is a open-source framework for acceptance testing using a Wiki. It is very powerful when your Product Owner is not collocated with the team.
- ProjectCards – ProjectCards is a great tool for distributed teams. It’s client/server, but the real time updates are just so addictive.
Thanks to everyone who came out! And if you have any other tools or tricks, please feel free to leave them in the comments below.
Technically Distributed – Tools and Techniques for Distributed Teams
View more presentations from Cory Foy.
Posted on June 22nd, 2010
Are you interested in learning about Innovation Games? This Friday, June 25th, 2010, I’m going to be facilitating a free game of Prune the Product Tree for the first 8 people to sign up at the registration page. If you’ve been interested in Innovation Games, or want to see what it is all about, this is a great way to do it!
Hope to see you on the list Friday!
Posted on June 15th, 2010
At XP2010 I gave a workshop on using Innovation Games to do product planning and prioritization. While I was able to post the pictures pretty quickly, I had to process the video and the slides.
Below is the video and slides that were presented during the first part of the workshop. The second part was playing the games – specifically Prune the Product Tree. I’ll also be doing sessions at the SQE Agile Development Practices conference in Orlando in November, at Agile 2010 in Orlando in August, and possibly at the Tampa PMI Symposium in October. And, of course, if you’d like help running games in your organization, please contact me!.
Serious Play: Product Planning and Prioritization using Innovation Games from Cory Foy on Vimeo.
Serious Games: Product Planning and Prioritization using Innovation Games
View more presentations from Cory Foy.
Posted on June 14th, 2010
Recently, D. André Dhondt posted to the Agile Developer Skills mailing list that he was stepping down from leading the group. The project, for those not familiar, was working to catalog the skills an agile developer would need, and provide roadmaps and guideposts to help grow people working with agile methods. There’s more information on the website.
With D. André stepping down, the question turned to what the group wanted next from the ADS project. I offered to facilitate three Innovation Games sessions using Prune the Product Tree for the group.
Because it is a collaborative, online session, I’ve scheduled three game times when people can sign up:
- Wednesday, June 16th, 2010 at 6am EST / 11am GMT
- Wednesday, June 16th, 2010 at 10pm EST / 3am GMT
- Thursday, June 17th, 2010 at 3pm EST / 8pm GMT
Each time slot can accommodate up to 8 players. If we don’t have at least 3 players for a slot an hour before, I’ll cancel the session. To sign up, go to the sign-up page and list your name in a slot. Then send me an email so I can send you the invite before the game starts.
If you have any questions, just let me know!
Posted on June 3rd, 2010
Yesterday at the XP2010 conference in Trondheim, Norway I gave a talk on Growing and Fostering Software Craftsmanship. The session was recorded by the conference, but I also recorded using my Flip camera. I put my video online, as well as the slides, and I’m hoping to also be able to post the conference video once it is available. (UPDATE: The conference video is available here, though it is Silverlight only)
Thanks so much to everyone who came out!
Growing and Fostering Software Craftsmanship from Cory Foy on Vimeo.
Growing and Fostering Software Craftsmanship
View more presentations from Cory Foy.
Posted on May 26th, 2010
This week, InfoQ published an article about the first Certified Scrum Developer class held by Ron Jeffries and Chet Hendrickson a few weeks ago. There has been a lot of fuss from the people quoted in the article that it misrepresents them and takes things out of context.
Perhaps that is the case, as I wasn’t there. But having already read Dave’s feedback on the class that the article was based off of, it pretty much summed up what my interpretations of the class were – there were struggles, and there were challenges, but overall it was a good and worthwhile class to attend, and that they were being “certified” was secondary to the class.
Here’s the deal – most people, in my experience, don’t think of the certification as “secondary”. Which is why certifications like the CSD and associated CSM classes are designed for one thing – repeatability. The class I get from Ron and Chet should be the same as what I get from someone else. Sure, there will be local differences in instructors. But the certification itself shouldn’t be based on the skills of the instructor.
Think about it this way. In his post, Dave talked about how the teams were not able to deliver stories every sprint. These are some of the top coaches and agile people available. While a team struggling is no surprise, what we should be asking ourselves is this:
1) If the team struggled, how were they helped to understand that?
2) If they struggled, how many others would struggle, and be much worse?
3) In light of that, how many instructors could guide them back to where they should be while keeping it a teachable moment?
Certainly I trust that Ron and Chet are highly capable of doing that. And I’d recommend anyone to take a class with Ron and Chet. But here’s the rub – the CSD is *not* Ron and Chet. In fact, the CSD program is not even limited to being taught by CSTs! Yep, anyone can apply to teach it.
Of course, if you are a CST, you have priority access. But, get this – so do Microsoft “MVP’s” in “Application Lifecycle Management”. Yep, people who may not have even heard of Scrum or even practiced it can sign up to be a CSD instructor!
Seriously, what the heck? Are these ALM MVPs going to be able to lead a class of people through the exercises? Can they help with the finer points of adopting agile. Are we going to trust that they can certify people to be Scrum Developers?
And that’s the thing – I don’t question Ron and Chet’s class. Neither, I think, did the InfoQ article. As Vikas said in the closing sentence:
Hence, though there were some valuable learnings coming back from the first CSD course, however it seemed that the value and the nature of the CSD certification still leaves more questions to be answered.
Darn right the “value” and “nature” of the CSD should be questioned. It’s all roses right now, because we just see the tip. But the water they are growing in is filled with festering money-grubbers and clueless managers who will twist and exploit this to great delight. And if that’s not enough to stop us from certifying people in this, then it’s just because of the ray of sunshine shining on the pot of money that is blinding us from it.
Posted on May 2nd, 2010
Post Agile. I first heard the term several weeks ago in Chicago. It was used to define companies who had started up from 2005 onward. Companies who had been versed in the Agile Manifesto, who understood Lean Principles, to whom agile was second nature. These are companies doing Scrum – Daily Standups, Product Backlogs, Scrum of Scrums. They are doing XP – Pairing stations, Test-Driven Development, Onsite customer. They have acceptance tests and weekly deploys. They have a fancy wall, or a fancy tool, and it shows all of their stories.
Over the past several weeks I’ve talked to several organizations who fit into the “post-agile” world. From the outside, they have all the appearances of “getting it”. I’ve seen Behaviour-Driven Development and iMacs. I’ve seen pairing stations, bright open workspaces, and index cards galore. And you would think after seeing all of that, I’d be overjoyed at the state of agile! It’s made it! It’s crossed the chasm!
Yet, after each organization I worked with, or talked to, I left a little more depressed. Sure, these were interesting places. Sure they were wildly successful. But the more I talked with them, the more I could see train wrecks waiting to happen – paths being taken that were only going to lead to long nights, crappy deploys, and the loss of the very thing that is making them leaders – agility.
Company A – High Cholestorol Will Kill You
The first company worth noting has all of the appearances of agility. The developers work in a big open workspace. The organization has adopted Scrum, and has Scrum of Scrums. They do weekly releases. All the developers have 27″ iMacs. In the past couple of years, they’ve grown beyond their wildest expectations. But the more I talked with their developers, their staff and their CTO, the more I couldn’t help but think of the arteries of the body when you eat high cholestorol. You might look really healthy, even exercising regularly, but inside the artery walls plaque continues to build up until, one day, blood can’t get through, or the plaque breaks loose, and you find yourself dead.
Some of the “Fried Eggs and Bacon” signs I saw:
- Executive Meetings were deemed as having a lot of waste in them. The suggestion had been made to ensure that each executive only give updates on things that were relevant to the group as a whole, or things they needed help on. Sounds innocent enough, right? Except this: If what they are working on is not relevant to the entire group, then activities which aren’t vital to the company won’t be exposed, and will continue to grow unchecked.
As an example, let’s say the sales department introduces a new incentivization program where their employees get bonuses based on how often they talk to customers. Because bonuses are involved, they develop a shared form which contains information about how the customer is using the system, how it is configured, and sales opportunities. Meanwhile, the product development team has begun a process of canvassing existing customers to understand more about their habits and how they use the system. Since both of these appear to be localized uses of the information, it may be possible that neither group knows about the other’s efforts. It’s only when customers are getting contacted by two or three separate people asking slightly different but similar questions that it is discovered.
- Even though the site has an international presence and highly targeted categories, there was no way to do A/B testing of different layouts, information schemes or other changes. If it was made for one, it was made for all. Unfortunately, this was seen as a limitation of the platform they wrote, rather than a business concern that could be addressed in small chunks. They saw the need for A/B testing, but had simply thrown up their arms saying it couldn’t be done. Anytime you say, “It can’t be done” you are really saying, “I’ve found a limitation but I don’t have the knowledge and/or experience to figure out how to adapt the current platform to do that in an incremental way. So I’ll just wait for it to be a crisis.” While crisisies are great motivators for change, it’s not a good way to introduce vital features into your core business technology.
- While pair programming was possible, it wasn’t encouraged. Like many organizations, it was seen as Ok to do sometimes, but not a daily way of working. This is a shame because judicious use of pair programming – especially early on – helps ensure a high bus number. In the short time I spoke with this company, at least two separate people were identified as being vital to critical processes. This is completely unacceptable in any organization, and especially for startups.
What stuck me the most was that this was an organization that appeared to fit squarely into the “post-agile” frame. But instead what I saw were cargo cult adoptions of very good practices that missed the real need for agility.
Company B – Going the Distance
The second company doesn’t fit into the “post-agile” company, since they’ve been around longer than 5 years, but as with our first company, have many outward signs of agility. The developers all do Test-Driven Development, and even Behavior-Driven Development. They do frequent releases. They have never missed a release date in 4-5 years. But yet, there were serious signs of cargo-cult adoption:
- Pair programming generally wasn’t encouraged. As with Company A, it’s allowed to solve some short term issue, but not at all encouraged as a normal way of working.
- The developers and QA department are separate entities, reporting to different managers. Generally this means that each department has differing goals. In practice, this usually means that QA is not seen as vital to the release cycle, but more of a “they’ll do the best they can with the time they have”. I generally frame the question like this: If you have a QA department, then you are already saying that the output of your development team is not good enough, and must be verified by an entirely different team. Think about that. Does the marketing department have a separate QA process team which verifies all production? And if they do, is something *ever* released without it going through that process? What about HR? Or Sales? In fact, the closest might be finance, but even then, it’s done via an auditing process. QA should be an integral part of your release process – so integral that they should all be on the same team.
- A further concern was that the Development and QA departments were physically separate from one another. Alistair Cockburn has materials and slides which show the costs of not having people collocated. QA and Development play such an important role together that the thought of them not being physically located together is frightening from a cost perspective.
- Incremental Improvement. This is also known as the boy scout rule. Every time you touch a piece of code you should make it a little better than how you saw it. This is especially vital when you have classic ASP pages which are 10-30kloc. Yes, *k*loc. Per page. You will never be able to just sit down and rewrite that. Only through slowly adding tests, slowly refactoring methods, slowly improving will you get to a state of understanding and safety that lets you slay it.
Of course, as I mentioned this was not a “Post-Agile” company per se. They had legacy code issues and other challenges. However, if you are claiming to be agile, then you need to tackle core issues and never be content to be happy with where you are. This is the principle of Kaizen and is reflected in many agile methodologies, most notably “Inspect and Adapt” from Scrum.
Company C – Shifting Sands Are Not Good Foundations
The final company was a brand new start up. I had the chance to meet several times with the founder and talked about technology, practices and methodologies. I found several interesting elements:
- The language choice was made without anyone else being involved. He had decided on Java, rejecting .NET and Ruby in addition to other languages. I found this interesting because what they seemed to need was a rapidly building Web platform with key areas being able to be optimized for high performance. Java can certainly fit into that, but so can many others, and locking a technology choice seemed very strange. Thankfully I sold him on Ruby on Rails, and he was willing to look at people who had both skill sets. The key here is to not limit your technology choices from the get-go. You’ll need all the flexibility you can get, and if you are hiring engineers with the understanding that they are the best and brightest and doing the best thing for the company, if they want to suggest a new framework, one should at least be open to talking about it
- Pair Programming, and Test-Driven Development, were rejected flat-out. I found this interesting because, again, as a founder making those kinds of decisions seems to be allowing oneself to be driving too deeply into the “How” of accomplishing business goals. It was also a sign that there is still much work to be done to truly bring agile over the chasm
- Similarly, the concept of a coach working with the team didn’t gain much traction. While it’s important to have hands-on people at all levels early on (“Architects must code, and so should coaches”), it is important to set solid practices from the get go. If you think you know the best way to write and produce software, and aren’t open to other approaches – even if they challenge long held assumptions – then any concept of “agility” is locked into whatever your world-view is. The narrower you hold that, the narrower the playing field your team has, and the less agility you can have.
When you are first starting a team, the practices you set early on are what define your identity for a long time to come. As my good friend Corey Haines says, TDD should be what you *fall back* on when things are reaching crunch mode. If you don’t set that early on, you only end up dug into a hole which is nothing but waste to get out of.
Your Organization
So look around your organization. Yes, even those of you who think you are in a “post-agile” organization. Do you understand the *principles* of what you are doing? Are you challenging wastes and deficiencies? Are you constantly striving to see where principles and practices make sense and stretching to try new things? If not, then you aren’t being the best you can be. So stop that. And start bring great.
Posted on April 8th, 2010
About two weeks ago Microsoft announced the release of Psscor2 – a managed debugging extension for WinDBG which is a superset of the awesome SOS debugging extension. This is an insanely useful tool when you are trying to debug problems on Production machines where you don’t (and can’t) install Visual Studio, or when you need a deeper understanding of what is going on with your managed code. While there are several articles listing out the commands and a detailed command help view, there isn’t much else.
I’m teaching my .NET Debugging class in two weeks for a client in NYC and wanted to include a section on Psscor2. Since I was going through the commands, I thought it would be fun to post them for the world to see as well.
(Note: Don’t leave this blog post without looking at the FindDebugTrue and FindDebugModules commands!)
Loading Psscor2
First, download it from the download site above. Run the executable which extracts a zip file. Now extract the files from the zip. You’ll see three folders – one for each architecture. To load psscor2 in WinDBG, start up WinDBG, attach to a process (or open a dump file) and type “.load C:\[psscor2installdir]\[arch]\psscor2.dll”. For example, my students will load it from “C:\labfiles\psscor2\x86\psscor2.dll” or “C:\labfiles\psscor2\amd64\psscor2.dll”.
Once you have it loaded, you can type “!help” to see all of the commands. If you’ve used SOS before, this should look very similar.
!dae – DumpAllExceptions
The first new command is !dae. This dumps out all exceptions on the heap along with the call stack. For this example, I loaded psscor2 and told WinDBG to break on any managed exceptions (“sxe clr”). When it broke in, I ran the “!dae” command and got the following output:
0:000> !dae
Going to dump the .NET Exceptions found in the heap.
Loading the heap objects into our cache.
Number of exceptions of this type: 1
Exception MethodTable: 000007ff001e1020
Exception object: 000000000265b0b8
Exception type: DebuggingWF.DebuggingWFException
Message: Exception occurred
InnerException: System.NullReferenceException, use !PrintException 000000000265b030 to see more
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131600
-----------------
Number of exceptions of this type: 1
Exception MethodTable: 000007feead3c240
Exception object: 000000000265b030
Exception type: System.NullReferenceException
Message: Service URL needs to be populated
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80004003
-----------------
Number of exceptions of this type: 1
Exception MethodTable: 000007feead0f288
Exception object: 00000000024c1158
Exception type: System.ExecutionEngineException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131506
-----------------
Number of exceptions of this type: 1
Exception MethodTable: 000007feead0f178
Exception object: 00000000024c10d0
Exception type: System.StackOverflowException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 800703e9
-----------------
Number of exceptions of this type: 1
Exception MethodTable: 000007feead0f068
Exception object: 00000000024c1048
Exception type: System.OutOfMemoryException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 8007000e
-----------------
Number of exceptions of this type: 2
Exception MethodTable: 000007feead0f398
Exception object: 00000000024c11e0
Exception type: System.Threading.ThreadAbortException
Message: <none>
InnerException: <none>
StackTrace (generated):
<none>
StackTraceString: <none>
HResult: 80131530
-----------------
Number of exceptions of this type: 3
Exception MethodTable: 000007feead3e060
Exception object: 0000000002568e10
Exception type: System.IO.FileNotFoundException
Message: Could not load file or assembly 'DebuggingWF.XmlSerializers, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. The system cannot find the file specified.
InnerException: System.IO.FileNotFoundException, use !PrintException 00000000025699a0 to see more
StackTrace (generated):
SP IP Function
00000000001FEB80 0000000000000001 System.Reflection.Assembly._nLoad(System.Reflection.AssemblyName, System.String, System.Security.Policy.Evidence, System.Reflection.Assembly, System.Threading.StackCrawlMark ByRef, Boolean, Boolean)
00000000001FEB80 000007FEEABEBF61 System.Reflection.Assembly.InternalLoad(System.Reflection.AssemblyName, System.Security.Policy.Evidence, System.Threading.StackCrawlMark ByRef, Boolean)
00000000001FEC10 000007FEEAC247C4 System.Reflection.Assembly.Load(System.Reflection.AssemblyName)
00000000001FEC50 000007FEE9285C0A System.Xml.Serialization.TempAssembly.LoadGeneratedAssembly(System.Type, System.String, System.Xml.Serialization.XmlSerializerImplementation ByRef)
StackTraceString: <none>
HResult: 80070002
-----------------
Total 10 exceptions
One thing you may have noticed are three odd-looking exceptions: ExecutionEngineException, StackOverflowException and OutOfMemoryException. These are three exceptions that every .NET process has on the heap. That’s because if you think about exception handling, when the runtime encounters an exception, it creates an exception object which is passed up the stack. If you are out of memory, or have a stack overflow guess what – you can’t create the object you need to report the exception.
One thing that would be very nice in !dae would be to somehow visually show hierarchies. For example, the first two exceptions have a parent/child relationship (NullReference is an inner exception of DebuggingWFException). But this is still quite helpful.
!CLRUsage
This command spits out statistics about the current CLR Usage. While you could infer this information through other means using SOS – “!eeheap –gc” for example – it is nice to have it in one place
0:000> !CLRUsage
Native Heap for .NET: 0x00480000
Number of GC Heaps: 1
------------------------------
GC Heap Size 0x1b2758(1,779,544)
Total Commit Size 00000000001c4000 (1 MB)
Total Reserved Size 0000000017e3c000 (382 MB)
Initial reservation type:
Initial Allocation Size: 0 (0) (0 MB)
Reserved Memory Size: 18000000 (402,653,184) (384 MB)
Reserved Memory Limit Size: 18000000 (402,653,184) (384 MB)
!DumpXmlDocument
There are several new dump commands available, but this looked interesting. If you have an XmlDocument on your heap, you can dump the contents using this command. Unfortunately trying to run this command I found two bugs:
- “!dumpheap –type” used to take in partial type names. It appears you have to know the full type name now
- “!DumpXmlDocument” isn’t working on my 64-bit system. It’s erroring with the following
0:000> !DumpXmlDocument 00000000026128b8
0x0000000000000000 is not an object
Which is important to note – Psscor2 is not supported by Microsoft, so you might find bugs in it.
The really cool features are in the ASP.NET support. To show this I attached WinDBG to my w3wp process and loaded psscor2.
!ASPXPages
This dumps out all HttpContexts in the heap and shows the relation to the .aspx page. This appears to be the same at !DumpHttpContext:
0:019> !ASPXPages
Going to dump the HttpContexts found in the heap.
Loading the heap objects into our cache.
HttpContext Timeout Completed Running ThreadId ReturnCode Verb RequestPath+QueryString
0x00000000ff0c7890 0 no 113 Sec XXX 200 /DebuggingASP/
0x000000013f0c92a8 19200 Sec no 113 Sec XXX 200 GET /DebuggingASP/default.aspx
Total 2 HttpContext objects
!DumpASPNETCache
This dumps out all of the Cache objects from the Heap:
0:019> !DumpASPNETCache
Going to dump the ASP.NET Cache.
Loading the heap objects into our cache.
000000017f0c6c40
Name: System.Web.Configuration.MapPathCacheInfo
MethodTable: 000007fee674fe38
EEClass: 000007fee63b15d0
Size: 40(0x28) bytes
GC Generation: 0
(C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007feead0ec90 40018d6 8 System.String 0 instance 000000017f0c6cd0 MapPathResult
000007feead0de60 40018d7 18 System.Boolean 1 instance 1 Evaluated
000007feead0ef58 40018d8 10 System.Exception 0 instance 0000000000000000 CachedException
Normal
04/08/2010 14:05:08
04/08/2010 14:15:08
04/08/2010 14:05:08
-----------------
[Output Truncated for blog purposes]
-----------------
000000013f048850
Name: System.Web.CachedPathData
MethodTable: 000007fee674ec38
EEClass: 000007fee6419bd0
Size: 64(0x40) bytes
GC Generation: 0
(C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fee674f7f8 4000d09 30 ...l.SafeBitVector32 1 instance 000000013f048880 _flags
000007feead0ec90 4000d0a 8 System.String 0 instance 000000013f0487d8 _configPath
000007fee674c818 4000d0b 10 ...m.Web.VirtualPath 0 instance 0000000000000000 _virtualPath
000007feead0ec90 4000d0c 18 System.String 0 instance 0000000000000000 _physicalPath
000007fee674ea68 4000d0d 20 ...ion.RuntimeConfig 0 instance 000000013f0687e0 _runtimeConfig
000007fee66c2f58 4000d0e 28 ...andlerMappingMemo 0 instance 0000000000000000 _handlerMemo
000007fee674ecc8 4000d08 18 ...emRemovedCallback 0 shared static s_callback
>> Domain:Value 0000000000d48450:NotInit 00000000037cdb30:000000013f042540 <<
Normal
04/08/2010 14:05:08
12/31/9999 23:59:59
04/08/2010 14:05:08
-----------------
Total 20 cache objects, Total size: 2,080
!DumpHttpRuntime
This dumps out all Http Runtime information found in the heap. This can be quite helpful when trying to diagnose things like AppDomain problems
0:019> !DumpHttpRuntime
Going to dump the HttpRuntimes found in the heap.
Loading the heap objects into our cache.
HttpRuntime 0x00000000ff0aeff0:
_shutdownInProgress: 0
_requestQueue:
0x000000013f0c9e90
_appDomainAppPath: C:\inetpub\wwwroot\DebuggingASP\
_appDomainAppId: /LM/W3SVC/1/ROOT/DebuggingASP
_fcm:
Name: System.Web.FileChangesMonitor
MethodTable: 000007fee674c730
EEClass: 000007fee6419778
Size: 104(0x68) bytes
GC Generation: 0
(C:\Windows\assembly\GAC_64\System.Web\2.0.0.0__b03f5f7f11d50a3a\System.Web.dll)
Fields:
MT Field Offset Type VT Attr Value Name
000007fee674e4d8 4000dcf 58 ...ReadWriteSpinLock 1 instance 00000000ff0b0758 _lockDispose
000007feead0de60 4000dd0 50 System.Boolean 1 instance 0 _disposed
000007feead165e8 4000dd1 8 ...ections.Hashtable 0 instance 00000000ff0b0950 _aliases
000007feead165e8 4000dd2 10 ...ections.Hashtable 0 instance 00000000ff0b09b0 _dirs
000007fee674e558 4000dd3 18 ....DirectoryMonitor 0 instance 00000000ff0b2bc0 _dirMonSubdirs
000007feead165e8 4000dd4 20 ...ections.Hashtable 0 instance 00000000ff0b0b28 _subDirDirMons
000007feead15b78 4000dd5 28 ...ections.ArrayList 0 instance 000000013f0400b0 _dirMonSpecialDirs
000007fee674e3f8 4000dd6 30 ...hangeEventHandler 0 instance 00000000ff0b2358 _callbackRenameOrCriticaldirChange
000007feead15f00 4000dd7 48 System.Int32 1 instance 0 _activeCallbackCount
000007fee674e558 4000dd8 38 ....DirectoryMonitor 0 instance 0000000000000000 _dirMonAppPathInternal
000007feead0ec90 4000dd9 40 System.String 0 instance 0000000000000000 _appPathInternal
000007feead15f00 4000dda 4c System.Int32 1 instance 0 _FCNMode
000007feead04748 4000dce 48 System.Object[] 0 shared static s_dirsToMonitor
>> Domain:Value 0000000000d48450:NotInit 00000000037cdb30:00000000ff0b24a0 <<
Finally, perhaps the most important commands one can run against a production application:
!FindDebugTrue and !FindDebugModules
It’s amazing how often people slip ASP.NET deployments to production while leaving the Debug=true element in their web.config. These commands find this problem and located any modules compiled in Debug mode:
0:019> !FindDebugTrue
Loading the heap objects into our cache.
Debug set to true for Runtime: ff0aeff0, AppDomain: C:\inetpub\wwwroot\DebuggingASP\
Total 2 HttpRuntime objects
0:019> !FindDebugModules
Loading all modules.
Searching for modules built in debug mode...
App_Web_mlxisvdk.dll not built release
DebuggingASP.DLL not built release
Done Seaching
Hope you’ve enjoyed this little journey. If you want to learn more about .NET Debugging, check out Tess Ferrandez’s blog and the slides from a talk I did recently called Debugging .NET Applications with WinDBG. Enjoy!