Debugging The Next Generation Application. | The Coffeehouse

SecretSoftware
Code to live, but Live to code.

Hi,
    I do a lot of code debugging, and I am sure that most devs do the same. However, there are just very annoying types of bugs that I call behavior bugs, that are just simply hard to catch or localize.

    One case is where you have a large multi-threaded application, and you get sometimes race-conditions and other threading related behavior bugs that are frustrating because you do not know what is happening in the application in real-time in a sort of visual way. So you spend hours upon hours to try to recreate multi-threaded application behaviors in order to fix the race-condition or behavior bug that exists.

    Naturally I want to be able to locate these types of "Behavior bugs" very quickly, and hence I have an idea that I want to share. The idea is this:
    Give us a tool that allows us developers to visually know what my application is doing at any given instance during real-time execution. That is to say, represent class objects in a visual way and show me what code each thread is executing at the moment, and display this telemetry in a visual way, so that when I look at it I can say , aha, this is the bug there, and then I can dig into the code and fix the behavior bug. Basically , allow me to see an image of the application in real-time, that shows what its doing and what lines or groups of lines of code that are being executed real-time.

    This visual way would save me many hours chancing after a behavior bug, and would make me more productive as a developer.

    If I know what my application is doing, in a visual way, then I can understand during runtime, what is happening and what code gets executed at what point during the course of the execution. This puts me in control in more intimate way. In this case , its ok to make the application execution speed slower.

Just a suggestion. Smiley

ScanIAm
On a scale of 1 to 10, people are stupid.

1) race conditions aren't random. Run your app in a debugger, and when the lockup occurs, 'break' the execution and look at the where each thread is locked. It should become pretty obvious.

2) there really isn't a good way to display the information you are looking for in a simple enough way that it would be obvious what is happening. Sure, you could do it for a trivial app, but start writing something that has lots of threads doing lots of different things, and you're out of luck on the 'visualize it' department.

If you disagree with #2, then you've found a new business you can start.

littleguru
<3 Seattle

The problem that I see is really to find a way to display what is going on... i mean in a way that you still can read it - because if for each threads all the variables and stuff would be printed that would be awful, wouldn't it?

SecretSoftware
Code to live, but Live to code.

ScanIAm wrote:
1) race conditions aren't random. Run your app in a debugger, and when the lockup occurs, 'break' the execution and look at the where each thread is locked. It should become pretty obvious.

yes, but its time consuming. And your doing it blind. In some cases , you have to step-in to large code blocks to continue executing to see what is happening. YOu dont get a real-time view of what is happening. Its really time consuming that way. I don't want to keep stepping into code that I am not interested in for the task at hand in order to find the behavior bug or the race-consditions or time bugs.

ScanIAm wrote:

2) there really isn't a good way to display the information you are looking for in a simple enough way that it would be obvious what is happening. Sure, you could do it for a trivial app, but start writing something that has lots of threads doing lots of different things, and you're out of luck on the 'visualize it' department.

If you disagree with #2, then you've found a new business you can start.

I remember in the Robotics studio they had a visual way of seeing what each thread is doing. I want this tool into Visual studio to use with my applications.

blowdart
Peek-a-boo

There was a little something on this in MSDN Mag this month

SecretSoftware
Code to live, but Live to code.

littleguru wrote:
The problem that I see is really to find a way to display what is going on... i mean in a way that you still can read it - because if for each threads all the variables and stuff would be printed that would be awful, wouldn't it?

Well, not all information / details needs to be present in the simple visual display of what is happening at runtime. If you need more info this can be viewed by right clicking or in the properties pannel, etc..

I am sure there is a way to display this effectively. Cool

SecretSoftware
Code to live, but Live to code.

blowdart wrote:
There was a little something on this in MSDN Mag this month

Thanks for that.

I think the Visual Studio as a tool , needs to evolve to the next level and afford us a way to understand thread interaction in real time. That way we can write more behaving applications and more effective ones.

I mean the way we have it now is non-visual, its like writing winform form code in code and not using the visual designer surface. I mean we need a visual way to allow us to see how threads interact , not just using descriptive text, but visually.

I am annoyed by the fact that I have to Step-into to release other threads that the debugger has stopped to debug/understand interaction. I need to see it all in real time and in a visual way, even if there are no break points.

The telemetry would then help me quickly pinpoint the bug and fix it on the spot.

ScanIAm
On a scale of 1 to 10, people are stupid.

SecretSoftware wrote:

blowdart wrote: There was a little something on this in MSDN Mag this month

Thanks for that.

I think the Visual Studio as a tool , needs to evolve to the next level and afford us a way to understand thread interaction in real time. That way we can write more behaving applications and more effective ones.

I mean the way we have it now is non-visual, its like writing winform form code in code and not using the visual designer surface. I mean we need a visual way to allow us to see how threads interact , not just using descriptive text, but visually.

I am annoyed by the fact that I have to Step-into to release other threads that the debugger has stopped to debug/understand interaction. I need to see it all in real time and in a visual way, even if there are no break points.

The telemetry would then help me quickly pinpoint the bug and fix it on the spot.

No it wouldn't. You either have a deadlock or you just have bugs. If you have a deadlock, it takes like 2 seconds to find all the threads that are running in a .NET app and figure out which ones are stuck on a lock. Seriously, use the 'threads' tab/window in the debugger and you'll be fine.

My suspicion is that you aren't using locks well, so what you are really dealing with is non-threadsafe code that is attempting to be multithreaded. No amount of visualization will help you in that regard because you just coded poorly. Put locks around objects that are accessed by multiple threads.

But ultimately, if you think that things should be visual, I'd love to see your concept for a visualization of a server app with asynchronous callbacks and mutliple threads.

Minh
WOOH! WOOH!

ScanIAm wrote:

But ultimately, if you think that things should be visual, I'd love to see your concept for a visualization of a server app with asynchronous callbacks and mutliple threads.

Ultimately, this won't be how multi-threaded apps are written & debug in the future. This thread locking & synching thing just don't scale very well at all.

I'd hate to develope this way when CPUs have 64 cores as the norm.

SecretSoftware
Code to live, but Live to code.

ScanIAm wrote:

SecretSoftware wrote:

blowdart wrote: There was a little something on this in MSDN Mag this month

Thanks for that.

I think the Visual Studio as a tool , needs to evolve to the next level and afford us a way to understand thread interaction in real time. That way we can write more behaving applications and more effective ones.

I mean the way we have it now is non-visual, its like writing winform form code in code and not using the visual designer surface. I mean we need a visual way to allow us to see how threads interact , not just using descriptive text, but visually.

I am annoyed by the fact that I have to Step-into to release other threads that the debugger has stopped to debug/understand interaction. I need to see it all in real time and in a visual way, even if there are no break points.

The telemetry would then help me quickly pinpoint the bug and fix it on the spot.

No it wouldn't. You either have a deadlock or you just have bugs. If you have a deadlock, it takes like 2 seconds to find all the threads that are running in a .NET app and figure out which ones are stuck on a lock. Seriously, use the 'threads' tab/window in the debugger and you'll be fine.

My suspicion is that you aren't using locks well, so what you are really dealing with is non-threadsafe code that is attempting to be multithreaded. No amount of visualization will help you in that regard because you just coded poorly. Put locks around objects that are accessed by multiple threads.

But ultimately, if you think that things should be visual, I'd love to see your concept for a visualization of a server app with asynchronous callbacks and mutliple threads.

Well, I know that you can find it. But its not only deadlocks. I included this as an example, but there are more complex bugs that has to do with program logic flow and program behavior that are hard to replicate and reproduce and detect generally.

The code is not buggy, its the interactivity of threads and the general plan of execution that needs to be debug, and that is a hard thing to do. On code as written its looks right but in real time , at some points weird behaviors are observed and its hard to reproduce this. For example some part of the execution plan finishes before some other part or somehow the general logic gets screwed only at certain random points that are hard to nail. Its not simply locks, its how threads interplay and interact in the app.

About a server application's visualization. I would represent each class in a visualization surface, and then i would designate a color to each thread. And I would place a border with the color of the thread around each class, these "thread-colored borders" then would move around the class representations as the application is executing, indicating to me what is the class-focus of the thread that is executing, and what is the line of code that is currently executing...etc. visually.

I can get information also as to what is the focus of the thread at any given moment in time. (ie, what method its executing, and so on). Some threads would not move around between classes, and would stay at a given function (for example Socket I/Os) others would move around. I can focus on one or more threads or on all threads that are executing, including those from ThreadPool.

THis is a general way of getting the whole execution picture, and then I can know what is causing a given behavior and at what point. THis way I can see the "conditions" and be able to reproduce it and understand it in a much more meaningful way than the way we have now.

This is a simplified way to doing it.

JChung2006

SecretSoftware wrote:
I think Visual Studio as a tool, needs to evolve to the next level and afford us a way to understand thread interaction in real time.

Visual Studio and other software development tools need to evolve beyond even that to the point where compilers are capable of generating provably correct, multithread-safe code without (much) human interaction.

If multithreaded programming has demonstrated anything, it's that most programmers can't program worth a damn when multiple threads are involved. Self included in this particular category.

wkempf

ScanIAm wrote:
1) race conditions aren't random. Run your app in a debugger, and when the lockup occurs, 'break' the execution and look at the where each thread is locked. It should become pretty obvious.

By very definition a race condition is random. Further, running the application within the debugger can change the behavior when race conditions are involved. Finally, many race conditions do not result in deadlock. Those are two different problems that can occur in multi-threaded code with broken synchronization.

ScanIAm wrote:

2) there really isn't a good way to display the information you are looking for in a simple enough way that it would be obvious what is happening. Sure, you could do it for a trivial app, but start writing something that has lots of threads doing lots of different things, and you're out of luck on the 'visualize it' department.

If you disagree with #2, then you've found a new business you can start.

I won't say there isn't any way, but I will agree that it's non-trivial and anyone that accomplishes this has done something very note worthy.

wkempf

Minh wrote:

ScanIAm wrote:
But ultimately, if you think that things should be visual, I'd love to see your concept for a visualization of a server app with asynchronous callbacks and mutliple threads.
Ultimately, this won't be how multi-threaded apps are written & debug in the future. This thread locking & synching thing just don't scale very well at all.

I'd hate to develope this way when CPUs have 64 cores as the norm.

Past 2, the number of cores doesn't really change the complexity (and in theory, though the behavior is different, the complexity isn't any less with a single core). The only reason why the multi-core machines are finally making this a hot topic is because more code is likely to be made multi-threaded in order to take advantage of the multiple cores. IOW, it will no longer be a practice reserved for niche applications by "expert" developers.

However, I agree with you. I don't believe the current threading models that rely on shared memory and synchronization are likely to remain dominant. Other models (which have existed for a very long time, btw) that eliminate shared memory, or other such concepts, are going to see a huge surge in popularity and research.

wkempf

ScanIAm wrote:
No it wouldn't. You either have a deadlock or you just have bugs. If you have a deadlock, it takes like 2 seconds to find all the threads that are running in a .NET app and figure out which ones are stuck on a lock. Seriously, use the 'threads' tab/window in the debugger and you'll be fine.

My suspicion is that you aren't using locks well, so what you are really dealing with is non-threadsafe code that is attempting to be multithreaded. No amount of visualization will help you in that regard because you just coded poorly. Put locks around objects that are accessed by multiple threads.

But ultimately, if you think that things should be visual, I'd love to see your concept for a visualization of a server app with asynchronous callbacks and mutliple threads.

And now it's obvious you are confusing deadlock with race condition. They are NOT the same thing. A race condition can lead to deadlock, but deadlocks can be created without a race and a race can result instead in corrupted data, segmentation faults or erroneous processing. Don't confuse the two.

victor louis

Race conditions are among the most common classes of bugs found in deployed software. They are only possible in environments in which there are multiple threads or processes occurring at once that may potentially interact (or some other form of asynchronous processing, such as with UNIX signals). People who have experience with multithread programming have almost certainly had to deal with race conditions, regardless of whether they know the term. Race conditions are a horrible problem because a program that seems to work fine may still harbor them. They are very hard to detect, especially if you're not looking for them. They are often difficult to fix, even when you are aware of their existence. Race conditions are one of the few places where a seemingly deterministic program can behave in a seriously nondeterministic way. In a world where multithreading, multiprocessing, and distributed computing are becoming more and more prevalent, race conditions will continue to become a bigger and bigger problem. . A good static source code analysis tool like Coverity Prevent will identify any data race it encounters while executing your program, and flag it for you to fix. Coverity has customers like Symbian, RIM (Blackberry), Juniper networks, Cisco, Texas instruments and is also used by the Department of Homeland security to scan lots of open source projects. You can get more info at http://www.Coverity.com

Isshou

victor louis wrote:
Race conditions are among the most common classes of bugs found in deployed software. They are only possible in environments in which there are multiple threads or processes occurring at once that may potentially interact (or some other form of asynchronous processing, such as with UNIX signals). People who have experience with multithread programming have almost certainly had to deal with race conditions, regardless of whether they know the term.

A race condition can happen, even on a single processor computer. It's less likely, but any time that you can not guarantee when your thread/process will be context switched out for another you can't guarantee that another one of your own threads won't run before your first thread get CPU time back. This of course means that you need to address race conditions and prevent multiple entrance into critical sections, etc from occuring. Of course this all seems "at once" to a user but is far from it in terms of performance.

victor louis wrote:
They are often difficult to fix, even when you are aware of their existence. Race conditions are one of the few places where a seemingly deterministic program can behave in a seriously nondeterministic way. In a world where multithreading, multiprocessing, and distributed computing are becoming more and more prevalent, race conditions will continue to become a bigger and bigger problem.

Race conditions should be easy to fix, once you understand what the race condition is, there's many thread synchronization flags, locks, mutex, etc to deal with it. Anyone that has looked at general programming or database programming is familiar with the concepts from the reader/writer problem (also consumer/producer, and many other example situations).

Reader/Writer situation is that a prime example: you have a reader wanting to read a record and a writer wanting to write the same record. You can't just have them both working on the same record because you're not guaranteed that the reader won't end up with part of the old record and part of the new record. In fact you couldn't accurately predict what amount of new vs old data the reader would pick up. Solutions are varied and each have benefits and drawbacks. You can lock the record while it is being read (allowing multiple readers though) or being written (allow only one writer), this prevents corruption but causes wait times, etc. You can do a transactional system where anyone working with the set of data deals with the same version of the data which prevents the wait time but is harder to implement and means the reader will get old data and thus you need to handle the case where the reader might pass this data to a writer to be written.

Graphics is another area of race condition when performing calculations on different threads. This is where we get the idea of frame state, frame skipping, and such things to allow the processing to continue and still allow for the display to update.

Shining Arcanine

Use flow charts when designing your programs:

http://www.mersenneforum.org/attachment.php?attachmentid=1608&d=1174569923