Why Open Bug Tracking Fails
Unlike proprietary platforms, Ubuntu allows end users to interact directly with developers through Launchpad’s bug-reporting system. In some cases, this approach allows bugs to be discovered and resolved quickly. In most situations, however, open bug tracking is a fiasco that Ubuntu would be better off without. Here’s why.
Open bug reporting policies, which allow anyone to file bug reports that developers are expected to address, operate under the principle, famously decreed by Eric S. Raymond, that “Given enough eyeballs, all bugs are shallow.” In other words, the more people you have searching for and reporting problems, the faster they can be fixed.
In the best of all possible worlds, where end users are also professional programmers, Raymond’s logic may hold true. But in real life, only a fraction of the people using a given software product have any idea how to read code or troubleshoot problems.
Distracted developers and frustrated users
When only 0.1% of your users know how to write code, but you give the other 99.9% equal access to your bug-tracking system, you end up with a lot of problems. Developers whose time is best spent fixing bugs are instead forced to walk end users through the tedious process of providing backtraces, and bug reports are left cluttered by redundant or off-topic information posted by users who don’t know any better.
More dangerously, too many end users are given the impression that the bug tracker is a support system whose chief goal is to fix individual problems. In reality, the tracker is a tool for developers, not users. By allowing anyone to file bug reports easily, Ubuntu discourages use of support channels like the forums, where end users stand a much better chance of having their issues addressed satisfactorily.
Ubuntu’s approach to bug tracking is also flawed in that the vast majority of reported bugs ultimately need to be fixed by upstream programmers, not Ubuntu’s developers. Yet Ubuntu, as Mark Shuttleworth wrote a while back, handles the lion’s share of bug reports. This makes Ubuntu an extraneous and redundant middleman, hampering efficiency.
Instead of encouraging everyone and his grandmother to file bug reports when something goes wrong, users should be made to understand that just because they can file reports doesn’t mean they should. They should also be required to provide backtraces before they submit reports, which would effectively discourage filings from people who are not–and should not be expected to be–qualified to assist developers.
In addition, better use should be made of apport, Ubuntu’s tool for automatically providing feedback to developers when a program crashes. Most normal users lack the technical skill to submit effective bug reports directly to Launchpad, but apport requires only a few clicks to send developers useful information about a problem.
Development is for developers
As of right now, Ubuntu has 56725 bug reports open in Launchpad. This number increases by several hundred each week, and historically, only a fraction of reported bugs are ever fixed; most linger in a zombie state until they’re closed by default when their packages become obsolete. If Ubuntu doesn’t reform its bug-tracking policies, this situation will grow increasingly out of hand.
Users should be encouraged to use Ubuntu, and leave development to the developers. Trying to bastardize these two groups serves no one.
Follow WorksWithU via Indenti.ca, Twitter and RSS (available now) and our newsletter (coming soon).
The situation is already out of hand and it’s normal to be like that.
That’s how its planned to be, and I doubt other distribution differ in any way.
Also getting common users to contribute bugs is another explicite choice and a way to get the users involved.
That requires a tremendous effort (and waste) of human resources to manage things, but that’s part of the Ubuntu way (and other distributions way, to some extent) to get the stuff done.
Having a packaging system of any kind, having official repositories, dropping explicit binary compatibility with debian all requires a tremendous work. Sabdfl already noted that many times.
Still the choice is to optimize as much as possible, not changing route in any way.
Launchpad itself is part of that effort. It makes sharing bugs between projects easy, to some extent even if the upstream uses a different bug tracker.
That’s messy, that’s Linux.
I wouldn’t be opposed to requiring some minimum karma score before being allowed to post bugs to launchpad.
Karma could be acquired before being permitted to post bugs by some mechanism.
Stack Overflow uses this strategy to great effect.
No system is perfect, and while many of the bugs reported to launchpad are really not bugs per se, i think the present system is quite good and all the user needs is better education on our to use it rather than scraping it altogether. You made a valid point about developers filing bugs but miss the point that and enlighten user is in a better position to file a bug compared to a developer. The user uses the software and its easier for them to catch bugs that were either over looked or escape the eye of the developer.
what most people need is to be educated how to file bugs, many users are not even aware of the answer page in launchpad where people can go and post question about a particular software. but even the process as it is now a bug has to be marked as valid and confirmed before it can be given attention.
removing the user input from launchpad imho is like throwing the baby away with the birth water.
You failed to support your assertion that it was a disaster so I’m going with bigbrovar on this one. I agree with expanding the use of apport, but in the end, you can only address bugs if you can find them and apport is never going to be a 100% solution.
Rather than throwing the baby out with the bathwater here, perhaps there are things that can be done in user education and in detail collection that can reduce the rate of problems. Some documentation listing tools that should be used to gather data would be nice to see. Perhaps requiring a report from checkbox before your report is listed as complete would prove the right approach.
I think you’re seeing Launchpad as a “bug tracking system” when Ubuntu developers see it more as a “user report system”. Reports on Launchpad don’t have to be proper bug reports. The developers can look and see which reports have enough information or participants to escalate them to a proper “bug” or “task”. Think of it as a discussion forum with added features for developers to mark specific threads as needing their attention and track their progress.
Viewed like this, there’s nothing wrong with the fact that most reports are never addressed: these aren’t unfixed bugs but just user reports that weren’t useful enough to become actionable.
Nonsense.
If developers have to lead bug reporters by the hand to provide debugging information, then the debugging facilities and documentation are inadequate. It should be as automatic as possible.
When you submit a bug to Apple, you’re required to submit a System Profiler report at the same time. Where’s the equivalent in Ubuntu? http://brainstorm.ubuntu.com/idea/4848/
If bug reports are cluttered with off-topic comments, provide a way to rank and filter them, highlighting relevant, helpful comments, hiding “me too” comments, and minimizing comments that were previously thought to hold solutions but are now known to no longer be true.
To minimize “me too” comments, provide a way to vote on bugs that affect you, with a visible indicator of how many people have previously voted on it, so the users know that the developers know that it’s a problem. Tie this voting to the hardware profiles associated with each user to help discover where the incompatibilities lie.
Provide a separate wiki-like page attached to each bug report where users can help each other workaround the problem (separate from the developer’s attempts to FIX the problem) instead of huge lists of comments that are hard to navigate through, leading to duplicate comments and lots of chaff.
I disgree with your points. In a proprietary system,
a) not all bugs reports that could have been filed get filed (registration and other annoyances)
b) users don’t help users sort out problems
c) patch writers can’t just come by a report and write a fix.
Canonical/Ubuntu need a team of people to filter the bug reports. This should be done on a full time basis. They should then send “real” bugs to the appropriate Devs or Upstream. With the number of bugs in the system, there would be more than enough work for a few people. They could also compile a FAQ for common submissions that are not actually bugs. Use them to educate users too. Nothing stopping this from happening, except money.
A greater problem for me is how Ubuntu developers tend to blame bugs on me, or on some system that they have no control over, when most often it is something that worked in the last version and now has stopped working after the upgrade.
I’m in agreement with most of the comments here. This feels like a hastily-written criticism designed to draw traffic than raise awareness (very Dvorak-like). Yes, the apport could be put to better use, and perhaps some language redirecting some questions to forums would be of use, but I just don’t see these as valid criticisms.
First, looking at the front page of launchpad — or any page of launchpad for that matter — there are tabs for Overview, Code, Bugs, Blueprints, Translations, and Answers. Bugs only make up part of the system. I have a little more faith that the majority of users will be able to read the screen.
Second, if your bug isn’t actually a bug and is a question, the Bugs page has an option for that. If you’re a fairly new or basic user and don’t really know what a bug is, you’re more likely to click on the big blue “Ask a question” button instead of the big read “Report a bug” button. Besides, if you’re a basic user and you get into the “Report a bug” section when you should have asked a question, before long, the process itself will imply that you might not be in the right place.
Third, whenever a bug is filed — or whenever I’ve filed a bug there — an automated return comes up requesting other necessary information with instructions on how to provide it. Most people who are clever enough to realize something is a bug and then find there way to launchpad will be able to run those simple codes and attach the necessary files. If the problem was already addressed someplace, it’s never very long before either a developer or the community tells you that it’s been addressed
That is, if the system doesn’t catch it first and make a few suggestions. Once someone describes their bug in a title, you get a big page that says “Is the bug you’re reporting one of these?” and it shows you a slew of options. That’s automated; there isn’t a developer wasting time fielding all those requests, finding the other possible options and then returning those options to the end user. So it’s not taking up developer time.
Fourth, how is it a flaw to have launchpad as a middleman between the user and the larger upstream ecosystem? If you have a problem with an application, but don’t know that your problem is actually with a library or some component that’s maintained upstream, launchpad seems to be a very effective way of getting that information to the upstream developers.
Besides, if you didn’t know it was a problem with some smaller component, how would you ever go out on the web and find the bug system for that unknown component in the first place? You can’t find the right tool for a problem if you don’t know what the problem is. In cases like this, launchpad exists as a knowledgeable system that can identify such issues and pass them on, effectively creating and maintaining some needed communication where it might have never existed.
I’ve read a lot of critical articles in this blog; I wish more of them were actually constructive. I think ten minutes reflection on your launchpad complaints (and maybe a visit to the page) would have changed the tenor of this piece and its responses.
I agree with every one else. The system can, and should be improved in many ways. Closing it up is not one of these many ways. That’s how we work in Open Source, like a Bazaar. People who prefer the Cathedral, well, they have the option to use closed source software.
By the way, anyone can report linux kernel errors, too. Should the kernel also go behind doors ??????
Interesting site, but much advertisments on him. Shall read as subscription, rss.
Statements like “I doubt other distribution differ in any way” or “That’s how we work in Open Source” are missing the point. Ubuntu has more users than any other distribution, and amongst Free Software projects in general, it is probably second only to Firefox in the number of users who are aware they are using it. And as the number of users increases, the ratio of competent bug triagers to bug reporters naturally decreases. So it would not be at all surprising if we introduced more alternative forums or prerequisites for bug reporting (in addition to the prerequisite of having a Launchpad account).
However, the suggestion that people “should also be required to provide backtraces before they submit reports” is also misguided. I’ve worked for Canonical for four years and reported hundreds of bugs, many of which have been fixed, but I don’t know how to generate a backtrace.
Has the author of this article ever worked on the receiving end of an open bug tracker? It’s an amazing and humbling experience, and doesn’t leave much doubt as to the value of open bug tracking. Sure, any given open tracker could be improved — but that’s true of all systems, whether open or restricted. Susceptibility to improvement is not a sign that anything’s wrong.
Some points:
The number of bug reports is proportional to the number of users, not to the number of actual bugs. A large and active bug tracker is a sign of large and active user base, and the ratio of open-to-closed bugs tells you nothing at all about the software’s health or stability.
Similarly, the ratio of unhelpful reports to helpful ones is not in itself relevant. You’d have to look at how much effort each kind costs; in general, unhelpful reports can be dispatched with very quickly — often by other users, rather than by the developers — and most developers spend most of their time dealing with substantive reports that could actually lead to a fix. Just because all that information is available doesn’t mean people can’t choose where to focus their attention!
You also can’t know the frequency of “bugs not filed”: that is, the number of times someone was about to file a bug, but then saw (thanks to the open tracker) that someone else had already filed it. Launchpad.net’s “this bug affects me too” button at least gives us a way to estimate the relative prevalence of bugs; again, this is entirely dependent on the open nature of the tracker, which gives people a way to follow a bugfix’s progress without having to enter into any relationship with the tracker or the tracking organization. (Just try engaging with a closed shop that way.)
Finally, it’s very difficult to compare an open bug tracker like Ubuntu’s with a closed bug tracker like, say, Microsoft’s, because by definition you don’t have access to the data in the closed tracker. For all we know, Microsoft has a zillion times more bug reports, but we just don’t see them. They could be of worse quality too, and we’d never know. One thing we do know, however, is that Microsoft is unable to enlist its users in helping to remove duplicate and invalid bug reports: instead, its own employees have to shoulder all of that burden.
I think this article would have needed to gather a LOT more real data, and think about the problem much more thoroughly… but then it would have been a very different article. You can’t just say “Look at all those Ubuntu bug reports — there must be a problem here!” That’s not how it works.
@mpt: I am not sure I missed the point. The point _I_ was trying to make is that open access to information is at the very heart of open-source / free software. In other words, Cristopher seems to have fundamental issues with the very principles upon which free software is built. The day Ubuntu goes behind doors I will become an ex-user. That won’t happen though (the former).
[…] project succeeds, it#8217;s going to grow forever. Trying to suppress that growth (for example, by discouraging filings from all but technically qualified reporters) simply squelches a useful information source. Far […]