Sunday, March 25, 2012

5 reasons editorial votes in Topcoder are useless

Know what? I am quite tired of that non-sense. I hate that after working hard for 2 days to write an editorial I am somehow supposed to care about anonymous votes that are completely uninformative.

5. Not sure people are actually voting about the editorial itself.
There have been plenty of times it seemed like votes indicated not the people's opinion about the editorial, but their opinions about other things.

Let me refer to the first editorial I wrote in the "new" system: editorial thread. Look at those votes, +53 net votes!. That is the best outcome I ever got on the editorial feedback post. But really, was that editorial so great? I doubt it. I can easily tell that I have improved over the time and that editorial has some problems I learned to avoid. I am quite sure that a humongous amount of pluses are really about the idea of finally dropping the wiki 100% contribution-based system which resulted in hundreds of editorials that lack explanations for most of their problems (SRM 453 happens to be the very second SRM to use the new method).

Now, take a look at TCHS 2010 round 3: editorial thread. The first and only time I got a negative net result. I am pretty sure that people were voting against the problem set rather than the editorial. It was a very messed up problem set that was also evil and kind of boring and the Folding Maze problem makes your brain melt. Yet it was the match to define advancers to the championship round. So, it is easy to see why many people disliked the problem set. Then my editorial who I am sure was slightly better than SRM 453, received much worse feedback than it. Coincidence?

Yet, I use words like worse or better to compare voting results. When it turns out I have no idea what is a better outcome and what isn't:

4. What is better anyway?
Compare an editorial that got +20/-7 vs. and editorial than had +12/0. In once case, the votes are unanimous, yet in the other there are actually more people that like the editorial. How are we supposed to interpret this?

But this is part of a greater issue:

3. As feedback, it is useless
I really have no idea what I do right when I get seemingly positive outcomes in this vote. I really have no idea what I do wrong when I get more negative outcomes.

A single post saying why the editorial is good and what can be improved. Or at least what is it that you hate about the editorial is the sort of thing that really allows you to improve and improve in the next editorials. It is also very important to know who is the one making such suggestions.

That's the irony, the completely anonymous and quick votes, actually reduce people's reasons to make such posts. Not content with being useless feedback, it reduces the chances you will get useful feedback.

So, maybe it is true that you can't find out how to improve an editorial by looking at the votes. They would at least be able to quantify the number of people that liked the editorial. If it wasn't that:

2. Nobody actually votes.
Quick exercise: There are 2500 registered coders for a SRM. Of them , 2300-ish actually participate. Some smaller percentage actually read the forums. Some other small percentage reads the editorials. And what's the maximum number of votes we ever had for an editorial? I would say it is unlikely it is over 100. And most frequently, only around 30 people seem to vote. 30 out of 1500? Does not that sound of quite a minority?

I am not making the discovery of the year. misof pointed this out 2.25 years ago.

* I don't think that the number of +s in mystic_tc's "Editorial" threads is a good measure of how good the editorial was. For rounds where the problems are such that more discussions occur, more people will read the post and consider voting. Additionally, only people who read forums will vote, those who go straight to the editorial page won't, and this makes the vote biased.


But maybe we shouldn't worry that much, because:

1. They make no effect whatsoever
I don't think any editorial writer has ever lost access to write editorials. In fact, most of the time, the admins are not even in position to pick an editorial writer, and if there are more than one editorial writer, the rules to pick have 0 consideration of these votes. The problem setter has priority over tester. The tester over unrelated editorial writers and otherwise. Otherwise, the priority goes to the person that has not written an editorial for the most time.

4 comments :

hacker007 said...

There are people who always enjoy reading your editorials, including me (plus votes of course). You did really well in explaining things with accurate words and nice pictures, thanks A LOT, vexorian.

kkab said...

I know how much effort is put into an editorial and you come up the best all the time:)That's amazing:) Keep up the good work:)Your recent one was great too:)Read it as soon as it was published:)I am a fan of sorts of your editorials.
It felt nice to read this post after I got some negative votes.It is true that we have to take all views, but it really isn't a view.For example someone who comes looking for a Huffman Encoding solution explanation in my editorial would be disappointed and he would instantly - the post.I console myself saying that.If only people can give actual feedback, maybe to our mail id's it would be real feedback and improvements could be done:)

vexorian said...

Thanks guys.

Anyway, I will from now on stop making Topcoder editorials until they get rid of the stupid random number generator that is the minuses to grade them. I am tired of working for two days only to find myself subject to this. Always 2 or 3 minuses from people who cannot be helped to explain how to improve the editorial or cannot improve the editorial themselves (it is a wiki). The 75 bucks per editorial won't be missed. In the past, I intended to do this but TC seemed needed of editorial writers too often, now it seems they won't have that issue. I'd much rather use the time I will win by not having to do editorials in this blog or in thinking for new problems. In conclusion, I see one reason to stop writing them and no reason to continue writing them

kkab said...

I know how much effort is put into an editorial and you come up the best all the time:)That's amazing:) Keep up the good work:)Your recent one was great too:)Read it as soon as it was published:)I am a fan of sorts of your editorials.
It felt nice to read this post after I got some negative votes.It is true that we have to take all views, but it really isn't a view.For example someone who comes looking for a Huffman Encoding solution explanation in my editorial would be disappointed and he would instantly - the post.I console myself saying that.If only people can give actual feedback, maybe to our mail id's it would be real feedback and improvements could be done:)