A few more thoughts... (I know, I know, when is he going to shut up, right?)
I think one way to address my suggestions would be to use a bit more of a quantifiable form for the panel judging. Something like the current system (as it appears in the Google doc) but with a bit more structure, and less abstract. Simple way to do that is to use a scale rating system that addresses all the necessary aspects of a complete build. e.g.:
====================
WRITING [ __/10 ] (Add up the points below)
• Grammar: __/3
• Spelling: __/2
• Phrasing, syntax, clarity, etc.: __/5
• Written Notes:
LORE & ROLEPLAYING [ __/10 ] (Add up the points below)
• Adherence to Canon: __/5 (3 being no canon to draw on, 1 being major deviance, 5 being strictly accurate)
• Roleplaying: __/5 (wrt. whether the roleplaying adds to the gameplay, makes sense, and is original)
• Written Notes:
PRESENTATION [ _/10 ] (Add up the points below)
• Images: __/5 (wrt. number of images, spacing, image quality, relevance, etc.)
• Formatting: __/5 (wrt. how text and images are organized, text effects, etc.)
• Written Notes:
PERSONAL SCORE [ __/20 ] (Add up the points below)
• Judge's subjective general rating: __/15
• Rank: __/5 (static +5 points if it's the best in that panelist's opinion, +3 if second best, +1 if third best, otherwise +0)
• Written Notes (Summary/Argument):
OVERALL SCORE: __/50 (Add up the points above)
====================
That's obviously a very rough example mock-up, and perhaps it looks like more work, but something like (but more polished and with an agreed upon weighting system) that should ultimately make it a lot easier to tally the votes. It'd also make it easier for the readers to sift through the panelists' responses, by providing more quantifiable feedback in addition to the personalized critiques. e.g., Rather than looking at "+/–" scores, I can look at average scores on a static point scale, and compare the scores between different panelists and/or different builds.
Another idea — and perhaps you guys did this as well, but it's not clear from the Google docs — would be for the panel to get together on Skype or IRC or something, and have a group roundtable where they dedicate 10-15 minutes to discuss each build and come to a consensus. I know timezone constraints make this difficult, but I think group collaboration would probably be a simpler and more effective way to improve the panel system moving forward.
I'll end with another disclaimer: I was neither a panelist nor a builder in the competition. Obviously, therefore, I do not have an intimate knowledge of how the panel operated. This may make my feedback appear presumptuous, but I do not mean for that. I'm trying to offer the best, most genuine advice I can to ensure this contest is an ongoing success. I do understand the organizers surely spent a hell of a lot of time and effort already going over the possible options, and I don't mean to undermine that. So feel free to disregard my suggestions, I won't take offense. But I hope you see where I'm coming from, and that at least some of what I've said is helpful.
Again, cheers guys. Great work. Sorry for all the words and stuff, I'm done now I swear.
A few more thoughts... (I know, I know, when is he going to shut up, right?)
I think one way to address my suggestions would be to use a bit more of a quantifiable form for the panel judging. Something like the current system (as it appears in the Google doc) but with a bit more structure, and less abstract. Simple way to do that is to use a scale rating system that addresses all the necessary aspects of a complete build. e.g.:
====================
WRITING [ __/10 ] (Add up the points below)
• Grammar: __/3
• Spelling: __/2
• Phrasing, syntax, clarity, etc.: __/5
• Written Notes:
LORE & ROLEPLAYING [ __/10 ] (Add up the points below)
• Adherence to Canon: __/5 (3 being no canon to draw on, 1 being major deviance, 5 being strictly accurate)
• Roleplaying: __/5 (wrt. whether the roleplaying adds to the gameplay, makes sense, and is original)
• Written Notes:
PRESENTATION [ _/10 ] (Add up the points below)
• Images: __/5 (wrt. number of images, spacing, image quality, relevance, etc.)
• Formatting: __/5 (wrt. how text and images are organized, text effects, etc.)
• Written Notes:
PERSONAL SCORE [ __/20 ] (Add up the points below)
• Judge's subjective general rating: __/15
• Rank: __/5 (static +5 points if it's the best in that panelist's opinion, +3 if second best, +1 if third best, otherwise +0)
• Written Notes (Summary/Argument):
OVERALL SCORE: __/50 (Add up the points above)
====================
That's obviously a very rough example mock-up, and perhaps it looks like more work, but something like (but more polished and with an agreed upon weighting system) that should ultimately make it a lot easier to tally the votes. It'd also make it easier for the readers to sift through the panelists' responses, by providing more quantifiable feedback in addition to the personalized critiques. e.g., Rather than looking at "+/–" scores, I can look at average scores on a static point scale, and compare the scores between different panelists and/or different builds.
Another idea — and perhaps you guys did this as well, but it's not clear from the Google docs — would be for the panel to get together on Skype or IRC or something, and have a group roundtable where they dedicate 10-15 minutes to discuss each build and come to a consensus. I know timezone constraints make this difficult, but I think group collaboration would probably be a simpler and more effective way to improve the panel system moving forward.
I'll end with another disclaimer: I was neither a panelist nor a builder in the competition. Obviously, therefore, I do not have an intimate knowledge of how the panel operated. This may make my feedback appear presumptuous, but I do not mean for that. I'm trying to offer the best, most genuine advice I can to ensure this contest is an ongoing success. I do understand the organizers surely spent a hell of a lot of time and effort already going over the possible options, and I don't mean to undermine that. So feel free to disregard my suggestions, I won't take offense. But I hope you see where I'm coming from, and that at least some of what I've said is helpful.
Again, cheers guys. Great work. Sorry for all the words and stuff, I'm done now I swear.
