Google Summer of Code Application Evaluation

Originally based on the NESCent Phyloinformatics Summer of Code procedure by Hilmar Lapp.

Overview

We'll have 3 rounds of scoring. In the first round, mentors review the applications for their project idea and score them (see below for specifics). In the second round we would also like all mentors to look at (at least) the top-scoring applications for project ideas other than their own and score them too so that we get a poll on consensus or divergence of opinions. In the third round where necessary Todd and I may weigh in to ensure optimal alignment with NESCent's objectives and mission.

Before I'll go into specifics, as a reminder we're not expecting to accept more than 6 students this year. With 17 project ideas listed at last count, simple math shows already that about 1/3 of our project ideas won't get funded. Some of those may not receive strong applications in the end, so the decision will be easier in those cases; however, we do expect that we'll have to make some very tough calls.

With that in mind, here are my suggested guidelines for scoring in the first round.

First-round scoring Guidelines

1. Review the applications that you would be primary or secondary mentor for (this should be apparent from the title). Point out any missing information, lack of detail, and any aspects of a proposed plan that you disagree with in the comment text box, check the box next to 'Public review', which makes it visible to the student, and click 'Submit'. The student should respond to the these and can presumably edit his/her application to respond to your request for information or criticism (at least they could last year).
2. For scoring, leave the 'Public review' box unchecked (i.e., the student will not see it, but other mentors will), change the 'Score' drop-down to where you want it, add explanatory comment for your score, and hit 'Submit'. Scores are cumulative, so you can revert a score by subtracting what you previously added. Scores without explanatory comment are hard to weigh in the case of differing opinions, so please do add a brief (!) statement as why you are giving this score.
3. If you are a secondary mentor, to avoid confusion I suggest you propose your score as a private comment (i.e., as above, except write the score in the comment and leave the drop-down at zero) for the primary mentor to take into consideration, or email the primary mentor.
4. Give the top 2 or 3 applications for your idea that may warrant funding (see below) a gradated positive score to establish your preference (+4 for the best, +3 second, +2 third). Add +1 to the best one (for a total of +5) if your comments only point out refinements and in principle you're happy with the application as is.
5. Only give a positive score if you have confidence that the student is already or would after requested improvements be worth accepting, and you spending your mentoring time. There may be fewer than 3 of those for your idea!
6. Score all others for your idea with a -1.
7. If you are a primary mentor, request mentorship by clicking 'I am willing to mentor' for those you have scored positively. This should be no more than 3.
8. Keep in mind for any of the above that total scores you assign aren't final. By adding or subtracting you can shift an application from one category to another, and you can withdraw your mentorship request later.

This should leave us with those applications that a slot is being requested for at score +4 or +5. It would be great if we can get to this point by Tuesday night. (Yes, you can change your mind later.)

In the second round of scoring, all mentors will be invited to look at all applications scored positively in the first round (except those they'd mentor themselves), and add scores corresponding to the degree they agree an application should receive a slot. We'll wait with this until Tuesday and the first round is hopefully complete.

Finally, the primary (or secondary) mentors should start to arrange an interview with at least their top applicant, and with both top and second if they are close in order to assess the student's communication abilities and whether personalities (yours and theirs) match well enough. The interview should take place *during the review period*, the earlier the better, and you should comment on your conclusions from it in a private comment. The interview may be by email, chat, Skype, or phone, depending on what you feel lets you best make the assessment.

Things to keep in mind

1. Your top student may also have applied to another org and may stand to be accepted there too. Students can only be accepted for one project, and hence one org; if in the end that student goes to the other org, you lose the slot if you don't have a second one of comparable quality "wait-listed", which is the reason you want to keep working with more than just the best one (though doing so with more than 3 is a waste of time).
2. When ranking student applications against each other, weigh the complete picture, a significant part of which is the long-term prospect of the student continuing to enrich or advancing your open- source project or phyloinformatics as a field through what you will teach him or her over the summer. If an application seems more solid technically than another but the student's career and personal interests and past activities suggest that he/she would likely disappear immediately after Summer of Code ends, would you still prefer to invest your mentoring time in that student? Our goal in participating in GSoC is not just to get things done, but to train and recruit future phyloinformatics open-source software contributors.