As I mentioned in my last post, I received the response to my technology grant proposal and, unfortunately, was not awarded funding for my iPad in Paramedic Education proposal this time around. The e-mail with the result also said that we’d be getting an additional e-mail with more details on what the reviewers had to say and how they scored our proposals. I got that e-mail and I’d like to (…finally) share some of that information!
Each of the proposals that was submitted was reviewed by three randomly selected people from a general pool of reviewers. The people in the review pool were from various educational fields and backgrounds and they were not selected based on expertise in the specific technology or program. Having said that, the e-mail that I got has four separate scores on it as well as comments from what appears to be a fourth review. I have an e-mail to the person in charge of the grant to find out if there was an extra review involved and what-have-you and I’ll update this post once I figure it out.
I won’t go into too much detail about scores – it involves a whole statistical analysis type thing that I’m pretty sure no one is interested in hearing about. If you are, by all means let me know!
Based on the four separate scores and comments from (what I think are) four separate people, it seems that two of the reviewers were in complete favor of my proposal, and the other two weren’t clear on what my goal was and reviewed in accordingly. For the time being I’m going to leave the fourth score and comments out until I understand exactly what role this person played (since I was under the impression it was only going to be three reviewers). Again, I’ll provide that information when I update this post after I find out more.
I’ll do my best to match raw scores and comments up so you’re seeing how the scores relate to the comments. So, here we go!
|Question (Score Range)||R1 Score||R2 Score||R3 Score|
|Overall quality of innovation (0-8)||5||6||7|
|Project narrative (0-8)||5||7||7|
|Project timeline and budget (0-4)||3||3||4|
|Assessment plan (0-6)||3||4||5|
|Communication plan (0-6)||3||6||5|
|Expertise level (0-4)||2||4||3|
|Adoption feasibility (0-8)||5||7||7|
|External funding or in-kind support (0-6)||4||5||5|
So there are the scores and the categories they were given in. The categories themselves are fairly self explanatory. As you can see, though, the scores from reviewers 2 and 3 were pretty good. Now, to put some more perspective on it with their comments. Reviewer 1’s comments will be last since I have the most to respond to there.
Reviewer 2 said,
Heigh, this can be patented and make loads of money. Earlier in my career, I worked on pre-cyber equivalents. There is no doubt that this will light up the sky and make money for SUNY. Macte anima ultima perfulget sola!
First, I’d like to say that I have no idea what the phrase at the end means. Anyone know? (Leave a comment below). Second, it was great to hear feedback like this from another educator that I’ve never even met before. In my mind the use of iPads in Paramedic education is a great one – it’s nice to see that I conveyed that!
Reviewer 3 commented,
The use of mobile technology, such as through iPads, can be very valuable for healthcare and emergency response scenarios, and also support the big idea for healthcare within the context of the power of SUNY plan.
Again – this reviewer got it. I’m happy to see that, at least to some degree, my proposal wasn’t completely outlandish and insane. That’s reassuring.
This is what reviewer 1 had to say:
The idea is sound. I would question the investment in hardware. Would it be better to develop an app (application) that students could download to their cell phone or their own iPad, android, portable PC or tablet? Also, if students use the iPads in the clinical environment (I imagine in an ambulance with real calls, etc, are they sturdy enough or can they be protected in this environment?). In the application the PI states institutional support – I did not see any supportive letters. The assessment section is vague. I am unsure exactly what is being assessed or what is being compared to demonstrate the value of the project. I gather the project will involve 20 or so students in their EMT program but I do not see any indication that the project has been discussed with any other SUNY schools or paramedic program directors to assess feasibility or support on a SUNY level. Additionally, all listed on the grant are EMT’s. The application development on a Lion based server is mentioned – who will do programming (if this is an app) or is it just an email based method of mailing info to an from students.
As frustrating as negative feedback can be, I also think it’s some of the most helpful. It often seems that when things go well or we are in favor of them we have a hard time explaining or providing evidence for why we think so, but when we don’t agree with an idea or aren’t in favor of it I think we’d all agree we have a much easier time explaining why. Though these comments ultimately contributed to my proposal not receiving funding I now know how I can revamp things and better explain my intentions.
That being said, it seems that there were parts of my proposal that this reviewer didn’t really understand and I think it’s because of that the scores from this reviewer were low. I’d almost like if there were a chance for those that applied for the grant to respond to these comments before the awards were decided. The purpose not being to change the proposal but to try to explain the goals in a different way so that the big picture could be better understood.
Reviewer 1 mentions developing an application instead of buying hardware – there are a few reasons that this shows me they didn’t understand why I wanted to use the funds for the purchase of iPads. EMS and education apps already exist, so why reinvent the wheel? The purpose of putting iPads in Paramedic Student’s hands is to bring learning to them; provide them with additional tools to access educational material and make their education more interactive. I don’t know how you could create one app to do this. It would also be reliant on the students having iPads or iPhones or Android phones or tablets. I’ve been in Paramedic school – 40 hours of class a week and 3 part-time jobs to make ends meet doesn’t provide the means to purchase this technology.
I’m also not sure why reviewer 1 mentions that there is no evidence I spoke to anyone from other schools or Paramedic programs. Based on my understanding of this grant the first phase (which is what this proposal was for) was meant to be a trial at one school within one program. Then, based on assessment and review, the second phase (which will be awarded only to a few of those that received funding for the first phase) is to bring the innovation to other programs and other schools. Maybe I should have gone more in-depth when I was explaining how it could be applied to other types of programs?
I will agree with reviewer 1, though, that my assessment section may have been a little weak. One of the issues I faced when determining how our program would assess the effectiveness of issuing our student’s iPads is that I haven’t been able to find evidence that any type of program has done something to the extent that I want to. It’s hard to know what standards to measure success against when you’re doing something that is completely different from anything that has been done before.
It sounds like this grant will be open for proposals on a yearly basis, so if it is I will certainly be revising my plan and thinking of how I can improve my proposal so that the true potential of what I want to do is clear. I know this is a good idea. I need to make sure the reviewers know it, too. In the mean time I’m searching feverishly for other ways to get funding. So far, I haven’t had much luck. Regardless, I’m determined to make this happen because it will be good for our students, our program, and our profession.
Update: Based on the response I received from my inquiry to the grant committee it seems that the mysterious fourth reviewer was from a member of the Provost’s staff. Based on the average of the scores from the peer reviews my grant proposal went to another level of review. The Provost review provided a score of 50 (:() and the following comment:
As presented, I do not recommend funding for this proposal. While I like the concept and the proposal presents some ideas about how i-Pad access might improve learning and success in the Paramedic Program, the curricular plan needs to be more clearly defined. Using a research model for assessment, e.g. control group, and/or adding a component that would support i-Pad interaction/communication with the hospitals would significantly strengthen the proposal. Considering the potential for replication, learning more about how new media (hardware and apps) can better support healthcare workers is essential.
I must say that I am happy to have learned that my proposal went on to what sounds like it was the final round of reviews, but I’m also disappointed that this is ultimately where the decision was made for my proposal to not receive funding. I will concede that my proposal should have a better defined plan and assessment, as I mentioned before. Knowing what I know now I am confident that, should I submit a modified version of this proposal in the future, they will not hesitate to recommend it for funding.
Until next time, mD.