Background: All personnel dealing with the birth of a newborn infant need to pass Neonatal Resuscitation Program (NRP) course .The validity of the certificate is for 2 years. Evaluation is done on various items learned at the course. All items are scored on scale of 0-2 on a form called megacode assessment form. Some items are mandatory and some non-mandatory. Bag and mask ventilation is one of the key skills acquired at the course. Objectives This study examined the predictive validity of the scores achieved in bag and mask ventilation skill during a formal neonatal resuscitation course and its impact over course of time. Methods This was a Prospective cohort study. 20 third year medical students were enrolled. Two simulated NRP scenarios over 6-8 months were offered and videotaped. Low fidelity manikins were similar to the one used in NRP course. Investigators and the independent rater (Neonatal Nurse Educator) were blinded to original scores. Video recordings scored on megacode forms by the independent rater. Total of 5 items scored, 3 items were mandatory and 2 non-mandatory. Original scores were then requested from the provincial NRP coordinator to compare with the study outcomes. Results The enrolled students had all similar original scores during NRP program (T1); everyone had 10/10 scores for bag and mask ventilation skill. R was constant and not calculated by SPSS. 5 students dropped out of the study. Pearsons r values were calculated for the first study session (T2) and second study session (T3). The 3 mandatory and 5 total bag and mask ventilation skill items were grouped and compared for the two study sessions. The sum of 3 mandatory items for the two study sessions showed Pearsons r of 0.299 for n=15, p value of 0.279, the sum of 5 total items showed Pearsons r of 0.188 and p value of 0.502 Analysis of variance (ANOVA) for pairwise comparison of the means was calculated.T1-T2 Anova for sum of 5 total items showed a very low p value of 0.000 whereas T1-T3 Anova for sum of 5 total items showed p Value of 0.001 T1-T2 as well as T1-T3 Anova for sum of 3 mandatory items showed very low p values of 0.000 Conclusion Most candidates lost BMV skills significantly within 6 months. Practice resuscitation (T2), improved the skill slightly but not significantly. The NRP scores on bag and mask ventilation does not have a good predictive value. More research is required to evaluate alternate ways to enhance retention of acquired skills at the NRP program.