Would a bad freshman year in high school have a large effect on my chances at the Ivy League Universities?

not if you got inside connections or if you have family or friends or know a professor / staff member at an ivy league school.