An autonomous vehicle should be able to operate amidst numerous other human-driven vehicles, each driving on its own trajectory. To safely navigate in such a dynamic environment, the autonomous vehicle should be able to predict trajectories of the vehicles operating in its vicinity and use these to plan its own path. Most related work uses a vehicle's past trajectory to model its behavior, based on which the future trajectory is predicted. However, they do not focus on the influence of contextual features such as road structure from the scene that may affect the vehicle's future trajectory. This work proposes an approach to predict a long-term vehicle trajectory using not only the past trajectory of a vehicle but also contextual features from the driving scene. We model the road structure to help prediction on curved road sections. A Recurrent Neural Network is used to learn vehicle behavior from past vehicle trajectories and predict future trajectories while incorporating road structure. Using a trajectory dataset collected from a test vehicle, we compare our model's performance with the conventional prediction approach based on only past vehicle trajectory.