Autonomous Surface Vessels (ASVs) have various applications in the maritime sector. However, considerable challenges need to be met before integrating the applications in the current environment. Moreover, infrastructures in ports and waterways need to be upgraded to satisfy the requirements for ASV applications. To have an insight into the potential of ASVs in ports, we analyze the applications of autonomous vessels and the impacts that these applications have on port infrastructures. Future scenarios of the application of ASVs in ports are determined based on the analysis of the following three aspects: ASV applications, ASV technology development, and port infrastructure development. To indicate the development of ASV technology and port infrastructures, the Technology Readiness Level (TRL) is employed. Eleven scenarios of ASVs in ports have been identified based on the analysis.