Floating structures oscillate in waves, where these wave-induced motions may be critical for various marine operations. An important consideration is thereby given to the sea states at the planning and operating stages for an offshore project. The most important information extracted from a sea state is the directional wave spectrum, indicating wave direction, significant wave height, and wave spectrum peak period. Among several available methods of measuring and estimating the directional wave spectrum, the wave buoy analogy technique based on vessel motion responses is an in situ and almost real-time solution without extra costs of devices. If the forms of the wave spectra are not predefined in the estimation, the method is called a nonparametric approach. Its most remarkable advantage is the flexible form, but the smoothness should be regulated. After the discrete Fourier transform has been applied to the measured vessel motions, smoothing is necessary. However, this process results in disturbed vessel cross-spectra and a lowpass characteristic of the windowing function. This paper presents a nonparametric approach for directional wave spectrum estimation based on vessel motion responses. It introduces novel smoothness constraints using Bézier surface and includes a more robust estimate using L1 optimization. Both techniques are applied to the wave buoy analogy for the first time. Numerical simulations are conducted to verify the proposed algorithm.