Trajectory prediction forecasts nearby agents' moves based on their historical trajectories. Accurate trajectory prediction (or prediction in short) is crucial for autonomous vehicles (AVs). Existing attacks compromise the prediction model of a victim AV by directly manipulating the historical trajectory of an attacker AV, which has limited real-world applicability. This paper, for the first time, explores an indirect attack approach that induces prediction errors via attacks against the perception module of a victim AV. Although it has been shown that physically realizable attacks against LiDAR-based perception are possible by placing a few objects at strategic locations, it is still an open challenge to find an object location from the vast search space in order to launch effective attacks against prediction under varying victim AV velocities. Through analysis, we observe that a prediction model is prone to an attack focusing on a single point in the scene. Consequently, we propose a novel two-stage attack framework to realize the single-point attack. The first stage of prediction-side attack efficiently identifies, guided by the distribution of detection results under object-based attacks against perception, the state perturbations for the prediction model that are effective and velocity-insensitive. In the second stage of location matching, we match the feasible object locations with the found state perturbations. Our evaluation using a public autonomous driving dataset shows that our attack causes a collision rate of up to 63% and various hazardous responses of the victim AV. The effectiveness of our attack is also demonstrated on a real testbed car ¹. To the best of our knowledge, this study is the first security analysis spanning from LiDAR-based perception to prediction in autonomous driving, leading to a realistic attack on prediction. To counteract the proposed attack, potential defenses are discussed.

Autonomous driving systems rely on precise trajectory prediction for safe and efficient motion planning. Despite considerable efforts to enhance prediction accuracy, inherent uncertainties persist due to data noise and incomplete observations. Many strategies entail formalizing prediction outcomes into distributions and utilizing variance to represent uncertainty. However, our experimental investigation reveals that existing trajectory prediction models yield unreliable uncertainty estimates, necessitating additional customized calibration processes. On the other hand, directly applying current calibration techniques to prediction outputs may yield suboptimal results due to using a universal scaler for all predictions and neglecting informative data cues. In this paper, we propose Customized Calibration Temperature with Regularizer (CCTR), a generic framework that calibrates the output distribution. Specifically, CCTR 1) employs a calibration-based regularizer to align output variance with the discrepancy between prediction and ground truth and 2) generates a tailor-made temperature scaler for each prediction using a post-processing network guided by context and historical information. Extensive evaluation involving multiple prediction and planning methods demonstrates the superiority of CCTR over existing calibration algorithms and uncertainty-aware methods, with significant improvements of 11%-22% in calibration quality and 17%-46% in motion planning.

As one of the crucial sensors for environment sensing, frequency modulated continuous wave (FMCW) radars are widely used in modern vehicles for driving assistance/autonomous driving. However, the limited frequency bandwidth and the increasing number of equipped radar sensors would inevitably cause mutual interference, degrading target detection and producing safety hazards. In this paper, a deep learning-based interference mitigation (IM) approach is proposed for FMCW radars by using the dilated convolution for network construction and a designated contrast learning strategy for training. The dilated convolution enlarges the receptive field of the neural network, and the designated contrastive learning strategy enforces to distinguish better between interferences and desired signals. The results of numerical simulation and experimental data processing show that the dilated convolution-based IM network, compared to the traditional convolution-based ones, can achieve a higher Signal-to-Interference-plus-Noise-Ratio (SINR) and target detection rate. Moreover, the designated contrastive learning strategy enables a better and more stable IM performance without increasing the complexity of the network, which can facilitate faster signal processing.

Multi-modal fusion has shown initial promising results for object detection of autonomous driving perception. However, many existing fusion schemes do not consider the quality of each fusion input and may suffer from adverse conditions on one or more sensors. While predictive uncertainty has been applied to characterize single-modal object detection performance at run time, incorporating uncertainties into the multi-modal fusion still lacks effective solutions due primarily to the uncertainty's cross-modal incomparability and distinct sensitivities to various adverse conditions. To fill this gap, this paper proposes Uncertainty-Encoded Mixture-of-Experts (UMoE) that explicitly incorporates single-modal uncertainties into LiDAR-camera fusion. UMoE uses individual expert network to process each sensor's detection result together with encoded uncertainty. Then, the expert networks' outputs are analyzed by a gating network to determine the fusion weights. The proposed UMoE module can be integrated into any proposal fusion pipeline. Evaluation shows that UMoE achieves a maximum of 10.67%, 3.17%, and 5.40% performance gain compared with the state-of-the-art proposal-level multi-modal object detectors under extreme weather, adversarial, and blinding attack scenarios.

We present a deep learning-based approach called DipSAR for reconstructing millimeter-wave synthetic aperture radar (SAR) images from sparse samples. The primary challenge lies in the requirement of a large training dataset for deep learning schemes. To overcome this issue, we employ the deep image prior (DIP) technique, which eliminates the need for a large dataset and instead utilizes only the sparse sample itself. Our proposed DipSAR model recovers missing samples from sparse data and reconstructs the SAR image using a conventional method. In this study, we utilize an existing SAR dataset and create fourteen different patterns to generate additional sparse samples by removing certain data points. We then evaluate the performance of DipSAR in comparison to the conventional method. The results show that DipSAR outperforms the conventional method in terms of the intersection over union (IoU) score.

Driver Behavior Modeling (DBM) aims to predict and model human driving behaviors, which is typically incorporated into the Advanced Driver Assistance System to enhance transportation safety and improve driving experience. Inverse reinforcement learning (IRL) is a prevailing DBM technique with the goal of modeling the driving policy by recovering an unknown internal reward function from human driver demonstrations. However, the latest IRL-based design is inefficient due to the laborious manual feature engineering processes. Besides, the reward function usually experiences increased prediction errors when deployed for unseen vehicles. In this paper, we propose a novel deep learning-based reward function for IRL-based DBM with efficient model personalization via machine unlearning. We evaluate our approach on a highway simulation constructed using the realistic human driving dataset NGSIM. We deploy our approach on both a server GPU and an embedded GPU. The evaluation results show that our approach achieves a higher prediction accuracy compared with the latest IRL-based DBM approach that uses a weighted sum of trajectory features as the reward function. Our model personalization method obtains the highest accuracy and lowest latency compared with the baselines.

Achieving high azimuth resolution is one of the main bottleneck for automotive radars, which generally demands a large aperture of antenna array. However, building an automotive radar system with a large antenna array is a very challenging task from the perspective of both technological readiness and cost. To circumvent this problem, we propose to fuse signals from multiple small automotive radars placed over the facade of a car as an alternative solution with low system complexity, where each radar with a small Multiple-Input Multiple-Output (MIMO) array operate independently without accurate synchronization. To (partially) coherently process the measurements from all the radars, a 2-D MUltiple Signal Classification (MUSIC) based algorithm is proposed for joint Direction-of-Arrival (DOA)-range estimation of targets in which spatial smoothing technique is exploited to tackle highly correlated signals. Taking advantage of the proposed estimation approach and multiple radars, it significantly improves the azimuth resolution of the system compared to that of a single MIMO radar. The performance of the proposed method is demonstrated through both numerical simulations and experimental results.

Due to the extensive usage of automotive radars on vehicles, mutual interference among radars on the road is becoming considerable. To address this, we propose a time domain strategy based on deep reinforcement learning (DRL). This approach helps avoid mutual interference for automotive radars in the time domain without extra communications. The numerical simulation results demonstrate that the proposed approach can avoid interference as effectively as frequency hopping. Moreover, the time domain strategy has more advantages than frequency hopping when encountering dynamic interference.

In this paper, constant false alarm rate (CFAR) detector-based approaches are proposed for interference mitigation of Frequency modulated continuous wave (FMCW) radars. The proposed methods exploit the fact that after dechirping and low-pass filtering operations the targets' beat signals of FMCW radars are composed of exponential sinusoidal components while interferences exhibit short chirp waves within a sweep. The spectra of interferences in the time-frequency (t-f) domain are detected by employing a 1-D CFAR detector along each frequency bin and then the detected map is dilated as a mask for interference suppression. The proposed approaches are applicable to the scenarios in the presence of multiple interferences. Compared to the existing methods, the proposed methods reduce the power loss of useful signals and are very computationally efficient. Their interference mitigation performances are demonstrated through both numerical simulations and experimental results.

In this paper, the interference mitigation for Frequency Modulated Continuous Wave (FMCW) radar system with a dechirping receiver is investigated. After dechirping operation, the scattered signals from targets result in beat signals, i.e., the sum of complex exponentials while the interferences lead to chirp-like short pulses. Taking advantage of these different time and frequency features between the useful signals and the interferences, the interference mitigation is formulated as an optimization problem: a sparse and low-rank decomposition of a Hankel matrix constructed by lifting the measurements. Then, an iterative optimization algorithm is proposed to tackle it by exploiting the Alternating Direction of Multipliers (ADMM) scheme. Compared to the existing methods, the proposed approach does not need to detect the interference and also improves the estimation accuracy of the separated useful signals. Both numerical simulations with point-like targets and experiment results with distributed targets (i.e., raindrops) are presented to demonstrate and verify its performance. The results show that the proposed approach is generally applicable for interference mitigation in both stationary and moving target scenarios.

Nowadays deep learning-based weather radar echo extrapolation methods have competently improved nowcasting quality. Current pure convolutional or convolutional recurrent neural network-based extrapolation pipelines inherently struggle in capturing both global and local spatiotemporal interactions simultaneously, thereby limiting nowcasting performances, e.g., they not only tend to underestimate heavy rainfalls' spatial coverage and intensity but also fail to precisely predict nonlinear motion patterns. Furthermore, the usually adopted pixel-wise objective functions lead to blurry predictions. To this end, we propose a novel motion-guided global-local aggregation Transformer network for effectively combining spatiotemporal cues at different time scales, thereby strengthening global-local spatiotemporal aggregation urgently required by the extrapolation task. First, we divide existing observations into both short- and long-term sequences to represent echo dynamics at different time scales. Then, to introduce reasonable motion guidance to Transformer, we customize an end-to-end module for jointly extracting motion representation of short- and long-term echo sequences (MRS, MRL), while estimating optical flow. Subsequently, based on Transformer architecture, MRS is used as queries to retrospect the most useful information from MRL for an effective aggregation of global long-term and local short-term cues. Finally, the fused feature is employed for future echo prediction. Additionally, for the blurry prediction problem, predictions from our model trained with an adversarial regularization achieve superior performances not only in nowcasting skill scores but also in precipitation details and image clarity over existing methods. Extensive experiments on two challenging radar echo datasets demonstrate the effectiveness of our proposed method.

Elevation resolution is an important indicator in tomographic SAR imaging as it represents the ability to discriminate closed targets in elevation. In general, the elevation resolution is proportional to the length of the elevation aperture. However, as the elevation aperture increases, the geometric consistency of the image will undesirably deteriorate and hence fails the image coregistration approach required by the traditional super-resolution tomographic imaging. In this paper, a new super-resolution tomographic imaging method is proposed to overcome the inconsistency problem caused by the large elevation aperture. The core strategy is to get rid of two-dimensional image coregistration by applying a three-dimensional (3D) back projection like imaging manner: the 3D space is firstly divided into a 3D imaging grid, each of which is individually imaged via compressive sensing for super-resolution. The effectiveness of the proposed approach is evaluated by both computer simulations and real P-band UAV SAR data.

Recently, frequency-modulated continuous-wave (FMCW) radar-based hand gesture recognition (HGR) using deep learning has achieved favorable performance. However, many existing methods use extracted features separately, i.e., using one of the range, Doppler, azimuth, or elevation angle information, or a combination of any two, to train convolutional neural networks (CNNs), which ignore the interrelation among the 5-D time-varying-range-Doppler-azimuth-elevation feature space. Although there have been methods using the 5-D information, their mining of the interrelation among the 5-D feature space is not sufficient, and there is still room for improvements. This article proposes a new processing scheme of HGR based on 5-D feature cubes that are jointly encoded by a 3-D fast Fourier transform (3-D-FFT)-based method. Then, a CNN is proposed by building two novel blocks, i.e., the spatiotemporal deformable convolution (STDC) block and the adaptive spatiotemporal context-aware convolution (ASTCAC) block. Concretely, STDC is designed to cope with hand gestures' large spatiotemporal geometric transformations in the 5-D feature space. Moreover, ASTCAC is designed for modeling long-distance global relationships, e.g., relationships between pixels of the feature at the upper left corner and lower right corner, and exploring the global spatiotemporal context, in order to enhance the target feature representation and suppress interference. Finally, our presented method is verified on a large radar dataset, including 19 760 sets of 16 common hand gestures, collected by 19 subjects. Our method obtains a recognition rate of 99.53% on the validation dataset and that of 97.22% on the test dataset, which is significantly better than state-of-the-art methods.

With the substantial increase of the FMCW radars used for autonomous driving and other applications in the area of surveillance, mutual interference has become a major concern. Recently, Deep Learning (DL) models have been used in FMCW radar interference mitigation with great success, but no research has been conducted in processing the time-frequency (t-f) maps of acquired beat signals. Considering the different distributions of useful beat signals and interferences in the t-f domain, a fully convolutional network (FCN) is proposed to suppress the interference and noise in the t-f spectrum obtained by the short-time Fourier transform (STFT) algorithm. The experimental results on the simulated radar signals show that the proposed FCN provides superior interference suppression with few parameters. Moreover, the qualitative results on the measured radar signals collected in real-world scenarios emphasize the excellent generalization capacity of the model. Finally, we show that our proposed approach achieves the best performance compared to state-of-the-art techniques.

A novel matrix-pencil (MP)-based interference mitigation approach for frequency-modulated continuous-wave (FMCW) radars is proposed in this article. The interference-contaminated segment of the beat signal is first cut out, and then, the signal samples in the cutout region are reconstructed by modeling the beat signal as a sum of complex exponentials and using the MP method to estimate their parameters. The efficiency of the proposed approach for the interference with different parameters (i.e., interference duration, signal-to-noise ratio (SNR), and different target scenarios) is investigated by means of numerical simulations. The proposed interference mitigation approach is intensively verified on experimental data. Comparisons of the proposed approach with the zeroing and other beat-frequency interpolation techniques are presented. The results indicate the broad applicability and superiority of the proposed approach, especially in low SNR and long interference duration situations.

Radio frequency interference, which makes it difficult to produce high-quality radar spectrograms, is a major issue for micro-Doppler-based human activity recognition (HAR). In this paper, we propose a deep-learning-based method to detect and cut out the interference in spectrograms. Then, we restore the spectrograms in the cut-out region. First, a fully convolutional neural network (FCN) is employed to detect and remove the interference. Then, a coarse-to-fine generative adversarial network (GAN) is proposed to restore the part of the spectrogram that is affected by the interferences. The simulated motion capture (MOCAP) spectrograms and the measured radar spectrograms with interference are used to verify the proposed method. Experimental results from both qualitative and quantitative perspectives show that the proposed method can mitigate the interference and restore high-quality radar spectrograms. Furthermore, the comparison experiments also demonstrate the efficiency of the proposed approach.

3-D imaging with irregular planar multiple-input-multiple-output (MIMO) arrays is discussed. Due to signal acquisition on irregular spatial sampling grids by using these antenna arrays, the fast Fourier transform (FFT)-based imaging algorithms cannot readily be used for image formation. To avoid the application of computationally intensive coherent summation algorithms such as filtered backprojection or Kirchhoff migration, we propose a nonuniform FFT (NUFFT)-based MIMO Range Migration Algorithm (i.e., NUFFT-based MIMO-RMA) for efficient microwave imaging. The algorithm exploits NUFFT to reconstruct the wavenumber-domain spectra related to each Fourier frequency. It is generic and applicable to 3-D imaging with irregular planar MIMO arrays. The effects of irregular spatial sampling and signal bandwidth on the imaging performance and computational efficiency of the proposed algorithm are analyzed. Finally, some numerical simulations and experimental results are presented to demonstrate the performance of the proposed imaging algorithm.

In this paper, we propose an Elevation-Radial scanned Synthetic Aperture Radar (E-RadSAR) for forward-looking ground penetrating radar (GPR) imaging. The E-RadSAR exploits the advantages of both RadSAR and Elevation-Circular SAR (E-CSAR) by utilizing the SAR technique in the cross- and down-range directions for signal acquisition. It could be implemented with fewer antennas compared to the RadSAR but provides higher spatial resolutions than that of E-CSAR. These features make it very attractive for space-and/or cost-constrained imaging applications, for instance, the GPR systems used for tunnel boring machines (TBM). However, the E-RadSAR synthesizes a three-dimensional (3-D) array by taking measurements in a volume, which makes the traditional sampling criterion no longer applicable for its sampling strategy design. To tackle 3-D (synthetic) array sampling/design, we formulate it as a sensor selection problem and suggest an efficient selection algorithm, i.e., modified clustered FrameSense (modified CFS). Then it is used for 3-D array sampling design. The imaging performances of the resultant near-optimal 3-D arrays are demonstrated through numerical simulations.