Rank detection is crucial in array processing applications, as many algorithms rely on accurately estimating the rank of the data matrix to ensure optimal performance. Under Gaussian white noise, rank can be detected through eigenvalue analysis. However, in arbitrary noise, prewhitening the data matrix with the noise covariance matrix is necessary, and rank detection is achieved by examining the generalized eigenvalues. Existing methods often assume the noise covariance structure or require a large number of noise samples. This thesis focuses on addressing the rank detection problem in scenarios with limited noise samples and arbitrary noise environments.

Firstly, we investigate the largest generalized eigenvalue threshold for the prewhitened data sample covariance matrix according to the random matrix theory. We develop a rank detection algorithm based on the threshold via a sequential test, and provide the performance analysis. A series of simulations demonstrate its superiority over conventional methods such as Minimum Description Length (MDL) and Akaike's Information Criterion (AIC).

Secondly, since the Short-time Fourier Transform (STFT) is commonly used for non-stationary signal analysis, we extend our rank detection method to the STFT domain. The correlations introduced by the STFT have a significant impact on the distribution of the noise. Therefore, we develop a technique to remove correlations among time-frequency bins based on exact expressions of these correlations. After successfully eliminating these correlations, our proposed rank detection method achieves enhanced reliability and performance in the STFT domain.

Lastly, we evaluate the effectiveness of our rank detection method in speech enhancement applications. Simulations confirm that utilizing the estimated rank improves speech quality compared to using the known number of sources.