To increase mission return, utilizing two or more spacecraft instead of one may sometimes be superior. This is especially true when a large spaceborne instrument needs to be created through larger and configurable baselines, such as telescopes and interferometers. However, coordinating the alignment of the individual components of such a spaceborne instrument on separate spacecraft (involving the estimation and control of baselines) will require a high level of accuracy for relative navigation and control. The increasing demand of such science missions or challenges on complex functions such as rendezvous and docking calls for high accuracy levels of ranging at centimeter or even millimeter levels. The objective of this research is to investigate key technologies of developing a relative navigation system based on radio-frequency (RF) metrology. This RF-based system inherits Global Navigation Satellite System (GNSS) technologies through transmission and reception of locally generated GNSS-like pseudo random noise (PRN) ranging codes and carrier phases via inter-satellite links. This enables operation, e.g., in high Earth orbits where GNSS constellations are poorly visible. The RF-based navigation system is designed to comprise one transmitter, one receiver and several antennas in order to enable coarse-mode inter-satellite distance estimation (meter level) based on pseudorange measurements and fine-mode distance (centimeter level) and line-of-sight (LOS) estimation (sub-degree level) based on carrier phases in addition to pseudorange. A benchmarking system, called the Formation Flying Radio Frequency (FFRF) sensor, has been successfully shown and demonstrated on PRISMA mission. This research improves the performance of FFRF with respect to the technologies 1) to deal with errors and uncertainties, especially multipath; 2) to perform an unaided, fast and reliable carrier phase integer ambiguity resolution (IAR); and 3) to share channels among multiple spacecraft. Multipath In space applications, receivers on space vehicles may suffer from very short- delay multipath (< 4 m), reflected from the vehicle itself or from other vehicles during the operations of rendezvous and docking. The thesis proposes a novel method, termed "Multipath Envelope Curve Fitting", to mitigate very-short-delay-multipath on pseudorange measurements by approximately 50%. It also exhibits a promising performance for medium or large delayed multipath as compared to state-of-the-art methods. The method is based on the fact that the signal strength information, reported by early or late correlators inside the receiver, has an in-phase correlation with the pseudorange multipath error. By linearly combining multiple signal strength estimators from multiple correlators, the pseudorange multipath error has been accurately estimated. The weights for the linear combination were obtained by curve fitting based on the least-squares adjustment. A simple implementation strategy was also proposed that enables a receiver-internal multipath estimation process operated in conjunction with the tracking loop with a minimal additional computational overhead. Compared to the pseudorange multipath, the carrier phase multipath has more significant impacts on high precision navigation, especially when it is coupled with the carrier phase IAR. By making use of the signal to noise ratio (SNR) data of multiple antennas, this thesis proposes a novel cascaded extended Kalman Filter (EKF) to mitigate carrier phase multipath. This method accelerates the IAR process significantly and guarantees an achievement of sub-degree LOS accuracy. Both real-valued and complex-valued EKF are proposed and evaluated. The complex-valued EKF has been found to be insensitive to poorly defined initial conditions, when the real-valued EKF has difficulties converging. Moreover, the complex-valued EKF has shown better convergence properties for SNR observations with a large amount of noise. Integer Ambiguity Resolution The second challenge of this research is to perform an unaided, fast and reliable carrier phase IAR. Single-epoch IAR algorithms are proposed in this thesis, by making use of a nonlinear quadratic LOS length constraint and taking advantages of antenna arrays. Two methods, namely, the validation method and the subset ambiguity bounding method, are proposed. They replace the equality quadratic constraint by inequality boundaries such that the well known Least-squares AMBiguity Decorrelation Adjustment (LAMBDA) integer ambiguity resolution process is implemented within a pre-defined threshold to increase the integer search fidelity. Numerical simulations and field tests demonstrated that both the validation method and the subset ambiguity bounding method provided remarkable improvements with up to 80% higher success rates than the original LAMBDA method based on single-epoch measurements. The validation method showed a slightly better performance than the subset ambiguity bounding method as they differ in utilizing all-ambiguity-set and subset-ambiguity, respectively. Better IAR robustness against multipath can also be observed as compared to the original LAMBDA method. An Ambiguity Dilution of Precision (ADOP) measure under the LOS constraint is derived, which is an easy-to-use and insightful indicator of the ambiguity resolution capability. A rule-of-thumb for the pre-defined threshold has also been derived in the closed-form expression, providing guidance on how to choose boundaries according to the noise level and antenna geometry. Multiple Access Technology Enabling multiple access capability is of critical importance for future missions with four or more spacecraft. The Code Division Multiple Access (CDMA) technology is recommended to be used in combination with a flexible role rotating topology in this research. This allows coping with time-critical relative navigation requirements and enables flexible operations during various mission phases. Through realistic formation case studies, the limitation of CDMA was extensively investigated in terms of the multiple access interference (MAI) which could result in a ranging error of several meters and is highly dependent on the Doppler offset. Recommendations are given in this thesis to reduce corresponding MAI errors.