In today's society, we are flooded with massive volumes of data in the order of a billion gigabytes on a daily basis from pervasive sensors. It is becoming increasingly challenging to locally store and transport the acquired data to a central location for signal/data processing (i.e., for inference). To alleviate these problems, it is evident that there is an urgent need to significantly reduce the sensing cost (i.e., the number of expensive sensors) as well as the related memory and bandwidth requirements by developing unconventional sensing mechanisms to extract as much information as possible yet collecting fewer data. The first aim of this thesis is to develop theory and algorithms for data reduction. We develop a data reduction tool called sparse sensing, which consists of a deterministic and structured sensing function (guided by a sparse vector) that is optimally designed to achieve a desired inference performance with the reduced number of data samples. The first part of this thesis is dedicated to the development of sparse sensing mechanisms and convex programs to efficiently design sparse sensing functions. We design sparse sensing functions under the assumption that the data is not yet available and the model information is perfectly known. Sparse sensing offers a number of advantages over compressed sensing (a state-of-the-art data reduction method for sparse signal recovery). One of the major differences is that in sparse sensing the underlying signals need not be sparse. This allows us to consider general signal processing tasks (not just sparse signal recovery) under the proposed sparse sensing framework. Specifically, we focus on fundamental statistical inference tasks, like estimation, filtering, and detection. In essence, we present topics that transform classical (e.g., random or uniform) sensing methods to low-cost data acquisition mechanisms tailored for specific inference tasks. The developed framework can be applied to sensor selection, sensor placement, or sensor scheduling, for example. In the second part of this thesis, we focus on some applications related to distributed sampling using sensor networks. Sensor networks can be used as a spatial sampling device, that is, to faithfully represent distributed signals (e.g., a spatially varying phenomenon such as a temperature field). On top of that, the distributed signals can exist in space and time, where the temporal sampling is achieved using analog-to-digital converters, for example. Each sensor has an independent sample clock, and its stability essentially determines the alignment of the temporal sampling grid across the sensors. Due to imperfections in the oscillator, the sample clocks drift from each other, resulting in the misalignment of the temporal sampling grids. To overcome this issue, we devise a mechanism to distribute the sample clock wirelessly. Specifically, we perform wireless clock synchronization based on the time-of-flight measurements of broadcast messages. In addition, clock synchronization also plays a central role in other time-based sensor network applications such as localization. Localization is increasingly gaining popularity in many applications, especially for monitoring environments beyond human reach, e.g., using robots or drones with several sensor units mounted on it. Consequently we now have to localize more than one sensor or even localize the whole sensing platform. Therefore, we extend the classical localization paradigm to localize a (rigid) sensing platform by exploiting the knowledge of the sensor placement on the platform. In particular, we develop algorithms for rigid body localization, i.e., for estimating the position and orientation of the rigid platform using distance measurements. Given the central role of sensing and sensor networks, the results presented in this thesis impacts a wide range of applications.