A desirable feature of distributed file systems is to enable clients to mount
external namespaces, allowing them to access physically remote data as if it was
transparently stored in the local cluster. This feature is desirable in a variety of
scenarios, such as in enterprise hybrid-cloud deployments, where some data may be stored remote on the cloud while other in the local file system. This feature introduces questions as how to design, implement, and handle various file system semantics, such as different consistency levels and operational performance between the remote and local storage. For instance, a POXIS-complient block based file system is strongly consistent and does not pose the same semantics as a more relaxed eventual consistent object-based cloud storage. Further, it is interesting to benchmark how such an integrated remote storage solution performs and what potential speedups that might be possible.

To examine such questions and provide the remote storage feature in HopsFS,
a scale-out metadata HDFS distribution, this thesis presents an Proof-of-Concept (PoC) implementation that enables the local file system to be transparently integrated with an external remote storage. The PoC is storage provider agnostic and enables a HopsFS client to on-demand pull in and access remote data from an external storage into the local file system.

Currently, the solution works by mounting a snapshot of a remote storage into
HopsFS, a read-only approach. For write and synchronization work, frameworks
that layers stronger consistency models on top of S3 may be used, such as s3mper.
Performance of the implemented feature has been benchmarked against remote
Amazon S3 buckets, showing results of fast throughput for scanning and persisting metadata, but slower (4 Mb/s) for pulling data into the cluster through the Amazon cloud connector S3A. To improve performance, caching by replication of the remote files onto local disk storage has shown to drastically increase the read speed (up to 100x). The work done in this thesis shows a promising PoC that successfully serves remote blocks into a local file system, with future work including supporting writes, improved caching mechanisms, and reflecting off-band changes between the remote and local storage.