Although Byzantine Fault Tolerant (BFT) protocols such as HotStuff are nominally resistant to a number of faulty or unreliable participants, implementation or design errors can cause violations in their expected properties. Because of this, it is useful to have reliable automated testing frameworks that can simulate Byzantine behaviour to make bug detection easier. In this paper, we examine the performance of the ByzzFuzz BFT testing tool using our implementation of the HotStuff protocol. We describe the design choices necessary to create a working HotStuff implementation. Then we purposefully introduce implementation flaws to evaluate the behaviour of ByzzFuzz with different parameters and mutation scopes. We compare its performance to that of a baseline random fault injection scheduler. Our results show that it was able to detect the introduced bugs using either process or network faults. ByzzFuzz's partition-based network faults were more effective at detecting bugs than the 'Random' scheduler's network faults. For process faults, we were unable to register significant differences in performance possibly due to HotStuff's simplistic pipelined structure. In our tests, any-scope mutations performed better than their small-scope counterparts for the same configuration. This could be attributed to the nature of the selected faults and HotStuff's pipelined structure.

The reliability of Byzantine Fault Tolerant (BFT) consensus protocols is critical for the robustness of modern distributed systems, i.e., in blockchain technologies. Testing of BFT protocols is crucial, as consequences of faults in their implementation can lead to malicious users exploiting vulnerabilities, resulting in financial losses, data corruption, or system unavailability. Such incidents, as seen in real-world attacks on blockchain systems, underscore the need for rigorous testing methodologies to ensure protocol correctness and resilience under adverse conditions.

This paper evaluates the implementation of the Tendermint protocol in the ByzzBench framework using ByzzFuzz, a testing approach for BFT consensus protocols. ByzzFuzz introduces structured mutations to simulate real-world fault scenarios, enabling the identification of incorrect behavior. The main question addressed in this study is: Can ByzzFuzz detect subtle protocol faults more effectively than baseline testing methods, and how do mutation strategies influence fault detection performance?

Through extensive testing, ByzzFuzz successfully uncovered violations in the Tendermint implementation, demonstrating its capability to detect subtle protocol faults. A comparative analysis with baseline testing methods revealed that ByzzFuzz provides greater fault coverage, identifying nuanced issues that the baseline approach missed. Furthermore, the study evaluated the effectiveness of small-scope and any-scope message mutations, where they change a value incrementally and arbitrarily respectively. This study found that small-scope mutations perform better in finding faults.

Byzantine Fault Tolerant (BFT) protocols are designed to achieve consensus even in the presence of Byzantine faults. Although BFT protocols provide strong theoretical guarantees, bugs in the implementation of the protocols can allow for malicious activity. While previous work, like Twins and Tyr, has focused on alternative methods to test BFT protocols, our work builds upon ByzzFuzz, an automated testing algorithm, which has previously identified bugs in protocols like Tendermint and Ripple. Despite its success, its effectiveness has not yet been tested on speculative BFT protocols like hBFT. This research evaluates the effectiveness of ByzzFuzz in assessing the correctness and safety of hBFT. To address this, we implemented hBFT in ByzzBench, a comprehensive framework where BFT protocols can be evaluated using ByzzFuzz and other testing algorithms. Through testing, ByzzFuzz successfully uncovered a potential violation in hBFT and detected an injected bug in the hBFT implementation. However, detecting the known violation of hBFT required a controlled environment, highlighting areas where ByzzFuzz could be improved. The findings show that ByzzFuzz is effective at identifying bugs in hBFT, while also suggesting the need for improvement to enhance its robustness and adaptability.

Byzantine fault-tolerant protocols have been around for decades, offering the guarantee of agreement on a correct value even in the presence of arbitrary failures. These protocols have become a critical part of achieving consensus in distributed systems and are widely used nowadays. As such, we should aim to ensure the correct functioning of these systems and one essential step to take in this direction is by finding systematic and automatic ways to test BFT protocols. This paper evaluates the performance of ByzzFuzz, an automatic testing framework designed to find bugs in the implementation of Byzantine fault-tolerant protocols through randomized testing. In that sense, we evaluate ByzzFuzz’s ability to find bugs in our implementation, compare its method of injecting network and process faults to a baseline method that arbitrarily injects faults and compare the performance in bug detection of small-scope and any-scope message mutations. We implemented the ”Fast Byzantine Consensus” protocol and employed ByzzFuzz to evaluate the framework’s capability of finding implementation bugs. We materialized a liveness violation previously uncovered in a theoretical analysis research.

Testing Byzantine Fault Tolerant (BFT) algorithms is crucial in uncovering potential liveness and safety violations for distributed systems. This pa- per focuses on testing Zyzzyva with ByzzFuzz and Twins and evaluating their performance with each other and a baseline testing strategy. We also inves- tigate if ByzzFuzz can uncover faults in Zyzzyva, and how small-scope mutations compare to any- scope mutations. We also discuss limitations with ByzzFuzz when it comes to testing BFT protocols. We find that ByzzFuzz is currently unable to find known safety violations in Zyzzyva, but can find injected violations and Twins does not find viola- tions given our small sample size.