The emergence of blockchain technologies has created the possibility of transforming business processes in the form of immutable agreements called smart contracts. Smart contracts suffer from a major limitation; they cannot authenticate the trustworthiness of real-world data sources, creating the need for intermediaries called oracles. Oracles are trusted entities that connect on-chain systems with off-chain data, allowing smart contracts to operate on real-world inputs in a trustworthy manner. A popular oracle protocol is a crowdsourced oracle, where unrelated individuals attest to facts through voting mechanisms in smart contracts. Crowdsourced oracles have unique challenges: the trustworthiness and correctness of outcomes cannot be explicitly verified. These problems are aggravated by inherent vulnerabilities to attacks, such as Sybil attacks. To address this weakness, this paper proposes a reputation-based mechanism, where oracles are given a reputation value depending on the implied correctness of their actions over time. This reputation score is used to eliminate malicious agents from the participant pool. Additionally, two reputation-based voting mechanisms are proposed. The effectiveness of the proposed mechanism is evaluated using an agent-based simulation of a crowdsourced oracle platform, where a pool of oracles performs evaluate Boolean queries.