Improve Conduit Load Distribution
Currently, Conduits use the channel ID as the basis for hashing, even for user-related events like channel.chat.message. While this may simplify internal handling, it can lead to uneven load distribution—especially when a single large channel generates a high volume of messages. This results in one shard being overloaded while others remain underutilized.
Feature Request:
Add support for configurable hashing strategies per Conduit, allowing developers to choose the most suitable approach based on their use case. Suggested options include:
user_id – For user-centric events, to spread load more evenly across users.
channel_id – Maintain current behavior for channel-focused workloads.
fully random – Distribute events entirely at random for maximum load spread.
Providing this flexibility would help developers better balance workloads, especially in high-traffic environments, and reduce the need for implementing custom rebalancing logic.
Would love to see this considered as a future improvement!
DeemonRider
shared this idea