What Is CrimsonQ
Publish messages to multiple consumers with persistent queues.
Last updated
Was this helpful?
Publish messages to multiple consumers with persistent queues.
Last updated
Was this helpful?
CrimsonQ is a new twist to Message Queue systems built on Go. It merges the idea of pub/sub with the ability to have queues for the consumers of pub/sub. This provides the ability to broadcast a message from producers to certain topics and then consumers will pickup the messages that target them, adding them to a message queue that is specific to this consumer. The broadcasting can be done on topics similar to MQTT and the consumer can subscribe to topics that can also contain wildcards similar to MQTT ex: category/+/myInbox/#.
Once a message is received by the consumer, it will be placed in its own consumer queue. The queue then labels each message with a status as below;
Pending: Where the message is now in the queue and pending its turn to be pulled by the consumer
Active: It has been pulled from the queue by the consumer and is being used by the consumer. It will remain in active state until the consumer tells the queue that this message has been completed or failed. If it remains in active for longer than the defined time it will move to delayed. *The defined time is in the configuration settings file, see configuration for details
Delayed: The message is moved to this status if it remains in the active state for longer than defined time.
Failed: A message is marked as failed if it either remained longer than the defined time in the delayed state or if it was marked by the consumer as failed.
Completed: This is the message final state which means it was pulled, active and completed properly.
Merging both the ability to Publish messages and receive them in persistent queues gives the ability for mission critical systems to communicate without the worry of missing out on important published messages, also keeping a queue of all the items that need to be executed if the consumer ever goes offline.
The illustration below explains the producer pushing messages to a specific topic and show cases how the consumers will receive them in their internal queues.
In many cases you might have multiple microservices that need to be orchestrated. Relying only on pub/sub models means that if the consumer is offline or dysfunctional it will miss critical messages. To avoid such a scenario CrimsonQ allows you to publish messages to multiple consumers at once, and each will then manage its queue. If messages have not been consumed by the consumer and flagged as executed properly, it will remain in the consumers queue until the issue is resolved, losing no data, and providing an easy restart.
If you are dealing with critical IoT messages, that you can not afford to lose, CrimsonQ will allow you to capture all the messages in a queue, so if the IoT device is back online it will be able to pull the messages and act on them as needed.
