One Broker, Two Paradigms: Real-Time MQTT Pub/Sub and Durable Queues, Natively in EMQX

The Architect's Dilemma

Every IoT architect faces the same fundamental challenge: your system needs to handle two fundamentally different messaging patterns simultaneously.

On one hand, you need instantaneous pub/sub for IoT device connectivity, think live dashboards showing vehicle locations, alerts firing when sensors detect anomalies, or operators watching production line metrics update every second.

On the other hand, you need durable queuing for enterprise integration: ensuring every transaction gets processed, every command reaches its destination (eventually), and every analytics job receives its complete dataset even when consumers go offline.

For years, this duality has forced architects into an uncomfortable compromise: stitch together an MQTT broker for edge connectivity and a separate enterprise message queue for backend reliability. The result? A fragmented architecture that's complex to build, expensive to run, and frustrating to maintain.

But what if a single, cohesive platform could handle both paradigms seamlessly? What if you didn't have to choose between real-time and reliable?

Let's explore how EMQX's unified architecture solves this decades-old dilemma.

The Old Way: A Segregated Architecture (EMQX + External MQ)

The Typical Setup

Here's what most IoT systems look like today:

The Workflow:

1. Devices connect to the EMQX cluster using MQTT
2. Real-time clients subscribe directly to EMQX topics for instant notifications
3. EMQX's rule engine or bridge forwards messages to the external message queue (RabbitMQ, Kafka, etc.)
4. Backend applications consume from the external MQ using AMQP, Kafka protocol, or proprietary APIs

Sounds reasonable? On paper, yes. In practice, this architecture creates serious pain points.

The Real Cost: A Case in Point

Consider a fleet management company operating at scale:

Infrastructure:

• EMQX cluster: 3 nodes handling 50,000 connected vehicles
• Kafka cluster: 5 nodes processing trip data and commands
• Total: 8 servers to manage, patch, and monitor

The Hidden Costs:

• $45K/year in additional cloud infrastructure (Kafka cluster + integration layer)
• 2 specialized teams required (MQTT experts + Kafka engineers)
• Average message latency increased by 45ms due to bridging overhead
• 3-day outage when the bridge service failed silently, messages were published to EMQX but never reached Kafka, causing billing discrepancies

This isn't a worst-case scenario. It's typical.

Why the Segregated Architecture Fails

Architectural Complexity
Two distinct systems to deploy, configure, and scale. Each has its own cluster management, replication strategy, and failure modes. Your ops team needs expertise in both systems.

Operational Overhead
Separate monitoring dashboards, alert systems, and backup procedures. When something breaks at 3 AM, which system failed? The broker? The bridge? The queue?

Latency and Failure Points
Every message destined for backend processing must traverse:

1. EMQX → Bridge service (network hop, serialization)
2. Bridge → External MQ (protocol conversion, network hop)
3. External MQ → Consumer (another protocol, another hop)

Each hop adds latency (typically 30-50ms total) and creates a potential point of failure.

Protocol Mismatch
Your edge developers work with MQTT clients. Your backend developers work with Kafka consumers or AMQP clients. Different APIs, different mental models, different debugging tools. Training new engineers takes months, not weeks.

Cost Accumulation
Infrastructure costs compound: you're paying for MQTT infrastructure + MQ infrastructure + integration layer + the operational overhead of managing all three.

The New Way: A Unified Architecture with EMQX

Simplified Architecture

Here's what the same system looks like with EMQX Message Queues:

The difference is striking: one system, two consumption patterns, zero external dependencies.

The Core Concept: Topic-to-Queue Mapping

EMQX's innovation lies in its ability to handle both real-time and asynchronous messaging within a single broker through intelligent topic-to-queue mapping.

How It Works:

When you create a message queue in EMQX, you bind it to a topic filter (including wildcards). From that point forward, every message published to matching topics flows through two parallel paths:

Path 1: Real-Time Pub/Sub (unchanged) Any client subscribed to the topic receives the message immediately, just like standard MQTT. Your dashboards, alerting systems, and real-time monitors work exactly as before.

Path 2: Durable Queue (new capability) The message is simultaneously persisted to durable storage and made available to queue consumers. Backend services subscribe to $q/{topic-filter} instead of the raw topic, gaining:

• Guaranteed delivery even if the consumer is offline
• Load balancing across multiple consumer instances
• Offset tracking so consumers can resume from where they left off
• Flexible dispatch strategies (round-robin, random, least-inflight)

Wildcard Mapping: The Architecture Multiplier

This is where EMQX's unified architecture becomes truly powerful: a single queue can aggregate messages from thousands of topics using wildcard patterns.

The Problem It Solves:

Imagine managing a smart city with 10,000 traffic sensors, each publishing to its own topic:

• sensor/001/data
• sensor/002/data
• ...
• sensor/10000/data

With EMQX Message Queue, one queue mapped to $q/sensor/+/data handles everything.

Comparing the Architectures: The Clear Winner

When placed side-by-side, the advantages of a unified platform become undeniable.

Feature	Segregated Architecture (EMQX + External MQ)	Unified EMQX Architecture
System Management	Complex: Two distinct distributed systems to deploy, monitor, and scale.	Simplified: One cohesive system to manage.
Infrastructure Cost	High: Double the server footprint, monitoring tools, and maintenance overhead.	Reduced: Consolidated infrastructure leads to lower TCO.
Data Latency	Higher: Incurs an extra network hop and processing delay at the bridge.	Lower: Direct dispatch between topic and queue.
Development Complexity	High: Requires knowledge of multiple protocols (MQTT, AMQP, etc.) and SDKs.	Low: A single protocol (MQTT) for both real-time and queueing clients.
System Resilience	Lower: More failure points (bridge, external MQ)	Higher: Fewer moving parts and a tightly integrated data path.
Security/ACLs	Duplicated policies	Single policy surface
Observability	Split metrics/traces	Unified metrics/traces
Change Management	Two release trains & upgrades	One coherent lifecycle

The unified architecture isn't just simpler—it's fundamentally more efficient, more reliable, and more cost-effective.

Migration Notes (Pragmatic Path Off an External MQ)

You don’t have to big-bang the transition.

1. Mirror first: Keep your existing Kafka/RabbitMQ consumers. In EMQX, declare queues on the same topic filters and point a subset of consumers to $q/....
2. Prove SLOs: Compare latency, error rates, and ops load between paths.
3. Retire bridges gradually: As services switch to $q/..., decommission the inter-broker connector and the external MQ footprint behind it.

Tip: Start with services that already use MQTT for control paths; they can adopt $q/... with minimal SDK churn.

When You Might Still Pair with Kafka (and How)

Some teams keep Kafka for long-term retention, batch analytics, or lake ingestion. That’s fine—treat it as an analytics boundary, not a runtime dependency for every device message:

• Use EMQX for device-facing pub/sub + queues (low latency, unified auth).
• Export curated streams to Kafka/Data Lake for archival and batch jobs.
• Keep apps on MQTT end-to-end to avoid dual-protocol complexity.

This keeps the operational heart of your IoT system inside one broker, while analytics live where they belong.

Conclusion: More Than a Broker, A Unified Messaging Platform

For years, architects were forced to make a trade-off: choose a lightweight MQTT broker for the edge or a heavy-duty message queue for the backend. The solution was often a complex and costly integration of both.

By natively integrating durable message queuing, EMQX eliminates this trade-off. It allows you to build a simpler, more cost-effective, and lower-latency architecture without sacrificing power or reliability. Your developers can now use the single MQTT protocol they already know to build both real-time dashboards and resilient, queue-based backend services.

This transforms EMQX from a best-in-class MQTT broker into a comprehensive, unified messaging platform that satisfies both edge and enterprise requirements in a single cluster.

Is your IoT architecture unnecessarily complex? It might be time to unify your messaging stack. Contact our experts to discuss how EMQX can streamline your system design.

Talk to an Expert

Contact Us →