How it works

One queue engine, five protocols, the official client libraries unmodified, and a tenant per random identifier.

What PowerQueue is

PowerQueue is a message broker written in Rust. It has one queue engine, with its own storage, and five protocol front ends on top of it:

• AMQP 0-9-1, the protocol of RabbitMQ and its client libraries;
• Amazon SQS, over the AWS JSON and Query protocols that the AWS SDKs and CLI use;
• Google Cloud Pub/Sub, over the gRPC API the Google Cloud client libraries use;
• Azure Queue Storage, over its REST API;
• Azure Service Bus, over AMQP 1.0 with its management operations, plus its administration REST API.

Each front end answers the way the real service answers, including its error codes, so the official SDKs work unmodified: you change only the endpoint and the credentials. Under every protocol the messages go into the same engine's queues; a Pub/Sub subscription, an SQS queue and a Service Bus queue are all queues of that engine.

Within a tenant the protocols share one set of queue names: a queue called orders is the same queue whether you reach it over AMQP 0-9-1, SQS, Azure Queue Storage or Service Bus, and a Pub/Sub subscription called orders is that queue too. Give each protocol its own names: in particular, a publish to a Pub/Sub topic X that does not exist yet is refused with FAILED_PRECONDITION when another protocol already has a queue called X, because the topic's subscription X could not be created; create the topic and a subscription explicitly, or use another name.

This site is a free public deployment of PowerQueue, run in its public mode: many users, none with an account, share one broker, each in a tenant of their own, within caps that keep any one client from using more than a fair share.

Tenants and the identifier

A tenant is a private space of queues, topics and subscriptions. It is named by a tenant identifier: exactly 24 characters, a lowercase letter followed by 23 lowercase letters or digits. That format is valid in the strictest field of every protocol at once (an Azure Storage account name allows only 3 to 24 lowercase letters and digits; a Google Cloud project id must start with a letter). A random identifier has about 2¹²³ possible values, so nobody can guess yours.

You do not send the identifier in anything new. Each protocol already has a credential field its SDK requires, and the identifier goes there:

Signatures and passwords are not checked: there are no keys to check them against. The identifier itself is the only secret, so treat it like a password. The same identifier names the same tenant in every protocol.

Tenants are created on first use. There is no signup: the first request that names a well-formed identifier creates its tenant. A request whose field holds anything else, including placeholders from SDK samples such as guest, test, test-project or devstoreaccount1, is refused with the protocol's own authentication or validation error, whose message says what a valid identifier looks like. It is never sent to a shared default tenant. Each client address may create up to 20 new tenants an hour, and the service holds at most 10,000 tenants.

Isolation between tenants

Tenants never see each other. The same queue name in two tenants is two queues, listings show only your own entities, and every reference that could point into another tenant is refused rather than resolved:

• SQS: another tenant's queue URL answers QueueDoesNotExist; a redrive policy or message move task naming another tenant's ARN is InvalidParameterValue.
• Pub/Sub: a subscription, dead-letter topic, snapshot or schema in another project is INVALID_ARGUMENT.
• Azure Queue Storage: a Shared Key header naming an account other than the one in the URL is 403 AuthenticationFailed.
• Service Bus: ForwardTo naming another namespace is 400, and a connection keeps the tenant of its first token.

Isolation is by name, not by encryption: everyone's messages are on the same machine, in the same broker. Do not send anything confidential.

Queues and topics need not be created

In the public service, using a name creates the entity, so the shortest program that sends and receives works without any setup:

• AMQP 0-9-1: publishing to a queue name through the default exchange, consuming, basic.get and a passive declare create the queue.
• SQS: GetQueueUrl, SendMessage, SendMessageBatch and ReceiveMessage on an unknown name create a standard queue; a name ending in .fifo creates a FIFO queue with SQS's defaults. CreateQueue works as usual.
• Azure Queue Storage: Put Message, Get Messages, Peek Messages and Clear Messages create the queue. A later Create Queue answers 201 Created and keeps the metadata you give.
• Azure Service Bus: sending to or receiving from an unknown entity creates a queue.
• Google Pub/Sub: publishing to an unknown topic X creates topic X together with a subscription also called X, so publish-then-pull on one name loses nothing. Pulling a subscription named X~name creates that subscription on topic X (and topic X with its subscription X if needed); it receives what is published from then on. Pulling any other unknown subscription is NOT_FOUND, because a pull names only the subscription and the topic cannot be known. CreateSubscription works as usual for any name.

Code written for the real services still works: if it creates its entities first, it finds them or is told it created them. The difference to know: a mistyped name silently creates a new entity, and code that relies on a not-found answer for a missing queue or topic does not get one.

Using one queue from several interfaces

Within a tenant every interface shares one set of queue names, so a message sent through one interface can be read through any other: send with SQS, read with AMQP 0-9-1; publish to a Pub/Sub topic, read its subscription with Service Bus. Each interface can express things the others cannot, so a message is delivered in the reading interface's natural form, marked where that interface can carry a mark, and never silently damaged. The interface that sent a message always reads it back exactly as sent. Nothing is refused for being hard to represent elsewhere.

Message bodies

To send binary through SQS, send the base64 of the bytes with the attribute x-pq-body-encoding = base64 (type String): the bytes are stored, and AMQP 0-9-1, Pub/Sub and Service Bus readers receive them. Text sent through Azure Queue Storage is stored as that text, so an Azure SDK that base64-encodes its messages (a message encoding policy) delivers base64 text to the other interfaces. If a queue may carry binary and you read it through Azure Queue Storage, decode base64 yourself.

A Service Bus body that is an AMQP string or binary value crosses as its plain bytes. Any other AMQP value (a number, a map, a list, a timestamp) or an AMQP sequence is rendered as JSON for the other interfaces: null, booleans and numbers as JSON values; strings and symbols as strings; binary as a base64 string; timestamps as RFC 3339 strings; uuids as their text; decimals as strings such as 125E-2; lists and sequences as arrays; maps as objects. For example a Service Bus message whose body is the map {"a": 1} reads as the text {"a":1} through SQS. Its stored size is the JSON and the AMQP encoding together.

Attributes, headers and properties

AMQP 0-9-1 header types no other interface has are delivered to SQS, Pub/Sub and Service Bus as text: a nested table or an array as JSON (SQS data type String.json), a decimal as its decimal text (SQS type Number), a void value as an empty string (a null property through Service Bus). For example the header meta = table {k: "v"} reads through SQS as the attribute meta of type String.json with value {"k":"v"}.

Names starting x-pq- are the service's own markers and metadata. Clients cannot set them, except x-pq-body-encoding = base64 on an SQS send as above. AMQP 0-9-1 readers see every header, these included.

Idle deletion

An entity that came into being implicitly is deleted once it has been empty and unused for an hour. An entity you create explicitly (with CreateQueue, queue.declare, Create Queue, CreateTopic and so on) stays until you delete it or its tenant is deleted, as does an implicit one you later create explicitly. This keeps the per-tenant entity cap from filling up with names used once.

The limits, and why

The service is shared and free, so each limit stops one client from taking what everyone else needs. Every refusal is the protocol's own error, so your SDK reports it the way it would against the real service. The documentation lists the exact error for each limit in each protocol.

The broker also stops accepting new messages when its disk runs short, so one busy tenant cannot fill it for everyone.

Endpoints and TLS

Every public endpoint uses TLS with a Let's Encrypt certificate, so clients need no extra trust settings:

The HTTPS and gRPC endpoints go through a reverse proxy (Caddy) that terminates TLS and passes each client's address on to the broker, so the per-address limits apply to the real client. The two AMQP ports are the broker's own TLS listeners, using the same certificate. AMQP 0-9-1 uses port 5673 rather than its usual 5671 because the Service Bus SDKs always connect to 5671 and offer no way to choose another port. There are no plain-text endpoints.

What is kept, and for how long

• Across maintenance. The service is down from time to time for maintenance, without notice. Tenants, queues, topics, subscriptions and messages are stored on disk and normally survive those restarts.
• Messages live at most 1 hour, less if you ask for a shorter retention or time to live. A tenant holds at most 16 MiB of messages at once.
• Idle tenants are deleted automatically, with everything in them, after 7 days without activity. Any request or connection naming the tenant, and any message sent to or received from its queues, counts as activity; a tenant with a connection open is never deleted. Use the identifier again after that and you get a new, empty tenant.
• No guarantee. Nothing is backed up, and data may be lost at any time, for example when a storage format changes in an upgrade.

When the live statistics on the front page have not been updated for a few minutes, the service is probably down. Write your code to cope: create (or simply use) your queues at start-up, retry connections, and do not rely on anything being there.

What is deliberately not offered

• Accounts, keys and access control. Anything that needs a signup is out of scope; the identifier is the whole credential. IAM policies, SQS permissions and Azure ACLs are stored and echoed back, but grant nothing.
• Durability and backups. Data normally persists, but nothing is backed up or guaranteed.
• Pub/Sub push subscriptions. They would make the service send requests to any address on the internet; use pull or streaming pull.
• The broker's management HTTP API. It is off in public mode.
• Plain-text endpoints. Only TLS is offered.
• Support, notice of maintenance, or any service level.