Manage subscriptions in InfluxDB
This page documents an earlier version of InfluxDB. InfluxDB v2.1 is the latest stable version.
InfluxDB subscriptions are local or remote endpoints to which all data written to InfluxDB is copied. Subscriptions are primarily used with Kapacitor, but any endpoint able to accept UDP, HTTP, or HTTPS connections can subscribe to InfluxDB and receive a copy of all data as it is written.
How subscriptions work
As data is written to InfluxDB, writes are duplicated to subscriber endpoints via HTTP, HTTPS, or UDP in line protocol. the InfluxDB subscriber service creates multiple “writers” (goroutines) which send writes to the subscription endpoints.
The number of writer goroutines is defined by the write-concurrency
configuration.
As writes occur in InfluxDB, each subscription writer sends the written data to the
specified subscription endpoints.
However, with a high write-concurrency
(multiple writers) and a high ingest rate,
nanosecond differences in writer processes and the transport layer can result
in writes being received out of order.
Important information about high write loads
While setting the subscriber
write-concurrency
to greater than 1 does increase your subscriber write throughput, it can result in out-of-order writes under high ingest rates. Settingwrite-concurrency
to 1 ensures writes are passed to subscriber endpoints sequentially, but can create a bottleneck under high ingest rates.What
write-concurrency
should be set to depends on your specific workload and need for in-order writes to your subscription endpoint.
InfluxQL subscription statements
Use the following InfluxQL statements to manage subscriptions:
CREATE SUBSCRIPTION
SHOW SUBSCRIPTIONS
DROP SUBSCRIPTION
Create subscriptions
Create subscriptions using the CREATE SUBSCRIPTION
InfluxQL statement.
Specify the subscription name, the database name and retention policy to subscribe to,
and the URL of the host to which data written to InfluxDB should be copied.
-- Pattern:
CREATE SUBSCRIPTION "<subscription_name>" ON "<db_name>"."<retention_policy>" DESTINATIONS <ALL|ANY> "<subscription_endpoint_host>"
-- Examples:
-- Create a SUBSCRIPTION on database 'mydb' and retention policy 'autogen' that sends data to 'example.com:9090' via HTTP.
CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ALL 'http://example.com:9090'
-- Create a SUBSCRIPTION on database 'mydb' and retention policy 'autogen' that round-robins the data to 'h1.example.com:9090' and 'h2.example.com:9090' via UDP.
CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ANY 'udp://h1.example.com:9090', 'udp://h2.example.com:9090'
In case authentication is enabled on the subscriber host, adapt the URL to contain the credentials.
-- Create a SUBSCRIPTION on database 'mydb' and retention policy 'autogen' that sends data to another InfluxDB on 'example.com:8086' via HTTP. Authentication is enabled on the subscription host (user: subscriber, pass: secret).
CREATE SUBSCRIPTION "sub0" ON "mydb"."autogen" DESTINATIONS ALL 'http://subscriber:secret@example.com:8086'
SHOW SUBSCRIPTIONS
outputs all subscriber URL in plain text, including those with authentication credentials.
Any user with the privileges to run SHOW SUBSCRIPTIONS
is able to see these credentials.
Sending subscription data to multiple hosts
The CREATE SUBSCRIPTION
statement allows you to specify multiple hosts as endpoints for the subscription.
In your DESTINATIONS
clause, you can pass multiple host strings separated by commas.
Using ALL
or ANY
in the DESTINATIONS
clause determines how InfluxDB writes data to each endpoint:
ALL
: Writes data to all specified hosts.
ANY
: Round-robins writes between specified hosts.
Subscriptions with multiple hosts
-- Write all data to multiple hosts
CREATE SUBSCRIPTION "mysub" ON "mydb"."autogen" DESTINATIONS ALL 'http://host1.example.com:9090', 'http://host2.example.com:9090'
-- Round-robin writes between multiple hosts
CREATE SUBSCRIPTION "mysub" ON "mydb"."autogen" DESTINATIONS ANY 'http://host1.example.com:9090', 'http://host2.example.com:9090'
Subscription protocols
Subscriptions can use HTTP, HTTPS, or UDP transport protocols.
Which to use is determined by the protocol expected by the subscription endpoint.
If creating a Kapacitor subscription, this is defined by the subscription-protocol
option in the [[influxdb]]
section of your kapacitor.conf
.
kapacitor.conf
[[influxdb]]
# ...
subscription-protocol = "http"
# ...
For information regarding HTTPS connections and secure communication between InfluxDB and Kapacitor, view the Kapacitor security documentation.
Show subscriptions
The SHOW SUBSCRIPTIONS
InfluxQL statement returns a list of all subscriptions registered in InfluxDB.
SHOW SUBSCRIPTIONS
Example output:
name: _internal
retention_policy name mode destinations
---------------- ---- ---- ------------
monitor kapacitor-39545771-7b64-4692-ab8f-1796c07f3314 ANY [http://localhost:9092]
Remove subscriptions
Remove or drop subscriptions using the DROP SUBSCRIPTION
InfluxQL statement.
-- Pattern:
DROP SUBSCRIPTION "<subscription_name>" ON "<db_name>"."<retention_policy>"
-- Example:
DROP SUBSCRIPTION "sub0" ON "mydb"."autogen"
Drop all subscriptions
In some cases, it may be necessary to remove all subscriptions.
Run the following bash script that utilizes the influx
CLI, loops through all subscriptions, and removes them.
This script depends on the $INFLUXUSER
and $INFLUXPASS
environment variables.
If these are not set, export them as part of the script.
# Environment variable exports:
# Uncomment these if INFLUXUSER and INFLUXPASS are not already globally set.
# export INFLUXUSER=influxdb-username
# export INFLUXPASS=influxdb-password
IFS=$'\n'; for i in $(influx -format csv -username $INFLUXUSER -password $INFLUXPASS -database _internal -execute 'show subscriptions' | tail -n +2 | grep -v name); do influx -format csv -username $INFLUXUSER -password $INFLUXPASS -database _internal -execute "drop subscription \"$(echo "$i" | cut -f 3 -d ',')\" ON \"$(echo "$i" | cut -f 1 -d ',')\".\"$(echo "$i" | cut -f 2 -d ',')\""; done
Configure InfluxDB subscriptions
InfluxDB subscription configuration options are available in the [subscriber]
section of the influxdb.conf
.
In order to use subcriptions, the enabled
option in the [subscriber]
section must be set to true
.
Below is an example influxdb.conf
subscriber configuration:
[subscriber]
enabled = true
http-timeout = "30s"
insecure-skip-verify = false
ca-certs = ""
write-concurrency = 40
write-buffer-size = 1000
Descriptions of [subscriber]
configuration options are available in the Configuring InfluxDB documentation.
Troubleshooting
Inaccessible or decommissioned subscription endpoints
Unless a subscription is dropped, InfluxDB assumes the endpoint should always receive data and will continue to attempt to send data. If an endpoint host is inaccessible or has been decommissioned, you will see errors similar to the following:
# Some message content omitted (...) for the sake of brevity
"Post http://x.y.z.a:9092/write?consistency=...: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" ... service=subscriber
"Post http://x.y.z.a:9092/write?consistency=...: dial tcp x.y.z.a:9092: getsockopt: connection refused" ... service=subscriber
"Post http://x.y.z.a:9092/write?consistency=...: dial tcp 172.31.36.5:9092: getsockopt: no route to host" ... service=subscriber
In some cases, this may be caused by a networking error or something similar preventing a successful connection to the subscription endpoint. In other cases, it’s because the subscription endpoint no longer exists and the subscription hasn’t been dropped from InfluxDB.
Because InfluxDB does not know if a subscription endpoint will or will not become accessible again, subscriptions are not automatically dropped when an endpoint becomes inaccessible. If a subscription endpoint is removed, you must manually drop the subscription from InfluxDB.
Support and feedback
Thank you for being part of our community! We welcome and encourage your feedback and bug reports for InfluxDB and this documentation. To find support, the following resources are available:
InfluxDB Cloud and InfluxDB Enterprise customers can contact InfluxData Support.