Add firebase subscriber to topics from cache upon initialization; stricter rate limits

.goreleaser.yml

@@ -4,11 +4,9 @@ before:
 builds:
   - binary: ntfy
     env:
-      - CGO_ENABLED=0
+      - CGO_ENABLED=1 # required for go-sqlite3
     goos:
       - linux
-      - windows
-      - darwin
     goarch:
       - amd64
 nfpms:

Makefile

@@ -94,6 +94,7 @@ build-snapshot:
 
 build-simple: clean
 	mkdir -p dist/ntfy_linux_amd64
+	export CGO_ENABLED=1
 	$(GO) build \
 		-o dist/ntfy_linux_amd64/ntfy \
 		-ldflags \

README.md

@@ -1,47 +1,112 @@
-# ntfy
+![ntfy](server/static/img/ntfy.png)
 
-ntfy (pronounce: *notify*) is a super simple pub-sub notification service. It allows you to send desktop and (soon) phone notifications
-via scripts. I run a free version of it on *[ntfy.sh](https://ntfy.sh)*. **No signups or cost.**
+# ntfy - simple HTTP-based pub-sub
+
+**ntfy** (pronounce: *notify*) is a simple HTTP-based [pub-sub](https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern) notification service.
+It allows you to **send notifications to your phone or desktop via scripts** from any computer, entirely **without signup or cost**.
+It's also open source (as you can plainly see) if you want to run your own.
+
+I run a free version of it at **[ntfy.sh](https://ntfy.sh)**, and there's an [Android app](https://play.google.com/store/apps/details?id=io.heckel.ntfy)
+too.
 
 ## Usage
 
-### Subscribe to a topic
-Topics are created on the fly by subscribing to them. You can create and subscribe to a topic either in a web UI, or in 
-your own app by subscribing to an [SSE](https://en.wikipedia.org/wiki/Server-sent_events)/[EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource),
-or a JSON or raw feed.  
+### Publishing messages
 
-Because there is no sign-up, **the topic is essentially a password**, so pick something that's not easily guessable.  
+Publishing messages can be done via PUT or POST using. Topics are created on the fly by subscribing or publishing to them.
+Because there is no sign-up, **the topic is essentially a password**, so pick something that's not easily guessable.
 
-Here's how you can create a topic `mytopic`, subscribe to it topic and wait for events. This is using `curl`, but you
-can use any library that can do HTTP GETs:
+Here's an example showing how to publish a message using `curl`:
 
-```
-# Subscribe to "mytopic" and output one message per line (\n are replaced with a space)
-curl -s ntfy.sh/mytopic/raw
-
-# Subscribe to "mytopic" and output one JSON message per line
-curl -s ntfy.sh/mytopic/json
-
-# Subscribe to "mytopic" and output an SSE stream (supported via JS/EventSource)
-curl -s ntfy.sh/mytopic/sse
-```
-
-You can easily script it to execute any command when a message arrives. This sends desktop notifications (just like 
-the web UI, but without it):
-```
-while read msg; do
-  [ -n "$msg" ] && notify-send "$msg"
-done < <(stdbuf -i0 -o0 curl -s ntfy.sh/mytopic/raw)
-```
-
-### Publish messages
-Publishing messages can be done via PUT or POST using. Here's an example using `curl`:
 ```
 curl -d "long process is done" ntfy.sh/mytopic
 ```
 
-Messages published to a non-existing topic or a topic without subscribers will not be delivered later. There is (currently)
-no buffering of any kind. If you're not listening, the message won't be delivered.
+Here's an example in JS with `fetch()` (see [full example](examples)):
+
+```
+fetch('https://ntfy.sh/mytopic', {
+  method: 'POST', // PUT works too
+  body: 'Hello from the other side.'
+})
+```
+
+### Subscribe to a topic
+You can create and subscribe to a topic either in this web UI, or in your own app by subscribing to an
+[EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource), a JSON feed, or raw feed.
+
+#### Subscribe via web
+If you subscribe to a topic via this web UI in the field below, messages published to any subscribed topic
+will show up as **desktop notification**.
+
+You can try this easily on **[ntfy.sh](https://ntfy.sh)**.
+
+#### Subscribe via phone
+You can use the [Ntfy Android App](https://play.google.com/store/apps/details?id=io.heckel.ntfy) to receive 
+notifications directly on your phone. Just like the server, this app is also [open source](https://github.com/binwiederhier/ntfy-android).
+
+#### Subscribe via your app, or via the CLI
+Using [EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) in JS, you can consume
+notifications like this (see [full example](examples)):
+
+```javascript
+const eventSource = new EventSource('https://ntfy.sh/mytopic/sse');<br/>
+eventSource.onmessage = (e) => {<br/>
+  // Do something with e.data<br/>
+};
+```
+
+You can also use the same `/sse` endpoint via `curl` or any other HTTP library:
+
+```
+$ curl -s ntfy.sh/mytopic/sse
+event: open
+data: {"id":"weSj9RtNkj","time":1635528898,"event":"open","topic":"mytopic"}
+
+data: {"id":"p0M5y6gcCY","time":1635528909,"event":"message","topic":"mytopic","message":"Hi!"}
+
+event: keepalive
+data: {"id":"VNxNIg5fpt","time":1635528928,"event":"keepalive","topic":"test"}
+```
+
+To consume JSON instead, use the `/json` endpoint, which prints one message per line:
+
+```
+$ curl -s ntfy.sh/mytopic/json
+{"id":"SLiKI64DOt","time":1635528757,"event":"open","topic":"mytopic"}
+{"id":"hwQ2YpKdmg","time":1635528741,"event":"message","topic":"mytopic","message":"Hi!"}
+{"id":"DGUDShMCsc","time":1635528787,"event":"keepalive","topic":"mytopic"}
+```
+
+Or use the `/raw` endpoint if you need something super simple (empty lines are keepalive messages):
+
+```
+$ curl -s ntfy.sh/mytopic/raw
+
+This is a notification
+```
+
+#### Message buffering and polling
+Messages are buffered in memory for a few hours to account for network interruptions of subscribers.
+You can read back what you missed by using the `since=...` query parameter. It takes either a
+duration (e.g. `10m` or `30s`) or a Unix timestamp (e.g. `1635528757`):
+
+```
+$ curl -s "ntfy.sh/mytopic/json?since=10m"
+# Same output as above, but includes messages from up to 10 minutes ago
+```
+
+You can also just poll for messages if you don't like the long-standing connection using the `poll=1`
+query parameter. The connection will end after all available messages have been read. This parameter has to be
+combined with `since=`.
+
+```
+$ curl -s "ntfy.sh/mytopic/json?poll=1&since=10m"
+# Returns messages from up to 10 minutes ago and ends the connection
+```
+
+## Examples
+There are a few usage examples in the [examples](examples) directory. I'm sure there are tons of other ways to use it.
 
 ## Installation
 Please check out the [releases page](https://github.com/binwiederhier/ntfy/releases) for binaries and
@@ -64,13 +129,13 @@ sudo apt install ntfy
 **Debian/Ubuntu** (*manual install*)**:**
 ```bash
 sudo apt install tmux
-wget https://github.com/binwiederhier/ntfy/releases/download/v1.1.2/ntfy_1.1.2_amd64.deb
-dpkg -i ntfy_1.1.2_amd64.deb
+wget https://github.com/binwiederhier/ntfy/releases/download/v1.2.0/ntfy_1.2.0_amd64.deb
+dpkg -i ntfy_1.2.0_amd64.deb
 ```
 
 **Fedora/RHEL/CentOS:**
 ```bash
-rpm -ivh https://github.com/binwiederhier/ntfy/releases/download/v1.1.2/ntfy_1.1.2_amd64.rpm
+rpm -ivh https://github.com/binwiederhier/ntfy/releases/download/v1.2.0/ntfy_1.2.0_amd64.rpm
 ```
 
 **Docker:**
@@ -85,8 +150,8 @@ go get -u heckel.io/ntfy
 
 **Manual install** (*any x86_64-based Linux*)**:**
 ```bash
-wget https://github.com/binwiederhier/ntfy/releases/download/v1.1.2/ntfy_1.1.2_linux_x86_64.tar.gz
-sudo tar -C /usr/bin -zxf ntfy_1.1.2_linux_x86_64.tar.gz ntfy
+wget https://github.com/binwiederhier/ntfy/releases/download/v1.2.0/ntfy_1.2.0_linux_x86_64.tar.gz
+sudo tar -C /usr/bin -zxf ntfy_1.2.0_linux_x86_64.tar.gz ntfy
 ./ntfy
 ```
 
@@ -104,7 +169,6 @@ To build releases, I use [GoReleaser](https://goreleaser.com/). If you have that
 ## TODO
 - add HTTPS
 - make limits configurable
-- limit max number of subscriptions 
 
 ## Contributing
 I welcome any and all contributions. Just create a PR or an issue.
@@ -116,4 +180,6 @@ Third party libraries and resources:
 * [github.com/urfave/cli/v2](https://github.com/urfave/cli/v2) (MIT) is used to drive the CLI
 * [Mixkit sound](https://mixkit.co/free-sound-effects/notification/) (Mixkit Free License) used as notification sound
 * [Lato Font](https://www.latofonts.com/) (OFL) is used as a font in the Web UI
-* [GoReleaser](https://goreleaser.com/) (MIT) is used to create releases 
+* [GoReleaser](https://goreleaser.com/) (MIT) is used to create releases
+* [github.com/mattn/go-sqlite3](https://github.com/mattn/go-sqlite3) (MIT) is used to provide the persistent message cache
+* [Firebase Admin SDK](https://github.com/firebase/firebase-admin-go) (Apache 2.0) is used to send FCM messages

cmd/app.go

@@ -19,9 +19,9 @@ func New() *cli.App {
 	flags := []cli.Flag{
 		&cli.StringFlag{Name: "config", Aliases: []string{"c"}, EnvVars: []string{"NTFY_CONFIG_FILE"}, Value: "/etc/ntfy/config.yml", DefaultText: "/etc/ntfy/config.yml", Usage: "config file"},
 		altsrc.NewStringFlag(&cli.StringFlag{Name: "listen-http", Aliases: []string{"l"}, EnvVars: []string{"NTFY_LISTEN_HTTP"}, Value: config.DefaultListenHTTP, Usage: "ip:port used to as listen address"}),
-		altsrc.NewStringFlag(&cli.StringFlag{Name: "cache-file", Aliases: []string{"C"}, EnvVars: []string{"NTFY_CACHE_FILE"}, Usage: "cache file used for message caching"}),
 		altsrc.NewStringFlag(&cli.StringFlag{Name: "firebase-key-file", Aliases: []string{"F"}, EnvVars: []string{"NTFY_FIREBASE_KEY_FILE"}, Usage: "Firebase credentials file; if set additionally publish to FCM topic"}),
-		altsrc.NewDurationFlag(&cli.DurationFlag{Name: "message-buffer-duration", Aliases: []string{"b"}, EnvVars: []string{"NTFY_MESSAGE_BUFFER_DURATION"}, Value: config.DefaultMessageBufferDuration, Usage: "buffer messages in memory for this time to allow `since` requests"}),
+		altsrc.NewStringFlag(&cli.StringFlag{Name: "cache-file", Aliases: []string{"C"}, EnvVars: []string{"NTFY_CACHE_FILE"}, Usage: "cache file used for message caching"}),
+		altsrc.NewDurationFlag(&cli.DurationFlag{Name: "cache-duration", Aliases: []string{"b"}, EnvVars: []string{"NTFY_CACHE_DURATION"}, Value: config.DefaultCacheDuration, Usage: "buffer messages for this time to allow `since` requests"}),
 		altsrc.NewDurationFlag(&cli.DurationFlag{Name: "keepalive-interval", Aliases: []string{"k"}, EnvVars: []string{"NTFY_KEEPALIVE_INTERVAL"}, Value: config.DefaultKeepaliveInterval, Usage: "default interval of keepalive messages"}),
 		altsrc.NewDurationFlag(&cli.DurationFlag{Name: "manager-interval", Aliases: []string{"m"}, EnvVars: []string{"NTFY_MANAGER_INTERVAL"}, Value: config.DefaultManagerInterval, Usage: "default interval of for message pruning and stats printing"}),
 	}
@@ -45,9 +45,9 @@ func New() *cli.App {
 func execRun(c *cli.Context) error {
 	// Read all the options
 	listenHTTP := c.String("listen-http")
-	cacheFile := c.String("cache-file")
 	firebaseKeyFile := c.String("firebase-key-file")
-	messageBufferDuration := c.Duration("message-buffer-duration")
+	cacheFile := c.String("cache-file")
+	cacheDuration := c.Duration("cache-duration")
 	keepaliveInterval := c.Duration("keepalive-interval")
 	managerInterval := c.Duration("manager-interval")
 
@@ -58,15 +58,15 @@ func execRun(c *cli.Context) error {
 		return errors.New("keepalive interval cannot be lower than five seconds")
 	} else if managerInterval < 5*time.Second {
 		return errors.New("manager interval cannot be lower than five seconds")
-	} else if messageBufferDuration < managerInterval {
-		return errors.New("message buffer duration cannot be lower than manager interval")
+	} else if cacheDuration < managerInterval {
+		return errors.New("cache duration cannot be lower than manager interval")
 	}
 
 	// Run server
 	conf := config.New(listenHTTP)
-	conf.CacheFile = cacheFile
 	conf.FirebaseKeyFile = firebaseKeyFile
-	conf.MessageBufferDuration = messageBufferDuration
+	conf.CacheFile = cacheFile
+	conf.CacheDuration = cacheDuration
 	conf.KeepaliveInterval = keepaliveInterval
 	conf.ManagerInterval = managerInterval
 	s, err := server.New(conf)

config/config.go

@@ -8,19 +8,19 @@ import (
 
 // Defines default config settings
 const (
-	DefaultListenHTTP            = ":80"
-	DefaultMessageBufferDuration = 12 * time.Hour
-	DefaultKeepaliveInterval     = 30 * time.Second
-	DefaultManagerInterval       = time.Minute
+	DefaultListenHTTP        = ":80"
+	DefaultCacheDuration     = 12 * time.Hour
+	DefaultKeepaliveInterval = 30 * time.Second
+	DefaultManagerInterval   = time.Minute
 )
 
 // Defines all the limits
-// - request limit: max number of PUT/GET/.. requests (here: 50 requests bucket, replenished at a rate of 1 per second)
+// - request limit: max number of PUT/GET/.. requests (here: 50 requests bucket, replenished at a rate of one per 10 seconds)
 // - global topic limit: max number of topics overall
 // - subscription limit: max number of subscriptions (active HTTP connections) per per-visitor/IP
 var (
 	defaultGlobalTopicLimit         = 5000
-	defaultVisitorRequestLimit      = rate.Every(time.Second)
+	defaultVisitorRequestLimit      = rate.Every(10 * time.Second)
 	defaultVisitorRequestLimitBurst = 50
 	defaultVisitorSubscriptionLimit = 30
 )
@@ -28,9 +28,9 @@ var (
 // Config is the main config struct for the application. Use New to instantiate a default config struct.
 type Config struct {
 	ListenHTTP               string
-	CacheFile          string
 	FirebaseKeyFile          string
-	MessageBufferDuration    time.Duration
+	CacheFile                string
+	CacheDuration            time.Duration
 	KeepaliveInterval        time.Duration
 	ManagerInterval          time.Duration
 	GlobalTopicLimit         int
@@ -43,9 +43,9 @@ type Config struct {
 func New(listenHTTP string) *Config {
 	return &Config{
 		ListenHTTP:               listenHTTP,
-		CacheFile: "",
 		FirebaseKeyFile:          "",
-		MessageBufferDuration:    DefaultMessageBufferDuration,
+		CacheFile:                "",
+		CacheDuration:            DefaultCacheDuration,
 		KeepaliveInterval:        DefaultKeepaliveInterval,
 		ManagerInterval:          DefaultManagerInterval,
 		GlobalTopicLimit:         defaultGlobalTopicLimit,

config/config.yml

@@ -10,10 +10,15 @@
 #
 # firebase-key-file: <filename>
 
+# If set, messages are cached in a local SQLite database instead of only in-memory. This
+# allows for service restarts without losing messages in support of the since= parameter.
+#
+# cache-file: <filename>
+
 # Duration for which messages will be buffered before they are deleted.
 # This is required to support the "since=..." and "poll=1" parameter.
 #
-# message-buffer-duration: 12h
+# cache-duration: 12h
 
 # Interval in which keepalive messages are sent to the client. This is to prevent
 # intermediaries closing the connection for inactivity.

server/cache_mem.go

@@ -7,8 +7,8 @@ import (
 )
 
 type memCache struct {
-	messages    map[string][]*message
-	mu sync.Mutex
+	messages map[string][]*message
+	mu       sync.Mutex
 }
 
 var _ cache = (*memCache)(nil)

server/cache_sqlite.go

@@ -19,8 +19,8 @@ const (
 		CREATE INDEX IF NOT EXISTS idx_topic ON messages (topic);
 		COMMIT;
 	`
-	insertMessageQuery = `INSERT INTO messages (id, time, topic, message) VALUES (?, ?, ?, ?)`
-	pruneMessagesQuery = `DELETE FROM messages WHERE time < ?`
+	insertMessageQuery           = `INSERT INTO messages (id, time, topic, message) VALUES (?, ?, ?, ?)`
+	pruneMessagesQuery           = `DELETE FROM messages WHERE time < ?`
 	selectMessagesSinceTimeQuery = `
 		SELECT id, time, message 
 		FROM messages 
@@ -46,7 +46,7 @@ func newSqliteCache(filename string) (*sqliteCache, error) {
 		return nil, err
 	}
 	return &sqliteCache{
-		db:     db,
+		db: db,
 	}, nil
 }
 
@@ -122,6 +122,6 @@ func (s *sqliteCache) Topics() (map[string]*topic, error) {
 }
 
 func (c *sqliteCache) Prune(keep time.Duration) error {
-	_, err := c.db.Exec(pruneMessagesQuery, time.Now().Add(-1 * keep).Unix())
+	_, err := c.db.Exec(pruneMessagesQuery, time.Now().Add(-1*keep).Unix())
 	return err
 }

server/index.html

@@ -33,7 +33,7 @@
     <h1><img src="static/img/ntfy.png" alt="ntfy"/><br/>ntfy.sh - simple HTTP-based pub-sub</h1>
     <p>
         <b>ntfy</b> (pronounce: <i>notify</i>) is a simple HTTP-based <a href="https://en.wikipedia.org/wiki/Publish%E2%80%93subscribe_pattern">pub-sub</a> notification service.
-        It allows you to send <b>desktop notifications via scripts from any computer</b>, entirely <b>without signup or cost</b>.
+        It allows you to send <b>notifications to your phone or desktop via scripts from any computer</b>, entirely <b>without signup or cost</b>.
         It's also <a href="https://github.com/binwiederhier/ntfy">open source</a> if you want to run your own.
     </p>
     <p id="error"></p>
@@ -83,8 +83,8 @@
 
     <h3>Subscribe via phone</h3>
     <p>
-        Once it's approved, you can use the <b>Ntfy Android App</b> to receive notifications directly on your phone. Just like
-        the server, this app is also <a href="https://github.com/binwiederhier/ntfy-android">open source</a>.
+        You can use the <a href="https://play.google.com/store/apps/details?id=io.heckel.ntfy">Ntfy Android App</a>
+        to receive notifications directly on your phone. Just like the server, this app is also <a href="https://github.com/binwiederhier/ntfy-android">open source</a>.
     </p>
 
     <h3>Subscribe via your app, or via the CLI</h3>
@@ -184,7 +184,7 @@
     <h2>Privacy policy</h2>
     <p>
         Neither the server nor the app record any personal information, or share any of the messages and topics with
-        any outside service. All data is exclusively used to make the service function properly. The notable exception
+        any outside service. All data is exclusively used to make the service function properly. The one exception
         is the Firebase Cloud Messaging (FCM) service, which is required to provide instant Android notifications (see
         FAQ for details).
     </p>

server/server.go

@@ -47,7 +47,7 @@ func (e errHTTP) Error() string {
 }
 
 const (
-	messageLimit = 1024
+	messageLimit = 512
 )
 
 var (
@@ -86,6 +86,11 @@ func New(conf *config.Config) (*Server, error) {
 	if err != nil {
 		return nil, err
 	}
+	for _, t := range topics {
+		if firebaseSubscriber != nil {
+			t.Subscribe(firebaseSubscriber)
+		}
+	}
 	return &Server{
 		config:   conf,
 		cache:    cache,
@@ -204,6 +209,9 @@ func (s *Server) handlePublish(w http.ResponseWriter, r *http.Request, v *visito
 		return err
 	}
 	w.Header().Set("Access-Control-Allow-Origin", "*") // CORS, allow cross-origin requests
+	if err := json.NewEncoder(w).Encode(m); err != nil {
+		return err
+	}
 	s.mu.Lock()
 	s.messages++
 	s.mu.Unlock()
@@ -360,7 +368,7 @@ func (s *Server) updateStatsAndExpire() {
 	}
 
 	// Prune cache
-	if err := s.cache.Prune(s.config.MessageBufferDuration); err != nil {
+	if err := s.cache.Prune(s.config.CacheDuration); err != nil {
 		log.Printf("error pruning cache: %s", err.Error())
 	}