storage: start db migration (badger(hold))

migrating to badger, with the badgerhold frontend. So far, done:
* Announcements (small, for a quick test)
* Discord/Telegram/Matrix/Email

most interaction with badgerhold is done through the standard
Get<x>/Get<x>Key/Set<x>Key/Delete<x>Key. UserExists functions have been
added for email and matrix, and those and the original ones now use a
query against the database rather than sifting through every record.
I've tagged these searched fields as "index" for badgerhold, although this
definitely isn't used yet, and i'm not entirely sure if it'll be useful.

migrateToBadger is now in migrations.go, and a temporary config key
"migrated_to_badger" has been added, although it isn't being used yet,
migration is just running every time during development.
This commit is contained in:
Harvey Tindall
2023-06-24 17:01:52 +01:00
parent fc7ae0ec4e
commit 833be688ac
12 changed files with 458 additions and 176 deletions

14
go.mod
View File

@@ -48,7 +48,12 @@ require (
require (
github.com/KyleBanks/depth v1.2.1 // indirect
github.com/bytedance/sonic v1.9.1 // indirect
github.com/cespare/xxhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.1.1 // indirect
github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
github.com/dgraph-io/badger/v3 v3.2103.1 // indirect
github.com/dgraph-io/ristretto v0.1.0 // indirect
github.com/dustin/go-humanize v1.0.0 // indirect
github.com/gabriel-vasile/mimetype v1.4.2 // indirect
github.com/getlantern/context v0.0.0-20220418194847-3d5e7a086201 // indirect
github.com/getlantern/errors v1.0.3 // indirect
@@ -69,12 +74,19 @@ require (
github.com/go-stack/stack v1.8.1 // indirect
github.com/go-test/deep v1.1.0 // indirect
github.com/goccy/go-json v0.10.2 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0 // indirect
github.com/golang/glog v0.0.0-20210429001901-424d2337a529 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/flatbuffers v2.0.0+incompatible // indirect
github.com/google/uuid v1.3.0 // indirect
github.com/gorilla/mux v1.8.0 // indirect
github.com/gorilla/websocket v1.5.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.13.1 // indirect
github.com/klauspost/cpuid/v2 v2.2.5 // indirect
github.com/leodido/go-urn v1.2.4 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
@@ -92,9 +104,11 @@ require (
github.com/tidwall/match v1.1.1 // indirect
github.com/tidwall/pretty v1.2.1 // indirect
github.com/tidwall/sjson v1.2.5 // indirect
github.com/timshannon/badgerhold/v4 v4.0.2 // indirect
github.com/toorop/go-dkim v0.0.0-20201103131630-e1cd1a0a5208 // indirect
github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
github.com/ugorji/go/codec v1.2.11 // indirect
go.opencensus.io v0.23.0 // indirect
go.opentelemetry.io/otel v1.16.0 // indirect
go.opentelemetry.io/otel/metric v1.16.0 // indirect
go.opentelemetry.io/otel/trace v1.16.0 // indirect