Implementing db caching functionality (#211)

* Implementing file cache functionality * Open or Create file * Refactoring code * Adding dbcache functionality * Adding dependencies * Updating dbcache * Updating dbcache * Updating dbcache * Removing unnecessary code * Fix linter issues * Adding tests * Adding tests * Adding interface * Setting up db cache within uhttp pkg * Adding validations * Adding expiration time * Adding expiration time * Refactoring code * Refactoring code * Refactoring code * Refactoring code * Refactoring code * Refactoring code * Code review changes * Code review changes * Code review changes * Code review changes * Adding db transactions * Fix bug when database is locked * Fix tests * Adding BATON_IN_MEMORY_HTTP_CACHE env * Adding stats structure * Adding statistics * Adding statistics * Adding statistics * Adding statistics * Refactoring code * Refactoring code * Refactoring code * Refactoring code * Code review changes * Refactoring code * Code review changes * Code review changes part 1 * Code review changes part 2 * Code review changes part 3 * Code review changes part 3 * Code review changes part 4 * Code review changes part 4 * Code review changes part 5 * Code review changes part 5 * Code review changes part 5 * Code review changes part 5 * Code review changes part 6 * Refactoring code * Refactoring code * Code review changes * Code review changes * Code review changes * Refactoring code * Refactoring code * Refactoring code * Refactoring code * Refactoring code * Refactoring code * Refactoring code * Refactoring code * Use goqu for sqlite like we do elsewhere in baton-sdk, removing the direct dependency on sqlite3. Start cleaning up DB cache code. * Fix linter error * Fix error where failing tests didn't cause CI to fail. * Upgrade github action versions. * Use latest version of golang-test-annotations. Output test results in a windows-compatible way. * Truncate tables on db cache initialization. Test setting same cache key again. * Update on conflict. Set default cache size to 50MB. * Clean up cache interface. Accept request instead of having to generate a cache key and pass in req.Context(). * Clean up cache config. Add noop cache instead of initializing a disabled mem or db cache. * Move cache config stuff into cache file. Don't create cachekey since get/set cache just needs req. Add tostring method to cache config. * Remove unused functions. * Use one variable for the cache we're returning. * Move all env var parsing into NewCacheConfigFromEnv. Clean up names and add caching behavior config. * Add TODO. * Add explicit GRPC call to clear caches/cleanup. Print cache stats. * Lots of DB cache cleanup. Use higher log levels for errors. Simplify functions. * Better function name for deleting expired cache entries. Fix stats calculations. * Make insert only accept bytes. Clean up pick() to build query and not return an error if no cache entry is found. * Add support for sparse mode in memory cache. Reduce default cache size from 50MB to 5MB. * Use request url, not response request url. * Error if we can't create the http cache. * Remove unused SqliteError. Cache clearing log line should be debug, not info. --------- Co-authored-by: Geoff Greer <geoff@greer.fm>
ConductorOne · Nov 21, 2024 · 50a56ee · 50a56ee
1 parent 72aea11
commit 50a56ee
Show file tree

Hide file tree

Showing 18 changed files with 1,531 additions and 333 deletions.
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -5,11 +5,11 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Install Go
-        uses: actions/setup-go@v4
+        uses: actions/setup-go@v5
         with:
           go-version: 1.21.x
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
       - name: Run linters
         uses: golangci/golangci-lint-action@v6
         with:
@@ -24,23 +24,26 @@ jobs:
     steps:
       - name: Install Go
         if: success()
-        uses: actions/setup-go@v4
+        uses: actions/setup-go@v5
         with:
           go-version: ${{ matrix.go-version }}
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
       - name: go tests
-        run: go test -v -covermode=count -json ./... | tee test.json
+        run: go test -v -covermode=count -json ./... > test.json
+      - name: Print go test results
+        if: always()
+        run: cat test.json
       - name: annotate go tests
         if: always()
-        uses: guyarb/golang-test-annotations@v0.5.1
+        uses: guyarb/golang-test-annotations@v0.8.0
         with:
           test-results: test.json
   buf-lint-and-breaking-change-detection:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
       - name: Setup
         uses: bufbuild/buf-setup-action@v1
         with:

diff --git a/.github/workflows/main.yaml b/.github/workflows/main.yaml
@@ -8,7 +8,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Checkout
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
       - name: Setup
         uses: bufbuild/buf-setup-action@v1
         with:
@@ -25,13 +25,13 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Install Go
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v5
         with:
           go-version: 1.21.x
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
       - name: Run linters
-        uses: golangci/golangci-lint-action@v3
+        uses: golangci/golangci-lint-action@v6
         with:
           version: latest
           args: --timeout=3m
@@ -44,15 +44,18 @@ jobs:
     steps:
       - name: Install Go
         if: success()
-        uses: actions/setup-go@v2
+        uses: actions/setup-go@v5
         with:
           go-version: ${{ matrix.go-version }}
       - name: Checkout code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v4
       - name: go tests
-        run: go test -v -covermode=count -json ./... | tee test.json
+        run: go test -v -covermode=count -json ./... > test.json
+      - name: Print go test results
+        if: always()
+        run: cat test.json
       - name: annotate go tests
         if: always()
-        uses: guyarb/golang-test-annotations@v0.5.1
+        uses: guyarb/golang-test-annotations@v0.8.0
         with:
           test-results: test.json
diff --git a/go.mod b/go.mod
@@ -74,6 +74,7 @@ require (
 	github.com/lufia/plan9stats v0.0.0-20240408141607-282e7b5d6b74 // indirect
 	github.com/magiconair/properties v1.8.7 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/mattn/go-sqlite3 v1.14.22 // indirect
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
 	github.com/ncruces/go-strftime v0.1.9 // indirect
 	github.com/pelletier/go-toml/v2 v2.2.2 // indirect

diff --git a/go.sum b/go.sum
@@ -140,8 +140,9 @@ github.com/magiconair/properties v1.8.7 h1:IeQXZAiQcpL9mgcAe1Nu6cX9LLw6ExEHKjN0V
 github.com/magiconair/properties v1.8.7/go.mod h1:Dhd985XPs7jluiymwWYZ0G4Z61jb3vdS329zhj2hYo0=
 github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
 github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
-github.com/mattn/go-sqlite3 v1.14.7 h1:fxWBnXkxfM6sRiuH3bqJ4CfzZojMOLVc0UTsTglEghA=
 github.com/mattn/go-sqlite3 v1.14.7/go.mod h1:NyWgC/yNuGj7Q9rpYnZvas74GogHl5/Z4A/KQRfk6bU=
+github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU=
+github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
 github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
 github.com/mitchellh/mapstructure v1.5.0/go.mod h1:bFUtVrKA4DC2yAKiSyO/QUcy7e+RRV2QTWOzhPopBRo=
 github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=