Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make (msg *Message) totalSize() public #1254

Open
taguhoiya opened this issue Jan 9, 2024 · 0 comments · May be fixed by #1337
Open

Make (msg *Message) totalSize() public #1254

taguhoiya opened this issue Jan 9, 2024 · 0 comments · May be fixed by #1337

Comments

@taguhoiya
Copy link

taguhoiya commented Jan 9, 2024

Suggestion

  • I would like to make totalSize() public to get to know the exact size of kakfa.Message
  • now the method of knowing the message size is the package-private one and too complicated to implement on the client side.
  • (I actually used totalSize() in my repository and had to copy and paste many methods all the way)

can we make totalSize() public?

reference

here are all the methods that I personally implemented and I copied and pasted all methods literally from the https://github.com/segmentio/kafka-go/blob/main/message.go

source code
package kafkas

import (
  "github.com/segmentio/kafka-go"
)

// This file is all implemented by https://github.com/segmentio/kafka-go/blob/main/message.go
const timestampSize = 8

func varIntLen(i int64) int {
  u := uint64((i << 1) ^ (i >> 63)) // zig-zag encoding
  n := 0

  for u >= 0x80 {
  	u >>= 7
  	n++
  }

  return n + 1
}

func varBytesLen(b []byte) int {
  return varIntLen(int64(len(b))) + len(b)
}

func varStringLen(s string) int {
  return varIntLen(int64(len(s))) + len(s)
}

func varArrayLen(n int, f func(int) int) int {
  size := varIntLen(int64(n))
  for i := 0; i < n; i++ {
  	size += f(i)
  }
  return size
}

func sizeofBytes(b []byte) int32 {
  return 4 + int32(len(b))
}

func size(msg *kafka.Message) int32 {
  return 4 + 1 + 1 + sizeofBytes(msg.Key) + sizeofBytes(msg.Value) + timestampSize
}

func headerSize(msg *kafka.Message) int {
  return varArrayLen(len(msg.Headers), func(i int) int {
  	h := &msg.Headers[i]
  	return varStringLen(h.Key) + varBytesLen(h.Value)
  })
}

// Make it public for external packages
func TotalSize(msg *kafka.Message) int32 {
  return int32(headerSize(msg)) + size(msg)
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant