[Question] Difference between *TrackLocalStaticRTP and *TrackLocalStaticSample ???

[KyungWonPark]

Hello comrades

I am writing an application that does:
cam -> ffmpeg h264 -> pion webrtc server -> browser
thing.

I am learning with these two examples:
https://github.com/pion/webrtc/tree/master/examples/play-from-disk
https://github.com/pion/webrtc/tree/master/examples/broadcast

I have two questions.

1. Difference between *TrackLocalStaticRTP and *TrackLocalStaticSample ?

First example(play-from-disk) uses

webrtc/track_local_static.go

Line 248 in ee255e8

func (s *TrackLocalStaticSample) WriteSample(sample media.Sample) error {

Second example(broadcase) uses

webrtc/track_local_static.go

Line 151 in ee255e8

func (s *TrackLocalStaticRTP) Write(b []byte) (n int, err error) {

for writing video data to the peer.
What is the difference between these two functions?

2. What data should I provide to (*TrackLocalStaticRTP).Write() or (*TrackLocalStaticSample).WriteSample() function ?

I have a h264 stream which is coming from ffmpeg pipe (ffmpeg ... -f h264 pipe:1)
and media.H264Reader is parsing it as NAL packets with NextNAL() function.

webrtc/pkg/media/h264reader/h264reader.go

Lines 110 to 113 in ee255e8

	// NextNAL reads from stream and returns then next NAL,
	// and an error if there is incomplete frame data.
	// Returns all nil values when no more NALs are available.
	func (reader *H264Reader) NextNAL() (*NAL, error) {

Can I just provide NAL.Data (which is []byte) to (*TrackLocalStaticRTP).Write() ?
Should I provide []byte(NAL) as a whole to .Write() function?
or should I do further processing to wrap NAL into something digestible for browser?

Thank you in advance!

[caiiiycuk]

I have the same question. I am trying to build webrtc streaming service that uses directly nvenc encoder thats write into pion. I have a working solution that display a stream. However it produces big lags.

@ashellunts can you help? I used your example:

                // reader reads from nvenc pipe
		for {
			nal, err := reader.NextNAL()
			if err == io.EOF {
				log.Println("All video frames parsed and sent")
			}

			if err != nil {
				log.Fatal(err)
			}

			nal.Data = append([]byte{0x00, 0x00, 0x00, 0x01}, nal.Data...)
			if nal.UnitType == h264reader.NalUnitTypeSPS || nal.UnitType == h264reader.NalUnitTypePPS {
				spsAndPpsCache = append(spsAndPpsCache, nal.Data...)
				continue
			} else if nal.UnitType == h264reader.NalUnitTypeCodedSliceIdr {
				nal.Data = append(spsAndPpsCache, nal.Data...)
				spsAndPpsCache = []byte{}
			}

			if err = videoTrack.WriteSample(media.Sample{Data: nal.Data, Timestamp: time.Now()}); err != nil {
				log.Fatal(err)
			}
		}

It renders correct picture through webrtc only if I used NV_ENC_H264_PROFILE_BASELINE_GUID or NV_ENC_H264_PROFILE_CONSTRAINED_HIGH_GUID.
If I use NV_ENC_H264_PROFILE_PROGRESSIVE_HIGH_GUID profile the image is corrupted:

1. Is it possible to use PROGRESSIVE profile with this h264reader?

Even if I have correct picture I see a strong lags in streaming. The picture is often outdated or can freeze for some time. Why it can happen? I am sure that nvenc is working correctly, it encodes 60 frames per second. I immediately read them and pass to WriteSample. If I do it through ffmpeg -> rtp -> pion there are no lags at all. nvenc send IDR, SPS, PPS periodically.

Also it's not clear what I should pass to Timestamp, Duration fields?

2. So why I experience strong lags with this code?

[KyungWonPark]

I have the same question. I am trying to build webrtc streaming service that uses directly nvenc encoder thats write into pion. I have a working solution that display a stream. However it produces big lags.

@ashellunts can you help? I used your example:

                // reader reads from nvenc pipe
		for {
			nal, err := reader.NextNAL()
			if err == io.EOF {
				log.Println("All video frames parsed and sent")
			}

			if err != nil {
				log.Fatal(err)
			}

			nal.Data = append([]byte{0x00, 0x00, 0x00, 0x01}, nal.Data...)
			if nal.UnitType == h264reader.NalUnitTypeSPS || nal.UnitType == h264reader.NalUnitTypePPS {
				spsAndPpsCache = append(spsAndPpsCache, nal.Data...)
				continue
			} else if nal.UnitType == h264reader.NalUnitTypeCodedSliceIdr {
				nal.Data = append(spsAndPpsCache, nal.Data...)
				spsAndPpsCache = []byte{}
			}

			if err = videoTrack.WriteSample(media.Sample{Data: nal.Data, Timestamp: time.Now()}); err != nil {
				log.Fatal(err)
			}
		}

It renders correct picture through webrtc only if I used NV_ENC_H264_PROFILE_BASELINE_GUID or NV_ENC_H264_PROFILE_CONSTRAINED_HIGH_GUID.
If I use NV_ENC_H264_PROFILE_PROGRESSIVE_HIGH_GUID profile the image is corrupted:

1. Is it possible to use PROGRESSIVE profile with this h264reader?

Even if I have correct picture I see a strong lags in streaming. The picture is often outdated or can freeze for some time. Why it can happen? I am sure that nvenc is working correctly, it encodes 60 frames per second. I immediately read them and pass to WriteSample. If I do it through ffmpeg -> rtp -> pion there are no lags at all. nvenc send IDR, SPS, PPS periodically.

Also it's not clear what I should pass to Timestamp, Duration fields?

2. So why I experience strong lags with this code?

Hello comrade.

I've found the answer for my second question.
https://github.com/baiyufei/pion-h264-repro
This nice guy made working pion-h264 example.
You don't have to append something to NAL data.

By the way, I also try to use nvenc but currently using ffmpeg with linux pipe for I'm not familiar with libav.
If you are using libav C++ code for nvenc encoding with cgo or swig, could you share your example?
That would have less latency than pipe method.

Tank you

[wangborong12345]

Where is the answer to the second question?h264 latency is still very high, but when I changed to vp8, there was almost no latency.