Dear Luckfox Support Team,
I am currently developing an application on the Luckfox Pico Ultra W (RV1106) that involves capturing video, encoding it to H.264 using RKMPI, and then streaming it via WebRTC using the libdatachannel library.
I am using the RK_MPI_VENC_GetStream function to retrieve the encoded H.264 data. To correctly packetize this data for WebRTC (specifically for libdatachannel's H264RtpPacketizer), I need to know the exact format of the H.264 elementary stream produced by the VENC and made available through this RKMPI call.
Could you please clarify the following:
1. H.264 Stream Format: Is the H.264 output from RK_MPI_VENC_GetStream in Annex B format ? Or is it in another format, such as AVCC format ?
2.Data Structure from RK_MPI_VENC_GetStream: When RK_MPI_VENC_GetStream returns a VENC_STREAM_S structure, does the data pointed to by pstPack->pMbBlk (after RK_MPI_MB_Handle2VirAddr) represent:
A single NAL unit?
Multiple NAL units concatenated (an Access Unit / a full frame), and if so, how are they separated within this data block?
Does it always include SPS/PPS with I-frames, or do these need to be retrieved separately?
3. Recommended libdatachannel Configuration: Given the output format, if I'm using libdatachannel's H264RtpPacketizer, would you recommend configuring its NAL unit separator as rtc::NalUnit::Separator::StartCode or rtc::NalUnit::Separator::Length?
4. Conversion (if necessary): If the output is Annex B and the WebRTC library I'm using (or a specific version of it) only robustly supports length-prefixed NAL units for its packetizer, are there any RKMPI settings or recommended utility functions/libraries for the RV1106 platform to efficiently convert the VENC output from Annex B to a length-prefixed format?
I am encountering "Invalid NAL Unit data" warnings with libdatachannel when its packetizer is set to expect length-prefixed NALs, which strongly suggests the VENC output is Annex B. However, I've also encountered compile issues when trying to use the StartCode separator option with my current version of libdatachannel (v0.22.6), so I'm trying to understand the best path forward.
Any detailed information or examples you can provide regarding the VENC H.264 output format via RKMPI would be extremely helpful for my WebRTC integration.
Thank you for your time and support.
Pico Ultra W RV1106 VENC H.264 Output Format for RKMPI (RK_MPI_VENC_GetStream) and WebRTC Integration
Hello, the details we have regarding RV1106 VENC are limited to the documentation. Most of your questions can be resolved by parsing the relevant structures in the RKMPI documentation, or by saving H264 locally and using H264BSAnalyzer for parsing. The VENC operation details not mentioned in the documentation should be consulted with Rockchip. We were not involved in the related development.
Based on your question, I provided a simple reply based on my past experience. For unclear details, you need to conduct experiments on your own to determine them:
1. The H264 saved locally can be determined by H264BSAnalyzer that SPS, PPS, and IDR do not contain NALU length information. It should be in Annex B format.
2. RK_MPI_VENC_GetStream retrieves the data of one frame. One frame of data may contain multiple NALUs.
3. I have no experience with libdatachannel and cannot provide suggestions.
4. I am not sure if there is a method for VENC to output AVCC encapsulation.
Based on your question, I provided a simple reply based on my past experience. For unclear details, you need to conduct experiments on your own to determine them:
1. The H264 saved locally can be determined by H264BSAnalyzer that SPS, PPS, and IDR do not contain NALU length information. It should be in Annex B format.
2. RK_MPI_VENC_GetStream retrieves the data of one frame. One frame of data may contain multiple NALUs.
3. I have no experience with libdatachannel and cannot provide suggestions.
4. I am not sure if there is a method for VENC to output AVCC encapsulation.
Thank you for your previous response regarding VENC output. I was able to resolve that using libdatachannel due to its Annex B support.
I am now encountering new issues when streaming video via WebRTC using libdatachannel on our Luckfox Pico Ultra W (RV1106) board. While the video stream starts and is visible on the web client, we experience:
1.Very high latency.
2.The connection disconnects automatically after approximately 1 minute. Board logs indicate this is due to STUN Consent expired. (We have also previously observed juice: Send failed, buffer is full errors, especially during motion in the video).
I suspect system performance limitations might be contributing. Below is some diagnostic information captured while running our application (luckfox_pico_rtsp_yolov5) with an active WebRTC connection:
top output:
Key observations from top:
- The Load average is extremely high (e.g., 11.11).
- Our main application (luckfox_pico_rtsp_yolov5, PID 1346) is often in an uninterruptible sleep state (STAT: D), indicating it's waiting for I/O, even though its own CPU usage might be ~19% at that moment and overall CPU idle is high.
dmesg output snippet:
(I will attach the full dmesg log if needed, but there is a very notable line)
I have the following questions:
1.Could the EXT4-fs errors on the mmcblk0p7 partition and the resulting high Load average be a primary cause for our application entering the STAT: D (I/O wait) state and contributing to the overall WebRTC instability (high latency, STUN consent expired disconnects)? What is the recommended procedure for running fsck on this partition for the RV1106?
2.Regarding our application (luckfox_pico_rtsp_yolov5) being in STAT: D: Is this a known behavior, or can you provide insights into what specific I/O operations (e.g., camera access via RKISP/RKCIF, MPP hardware encoder, or storage access) might be causing bottlenecks on the RV1106 under real-time streaming load?
3.For achieving low-latency WebRTC streaming with libdatachannel (which utilizes libjuice and the Rockchip MPP for H.264 encoding on the RV1106), what are your recommended:
H.264 encoding parameters (resolution, FPS, target bitrate, GOP structure, quality/rate control settings)?
Strategies for managing network congestion or adaptively adjusting bitrate?
4.Considering the top output (high load average, application in STAT: D), do you believe the primary bottleneck is CPU processing power, or is the I/O wait (indicated by STAT: D and filesystem errors) a more pressing concern to address first?
I would appreciate any guidance or recommendations you can provide to help us achieve stable, low-latency WebRTC video streaming.
Thank you for your time and support.
I am now encountering new issues when streaming video via WebRTC using libdatachannel on our Luckfox Pico Ultra W (RV1106) board. While the video stream starts and is visible on the web client, we experience:
1.Very high latency.
2.The connection disconnects automatically after approximately 1 minute. Board logs indicate this is due to STUN Consent expired. (We have also previously observed juice: Send failed, buffer is full errors, especially during motion in the video).
I suspect system performance limitations might be contributing. Below is some diagnostic information captured while running our application (luckfox_pico_rtsp_yolov5) with an active WebRTC connection:
top output:
Code: Select all
Mem: 90636K used, 84580K free, 876K shrd, 6312K buff, 51100K cached
CPU: 17% usr 7% sys 0% nic 74% idle 0% io 0% irq 0% sirq
Load average: 11.11 10.75 10.29 2/123 1531
PID PPID USER STAT VSZ %VSZ %CPU COMMAND
1346 1574 root D 87308 50% 19% ./luckfox_pico_rtsp_yolov5
424 2 root SW 0 0% 5% [aicwf_bustx_thr]
1555 289 root S 5276 3% 0% sshd: root@pts/1
1354 289 root S 5264 3% 0% sshd: root@pts/0
1557 1369 root R 1800 1% 0% top
255 2 root SW 0 0% 0% [vcodec_thread_0]
252 2 root SW 0 0% 0% [queue_work0]
431 2 root SW 0 0% 0% [irq/67-dwc3]
425 2 root SW 0 0% 0% [aicwf_busrx_thr]
3514 2 root IW 0 0% 0% [kworker/0:3-eve]
423 2 root SW 0 0% 0% [aicwf_pwrctl]
563 1 root S 33016 19% 0% smbd -D
566 563 root S 32084 18% 0% {smbd-notifyd} smbd -D
567 563 root S 32076 18% 0% {smbd-cleanupd} smbd -D
648 1 root S 24600 14% 0% nmbd -D
415 1 root S 10220 6% 0% /usr/bin/adbd
942 289 root S 5272 3% 0% sshd: root@pts/2
460 1 root S 5192 3% 0% rkwifi_server start
289 1 root S 4964 3% 0% sshd: /usr/sbin/sshd [listener] 0 of 10-100 startups
182 1 root S 4924 3% 0% /usr/libexec/bluetooth/bluetoothd -n
1574 1555 root S 1836 1% 0% -sh
968 942 root S 1832 1% 0% -sh
1369 1354 root S 1828 1% 0% -sh
175 1 dbus S 1812 1% 0% dbus-daemon --system
1320 1 root S 1804 1% 0% {S99usb0config} /bin/sh /etc/init.d/S99usb0config start
1 0 root S 1796 1% 0% init
1321 1 root S 1792 1% 0% /sbin/getty -L console 0 vt100
86 1 root S 1788 1% 0% /sbin/syslogd -n
310 1 root S 1788 1% 0% /usr/sbin/telnetd -F
90 1 root S 1784 1% 0% /sbin/klogd -n
101 1 root S 1628 1% 0% /sbin/udevd -d
556 1 root S 1264 1% 0% wpa_supplicant -B -i wlan0 -c /data/wpa_supplicant.conf -d
668 1 root S 956 1% 0% hciattach -s 1500000 /dev/ttyS1 any 1500000 flow nosleep
1301 1 root S 768 0% 0% luckfox_switch_rgb_resolution
7 2 root SW 0 0% 0% [ksoftirqd/0]
272 2 root DW 0 0% 0% [vmcu]
1993 2 root IW 0 0% 0% [kworker/0:0-eve]
269 2 root DW 0 0% 0% [rkisp-vir0]
11 2 root SW 0 0% 0% [kcompactd0]
66 2 root IW< 0 0% 0% [kworker/0:1H-kb]
2339 2 root IW 0 0% 0% [kworker/u2:1-ev]
268 2 root DW 0 0% 0% [venc]
- The Load average is extremely high (e.g., 11.11).
- Our main application (luckfox_pico_rtsp_yolov5, PID 1346) is often in an uninterruptible sleep state (STAT: D), indicating it's waiting for I/O, even though its own CPU usage might be ~19% at that moment and overall CPU idle is high.
dmesg output snippet:
(I will attach the full dmesg log if needed, but there is a very notable line)
Code: Select all
[ 305.125606] EXT4-fs (mmcblk0p7): error count since last fsck: 471.Could the EXT4-fs errors on the mmcblk0p7 partition and the resulting high Load average be a primary cause for our application entering the STAT: D (I/O wait) state and contributing to the overall WebRTC instability (high latency, STUN consent expired disconnects)? What is the recommended procedure for running fsck on this partition for the RV1106?
2.Regarding our application (luckfox_pico_rtsp_yolov5) being in STAT: D: Is this a known behavior, or can you provide insights into what specific I/O operations (e.g., camera access via RKISP/RKCIF, MPP hardware encoder, or storage access) might be causing bottlenecks on the RV1106 under real-time streaming load?
3.For achieving low-latency WebRTC streaming with libdatachannel (which utilizes libjuice and the Rockchip MPP for H.264 encoding on the RV1106), what are your recommended:
H.264 encoding parameters (resolution, FPS, target bitrate, GOP structure, quality/rate control settings)?
Strategies for managing network congestion or adaptively adjusting bitrate?
4.Considering the top output (high load average, application in STAT: D), do you believe the primary bottleneck is CPU processing power, or is the I/O wait (indicated by STAT: D and filesystem errors) a more pressing concern to address first?
I would appreciate any guidance or recommendations you can provide to help us achieve stable, low-latency WebRTC video streaming.
Thank you for your time and support.
Since we cannot fully understand all the usage scenarios of the luckfox pico for each customer, we are unable to provide solutions to the problems encountered during the secondary development. Please refer to the https://wiki.luckfox.com/intro/ for details.
We can only respond to the content of the SDK and some of the experiences we have gained during our own development. Your problem mainly relates to libdatachannel and other issues arising during the usage process. We do not have relevant experience and therefore cannot provide specific suggestions.
If you don't insist on using libdatachannel to implement webrtc, you can refer to the JETKVM project on GitHub. It has implemented webrtc on the RV1106 platform based on pion.
We can only respond to the content of the SDK and some of the experiences we have gained during our own development. Your problem mainly relates to libdatachannel and other issues arising during the usage process. We do not have relevant experience and therefore cannot provide specific suggestions.
If you don't insist on using libdatachannel to implement webrtc, you can refer to the JETKVM project on GitHub. It has implemented webrtc on the RV1106 platform based on pion.
Cảm ơn bạn rất nhiều!