Luckfox Forums

Posted: **2024-07-22 14:01**

Hello! It's been a week since i got my pico mini b board, and i have a little bit of success in capturing images from camera using only V4L2 driver API. I don't want to use RKMPI sdk as it is not documented in english, and there is no opportunity to use ffmpeg bc V4L2 driver is not compatible with it by some reason. For now i have 2 major problems:
1) I can't control ISP parameters except exposure and analogue_gain that are provided for controlling via V4L2 ioctl's to /dev/v4l-subdev2 which is camera controller. Why there is only 2 sensor parameters that i can set? Other ones are not related to sensor parameters or marked as read-only

2) I have no clue how to access hardware h264/h265 encoder. There is magic device called /dev/vcodec in my device tree, but i dont have any information if this is right thing. ffmpeg is trying to seek for encoder device when i provide m2m encoder (which should be bound to hardware), but it is only looking for /dev/video* devices. Using software h264 encoder is highly discouraged as it is very slow for this tiny SoC rv1103. Maybe someone had an experience in using hardware rockchip encoder via driver API or other multimedia frameworks?
this is the output of ffmpeg command to encode raw NV21 video:

Code: Select all

ffmpeg -f rawvideo -c:v hevc_v4l2m2m -pixel_format nv21 -v
ideo_size 480x320 -i out.raw out.mp4 -loglevel debug

Code: Select all

Splitting the commandline.
Reading option '-f' ... matched as option 'f' (force format) with argument 'rawvideo'.
Reading option '-c:v' ... matched as option 'c' (codec name) with argument 'hevc_v4l2m2m'.
Reading option '-pixel_format' ... matched as AVOption 'pixel_format' with argument 'nv21'.
Reading option '-video_size' ... matched as AVOption 'video_size' with argument '480x320'.
Reading option '-i' ... matched as input url with argument 'out.raw'.
Reading option 'out.mp4' ... matched as output url.
Reading option '-loglevel' ... matched as option 'loglevel' (set logging level) with argument 'debug'.
Finished splitting the commandline.
Parsing a group of options: global .
Applying option loglevel (set logging level) with argument debug.
Successfully parsed a group of options.
Parsing a group of options: input url out.raw.
Applying option f (force format) with argument rawvideo.
Applying option c:v (codec name) with argument hevc_v4l2m2m.
Successfully parsed a group of options.
Opening an input file: out.raw.
[rawvideo @ 0x5083d0] Opening 'out.raw' for reading
[file @ 0x508bc0] Setting default whitelist 'file,crypto,data'
[rawvideo @ 0x5083d0] Before avformat_find_stream_info() pos: 0 bytes read:32768 seeks:0 nb_streams:1
[extract_extradata @ 0x510e40] No start code is found.
[rawvideo @ 0x5083d0] After avformat_find_stream_info() pos: 230400 bytes read:230400 seeks:0 frames:0
out.raw: could not find codec parameters
Input #0, rawvideo, from 'out.raw':
  Duration: N/A, bitrate: N/A
  Stream #0:0, 0, 1/25: Video: hevc, 1 reference frame, nv21, 480x320 (0x0), 0/1, 46080 kb/s, 25 tbn
Successfully opened the file.
Parsing a group of options: output url out.mp4.
Successfully parsed a group of options.
Opening an output file: out.mp4.
[file @ 0x54cd80] Setting default whitelist 'file,crypto,data'
Successfully opened the file.
[hevc_mp4toannexb @ 0x508210] The input looks like it is Annex B already
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video20
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp-input-params' in unknown mode
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video19
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp-statistics' in unknown mode
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video18
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp_rawrd2_s' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video17
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp_rawrd0_m' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video16
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp_lumapath' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video15
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp_bypasspath_4x4sampling' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video14
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp_mainpath_4x4sampling' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video13
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp_bypasspath' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video12
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp_selfpath' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video11
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkisp_v7' on card 'rkisp_mainpath' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video10
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video9
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video8
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video7
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video6
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video5
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video4
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video3
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video2
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video1
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] probing device /dev/video0
[hevc_v4l2m2m @ 0x54a4f0] driver 'rkcif' on card 'rkcif' in mplane mode
[hevc_v4l2m2m @ 0x54a4f0] v4l2 output format not supported
[hevc_v4l2m2m @ 0x54a4f0] Could not find a valid device
[hevc_v4l2m2m @ 0x54a4f0] can't configure decoder
Stream mapping:
  Stream #0:0 -> #0:0 (hevc (hevc_v4l2m2m) -> h264 (libx264))
Error while opening decoder for input stream #0:0 : Invalid argument
[AVIOContext @ 0x54cde0] Statistics: 0 seeks, 0 writeouts
[AVIOContext @ 0x510d70] Statistics: 230400 bytes read, 0 seeks

Posted: **2024-07-31 6:38**

Hello,

Based on the current available information, using hardware encoding on the RV1106 relies on the RKMPI library. The ISP and hardware encoding parts of the RV1106 have not been open-sourced, and there isn't a better solution at the moment to access the ISP and hardware encoding. Rockchip is still updating the RKMPI library, and in the short term, they are unlikely to release an mpp library that supports the RV1106, which would enable ffmpeg hardware encoding similar to https://github.com/nyanmisaka/ffmpeg-rockchip on the luckfox-pico.

If you require hardware encoding, it is recommended that you explore using RKMPI based on the available demos

Posted: **2024-08-16 12:47**

Crocodile wrote: ↑2024-07-31 6:38 Hello,

Based on the current available information, using hardware encoding on the RV1106 relies on the RKMPI library. The ISP and hardware encoding parts of the RV1106 have not been open-sourced, and there isn't a better solution at the moment to access the ISP and hardware encoding. Rockchip is still updating the RKMPI library, and in the short term, they are unlikely to release an mpp library that supports the RV1106, which would enable ffmpeg hardware encoding similar to https://github.com/nyanmisaka/ffmpeg-rockchip on the luckfox-pico.

If you require hardware encoding, it is recommended that you explore using RKMPI based on the available demos

Thank you for your answer! Is there documentation of RKMPI available in English?

Posted: **2024-08-17 9:14**

The RKMPI documentation is currently not available in English.

Posted: **2024-08-19 7:16**

Luckfox Taylor wrote: ↑2024-08-17 9:14 The RKMPI documentation is currently not available in English.

Ok. Thank you anyways. As i can see, RKMPI code and libraries are built-in with your buildroot sdk. So i don't need to download anything else, just find RKMPI headers and link with librockiva, librockit, librockchip_mpp and librkaiq?

Posted: **2024-08-19 8:08**

The RKMPI layer has been encapsulated through multiple layers, and in practice, it needs to be used in conjunction with other dynamic libraries (such as the RGN library). Therefore, in actual use, it's necessary to adjust the linked libraries and header files. You can refer to the example provided at https://wiki.luckfox.com/zh/Luckfox-Pico/RKMPI-example, where the CMakeLists.txt used can serve as a reference for custom projects.

Posted: **2024-08-19 8:45**

Crocodile wrote: ↑2024-08-19 8:08 The RKMPI layer has been encapsulated through multiple layers, and in practice, it needs to be used in conjunction with other dynamic libraries (such as the RGN library). Therefore, in actual use, it's necessary to adjust the linked libraries and header files. You can refer to the example provided at https://wiki.luckfox.com/zh/Luckfox-Pico/RKMPI-example, where the CMakeLists.txt used can serve as a reference for custom projects.

Ok. Thank you very much! I hope i'll will be able to build working pipeline with your library eventually

Luckfox Forums

Pico hardware encoder

Pico hardware encoder

Re: Pico hardware encoder

Re: Pico hardware encoder

Re: Pico hardware encoder

Re: Pico hardware encoder

Re: Pico hardware encoder

Re: Pico hardware encoder