It's looks like some electrical/interference/signal integrity issue. After removing RJ45 magnetics and connecting directly to media converter, packet loss dropped to 0.1-1%.
Btw, with luckfox pico mini, there are no packet loss.

While developing a custom video streaming app using gstreamer I encounter problem. Using udpsink to send jpeg compressed video stream to host PC via Ethernet cable (both PC and board connected to router in local network with 1m cat5 cables), all works fine. I get solid 30 fps, network shows around ...