I spun up a jellyfin container on my orange pi 5 on nixos, and it logged this:

Oct 16 19:03:17 orangutan jellyfin[1380]: [23:03:17] [INF] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Found ffmpeg version 6.0.1
Oct 16 19:03:17 orangutan jellyfin[1380]: [23:03:17] [INF] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available decoders: ["libdav1d", "av1", "av1_rkmpp", "h264", "h264_rkmpp", "hevc", "hevc_rkmpp", "mpeg1_rkmpp", "mpeg2video", "mpeg2_rkmpp", "mpeg4", "mpeg4_rkmpp", "msmpeg4", "vp8", "vp8_rkmpp", "libvpx", "vp9", "vp9_rkmpp", "libvpx-vp9", "aac", "ac3", "dca", "flac", "mp3", "truehd"]
Oct 16 19:03:17 orangutan jellyfin[1380]: [23:03:17] [INF] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available encoders: ["libsvtav1", "libx264", "h264_v4l2m2m", "h264_rkmpp", "libx265", "hevc_rkmpp", "mpeg4", "msmpeg4", "libvpx", "libvpx-vp9", "aac", "libfdk_aac", "ac3", "alac", "dca", "flac", "libmp3lame", "libopus", "truehd", "libvorbis", "srt"]
Oct 16 19:03:17 orangutan jellyfin[1380]: [23:03:17] [INF] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available filters: ["overlay_opencl", "overlay_rkrga", "scale_opencl", "scale_rkrga", "tonemap_opencl", "vpp_rkrga", "zscale", "alphasrc"]
Oct 16 19:03:17 orangutan jellyfin[1380]: [23:03:17] [WRN] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Filter: scale_cuda with option Output format (default "same") is not available
Oct 16 19:03:17 orangutan jellyfin[1380]: [23:03:17] [WRN] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Filter: tonemap_cuda with option GPU accelerated HDR to SDR tonemapping is not available
Oct 16 19:03:18 orangutan jellyfin[1380]: [23:03:18] [WRN] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Filter: overlay_vaapi with option Action to take when encountering EOF from secondary input is not available
Oct 16 19:03:18 orangutan jellyfin[1380]: [23:03:18] [WRN] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Filter: overlay_vulkan with option Action to take when encountering EOF from secondary input is not available
Oct 16 19:03:18 orangutan jellyfin[1380]: [23:03:18] [INF] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available hwaccel types: ["drm", "opencl", "rkmpp"]
Oct 16 19:03:18 orangutan jellyfin[1380]: [23:03:18] [INF] [4] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: FFmpeg: /usr/lib/jellyfin-ffmpeg/ffmpeg
Oct 16 19:03:18 orangutan jellyfin[1380]: [23:03:18] [INF] [4] Emby.Server.Implementations.ApplicationHost: ServerId: 75f42f46993e44689bc26fe7756194d4

Notably, it does appear to support hardware av1, h264, h265, vp8, and vp9 decoding, and h264 and h265 encoding (the "_rkmpp" suffix denotes rockchip hw acceleration)

 MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Found ffmpeg version 6.0.1
 MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available decoders: ["libdav1d", "av1", "av1_cuvid", "h264", "h264_cuvid", "hevc", "hevc_cuvid", "mpeg2video", "mpeg2_cuvid", "mpeg4", "mpeg4_cuvid", "msmpeg4", "vc1_cuvid", "vp8", "libvpx", "vp8_cuvid", "vp9", "libvpx-vp9", "vp9_cuvid", "aac", "ac3", "dca", "flac", "mp3", "truehd"]
 MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available encoders: ["av1_nvenc", "av1_vaapi", "libx264", "h264_nvenc", "h264_v4l2m2m", "h264_vaapi", "libx265", "hevc_nvenc", "hevc_vaapi", "mjpeg_vaapi", "mpeg4", "msmpeg4", "libvpx", "libvpx-vp9", "aac", "ac3", "alac", "dca", "flac", "libmp3lame", "libopus", "truehd", "libvorbis", "srt"]
 MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available filters: ["deinterlace_vaapi", "hwupload_cuda", "hwupload_vaapi", "libplacebo", "overlay_opencl", "overlay_vaapi", "overlay_vulkan", "overlay_cuda", "procamp_vaapi", "scale_cuda", "scale_opencl", "scale_vaapi", "scale_vulkan", "tonemap_cuda", "tonemap_opencl", "tonemap_vaapi", "yadif_cuda", "zscale", "alphasrc"]
 MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available hwaccel types: ["vdpau", "cuda", "vaapi", "drm", "opencl", "vulkan"]

[ 12.685043] hantro-vpu fdc70000.video-codec: registered rockchip,rk3588-av1-vpu-dec as /dev/video0

In reply to @hexa:lossy.network
with just services.jellyfin.enable on the rock 5b with mainline 6.11.3

In reply to @hexa:lossy.network
hm

I get the same when I use services.jellyfin.enable:

journalctl -xeu jellyfin | grep -A 5 "Found ffmpeg"
Oct 16 19:20:28 orangutan jellyfin[12334]: [19:20:28] [INF] [6] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Found ffmpeg version 6.0.1
Oct 16 19:20:28 orangutan jellyfin[12334]: [19:20:28] [INF] [6] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available decoders: ["libdav1d", "av1", "av1_cuvid", "h264", "h264_cuvid", "hevc", "hevc_cuvid", "mpeg2video", "mpeg2_cuvid", "mpeg4", "mpeg4_cuvid", "msmpeg4", "vc1_cuvid", "vp8", "libvpx", "vp8_cuvid", "vp9", "libvpx-vp9", "vp9_cuvid", "aac", "ac3", "dca", "flac", "mp3", "truehd"]
Oct 16 19:20:28 orangutan jellyfin[12334]: [19:20:28] [INF] [6] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available encoders: ["av1_nvenc", "av1_vaapi", "libx264", "h264_nvenc", "h264_v4l2m2m", "h264_vaapi", "libx265", "hevc_nvenc", "hevc_vaapi", "mjpeg_vaapi", "mpeg4", "msmpeg4", "libvpx", "libvpx-vp9", "aac", "ac3", "alac", "dca", "flac", "libmp3lame", "libopus", "truehd", "libvorbis", "srt"]
Oct 16 19:20:29 orangutan jellyfin[12334]: [19:20:29] [INF] [6] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available filters: ["deinterlace_vaapi", "hwupload_cuda", "hwupload_vaapi", "libplacebo", "overlay_opencl", "overlay_vaapi", "overlay_vulkan", "overlay_cuda", "procamp_vaapi", "scale_cuda", "scale_opencl", "scale_vaapi", "scale_vulkan", "tonemap_cuda", "tonemap_opencl", "tonemap_vaapi", "yadif_cuda", "zscale", "alphasrc"]
Oct 16 19:20:30 orangutan jellyfin[12334]: [19:20:30] [INF] [6] MediaBrowser.MediaEncoding.Encoder.MediaEncoder: Available hwaccel types: ["vdpau", "cuda", "vaapi", "drm", "opencl", "vulkan"]
Oct 16 19:20:31 orangutan jellyfin[12334]: [19:20:31] [INF] [4] Emby.Server.Implementations.ScheduledTasks.TaskManager: Clean up collections and playlists Completed after 0 minute(s) and 0 seconds

I can confirm hardware encoding appears to be working on the rk3588 with joshua-riek's 6.1 bsp kernel

Oct 16 19:31:16 orangutan jellyfin[13855]: [23:31:16] [INF] [30] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: /usr/lib/jellyfin-ffmpeg/ffmpeg -analyzeduration 200M -probesize 1G -ss 00:03:12.000 -noaccurate_seek -init_hw_device rkmpp=rk -hwaccel rkmpp -hwaccel_output_format drm_prime -afbc rga -noautorotate -i file:"/media/media/bbb_sunflower_1080p_60fps_normal.mp4" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:s -codec:v:0 hevc_rkmpp -tag:v:0 hvc1 -b:v 1116000 -maxrate 1116000 -bufsize 2232000 -profile:v:0 main -level 150 -g:v:0 180 -keyint_min:v:0 180 -vf "setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_rkrga=w=720:h=404:format=nv12:afbc=1" -codec:a:0 libfdk_aac -ac 2 -ab 256000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type fmp4 -hls_fmp4_init_filename "f7e1d02a5d90104b98e2077d78225449-1.mp4" -start_number 64 -hls_segment_filename "/cache/transcodes/f7e1d02a5d90104b98e2077d78225449%d.mp4" -hls_playlist_type vod -hls_list_size 0 -y "/cache/transcodes/f7e1d02a5d90104b98e2077d78225449.m3u8"

with the oci container image