Repeated error and warning concerning zm_packetqueue.cpp

[andornaut]

Describe Your Environment

• ZoneMinder 1.36.1
• Running in Docker on a customized image based on zoneminderhq/zoneminder:latest-ubuntu18.04, but upgraded to v1.36.1 using ppa:iconnor/zoneminder-1.36.
• Ubuntu bionic
• Linux 5.4.0-73-generic Update zmupdate.pl.in #82-Ubuntu SMP Wed Apr 14 17:39:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Firefox v88.0.1

If the issue concerns a camera

I have 3 cameras, which all appear to be working as expected.

Describe the bug

AFAICT, everything is working correctly (I can stream, I see thumbnails in the Console, etc), but the logs (via the Web UI) are filling up with repeated entries like the following:

2021-05-22 20:32:14 | zmc_m3 |   | 514 | ERR |   | zm_packetqueue.cpp | 135
-- | -- | -- | -- | -- | -- | -- | --
2021-05-22 20:32:14 | zmc_m3 |   | 514 | WAR |   | zm_packetqueue.cpp | 95

Note that this issue did not occur when I was running ZoneMinder v1.34.x

[welcome]

Thanks for opening your first issue here! Just a reminder, this forum is for Bug Reports only. Be sure to follow the issue template!

[CyberGuerro]

Same issue here on Gentoo Linux 2.7 Kernel 5.4.109-gentoo (SMP) x86_64

[MoHf7f4]

I'm seeing the same issue and my syslog is filling with these:
May 23 16:26:06 GlassEyes zmc_m16[8063]: ERR [zmc_m16] [Unable to free up older packets. Not queueing this video packet.]
May 23 16:26:06 GlassEyes zmc_m22[885]: ERR [zmc_m22] [Unable to free up older packets. Not queueing this video packet.]
May 23 16:26:06 GlassEyes zmc_m18[858]: ERR [zmc_m18] [Unable to free up older packets. Not queueing this video packet.]
May 23 16:26:06 GlassEyes zmc_m22[885]: ERR [zmc_m22] [Unable to free up older packets. Not queueing this video packet.]
May 23 16:26:06 GlassEyes zmc_m22[885]: ERR [zmc_m22] [Unable to free up older packets. Not queueing this video packet.]
May 23 16:26:06 GlassEyes zmc_m18[858]: ERR [zmc_m18] [Unable to free up older packets. Not queueing this video packet.]
May 23 16:26:07 GlassEyes zmc_m22[885]: ERR [zmc_m22] [Unable to free up older packets. Not queueing this video packet.]
May 23 16:26:07 GlassEyes zmc_m18[858]: ERR [zmc_m18] [Unable to free up older packets. Not queueing this video packet.]
May 23 16:26:07 GlassEyes zmc_m22[885]: ERR [zmc_m22] [Unable to free up older packets. Not queueing this video packet.]
May 23 16:26:07 GlassEyes zmc_m16[8063]: ERR [zmc_m16] [Unable to free up older packets. Not queueing this video packet.]

I have plenty of RAM

*@glasseyes:~# free -h
total used free shared buff/cache available
Mem: 23G 12G 5.2G 369M 5.5G 9.9G
Swap: 15G 0B 15G

version: 1.36.1-bionic1
Linux glasseyes 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.5 LTS
Release: 18.04

happy to dig further or test.

[connortechnology]

Increase MaxImageBufferCount.

[andornaut]

Increase MaxImageBufferCount.

That worked, thanks!

Suggested improvements:

• Include the error message in the web logs (please see the screenshot in the issue description)
• [UX] Add a link to the FAQ, or a description of the workaround/fix (ie. "Increase MaxImageBufferCount") to the error message.

[MoHf7f4]

I bumped to the sum of all the other frame settings and errors seem to have stopped.

Big thanks

[CyberGuerro]

@MoHf7f4 can you explain what do you mean with "I bumped to the sum of all the other frame settings"
@connortechnology, which is the correct value for the MaxImageBufferCount?

[MoHf7f4]

@MoHf7f4 can you explain what do you mean with "I bumped to the sum of all the other frame settings"
@connortechnology, which is the correct value for the MaxImageBufferCount?
@CyberGuerro I was making a best guess, here's what I did.
Sumed:
Image Buffer Size (frames)
Pre Event Image Count
Post Event Image Count
Stream Replay Image Buffer

and put that in: Maximum Image Buffer Size (frames)

[CyberGuerro]

@MoHf7f4 thx for your explanation, I'll try that :-)
EDIT:
Unfortunely Issue persist..... ....I noticed that issue is present only when I activate MOTION DETECTION of my 10 monitors... If I set monitors to MONITOR mode, Issue stops.

Attached screenshot of one of my monitor's as sample

PS. Is it correct that Message field of issue's log entries is empty?

[ASoTNetworks]

I am having this issue too I have made MaxImageBufferCount the sum of those values and also tried making it very big much higher than the sum but with same results.

Not sure if this is related but I see events get ended before it reaches 10 minutes.

My syslog shows [You have set the max video packets in the queue to 300. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.]

[gerazo]

@ASoTNetworks
Increasing MaxImageBufferCount was the key for. However, the behavior of 1.36 changed compared to 1.34 in several ways:

• The shared memory usage was always MaxImageBufferCount in 1.34, now it is ImageBufferCount only
• MaxImageBufferCount had to be bigger than max GOP size, now this is not enough and you probably need 2x times the GOP size.
• In 1.34, you got a warning message explicitly telling what GOP size the decoder met, so you need bigger, now you only get a general message that X value is too small.
• When using H264 passthrough, it is quite impossible to do alarm mode adaptive video slicing without having a full GOP all the time in the buffer. So I think that ImageBufferCount should also be GOP size, but if you set it lower, you do not get any error messages. But I am not sure whether this would improve anything.

Anyway, you should take a look into your IP cam settings. Many modern cams have dynamic GOP, so although their default setting is 32, they can go up the 256. It does make sense to lower it if you use dynamic GOP or dynamic framerate.

[gerazo]

Well, I was wrong. After several hours, having MaxImageBufferCount 2x of the max GOP size also produced the error. Something fishy is going on here.
(Also note that I'm using 1.36.3 right now as being on the official Debian repo... and 1.36.4 seems to have some queue related fixes)

[gerazo]

I had more look into it: I am using Settings/EVENT_CLOSE_MODE/alarm to have videos separated when alarm kicks in or when it is over, so I can test how accurately the cut is done (again, by using h264 passthrough). When setting ImageBufferCount to a larger value, this cut is way off, or sometimes not even happening (usually at the end of the event). If ImageBufferCount is 3 or 5, the cut is mostly done correctly.
Now I did not have the above error for 24hours by having ImageBufferCount at 3 and MaxImageBufferCount 2x max GOP size. (and miraculously the memory consumption is only 3 frames large)

[CyberGuerro]

Excuse me all, can someone explain to me what does GOP mean?
So I can set correctly MaxImageBufferCount.

Thanx

[connortechnology]

GOP is Group Of Pictures. It is basically your keyframe interval. Set MaxImageBufferCount as high as your ram can sustain. Likely at least 2*GOP size but setting higher should be ok, it just sets the maximum in case slow disks, slow db or something else causes capturing to go faster than analysis.

[connortechnology]

Please update to 1.36.7 which massively reduces cpu use, which means zm can keep up with freeing the memory. It may resolve this issue.

[Tamahome-M]

i upgraded from zm-1.34.24 to 1.36.7. (at 1.34.24 everything was fine)
on 1.36.7 - 46GB RAM and kernel: zmc invoked oom-killer

Sep 23 16:04:40 nat zmc_m2[17785]: WAR [zmc_m2] [You have set the max video packets in the queue to 900. The queue is full. Either Analysis is not keeping up or your camera's keyframe interval is larger than this setting. We are dropping packets.]

[MoHf7f4]

I'm having no issues now. On 1.36.7.

[PhilippeRaven]

Hi there,
I recently updated from 1.34 to 1.36.10 on Ubuntu 20.04 through the PPA, and I got this issue too.

[hspaay]

Same issue here after upgrading to 1.36.12.
@gerazo had an interesting comment:

When using H264 passthrough, it is quite impossible to do alarm mode adaptive video slicing without having a full GOP all the time in the buffer. So I think that ImageBufferCount should also be GOP size, but if you set it lower, you do not get any error messages. But I am not sure whether this would improve anything.

While I don't understand exactly what it means my guess is that with imagepassthrough enabled, there is a higher demand for the buffer.

After disabling image passthrough and using h264 encoding, the warning went away. Problem solved for me. Hope this helps someone.

[gerazo]

@hspaay Reencoding a h264 bitstream probably requires a lot of CPU resources. H264 passthrough is supposed to use the incoming bitstream as is, and save it directly to disk without using much CPU. In typical mpeg4 streams, you can only save a full GOP (group of pictures) which is a keyframe (a full image) and some predicated images (differentials based on the previous frame). If you reencode, the reencoding process can decide whenever needed to encode and save a keyframe, so a cut in the video can be done anywhere. But with passthrough, you are tied to cut at place where you had a keyframe in the original stream, which was not up to your system, but the sender IP cam.
At least this is how mpeg4 streams work. I do not know the internals of the reencoder used in ZM, but I guess, it should work similarly. And this explains why you could fix this.

[hspaay]

@gerazo Thank you for the clarification and it makes sense.
So if I understand correctly, the ImageBuffer must be able to hold at least a keyframe + intermediate frames. Therefore if a camera's H264 stream IFrame interval is 5. Does that mean the buffer must be 6 or maybe 12 (two GOP's)?

In my case, the ImageBufferCount is set to 30 and the camera IFrame interval is set to 5. The frame rate is 4 FPS.
Based on the above it sounds like the buffer is large enough so would that mean that the analysis process can't empty it fast enough?

[gerazo]

@hspaay You are right. Having a buffer of 5 in case of a 5 GOP size is adequate even in the worst case scenario: a new record is to be done starting with the current frame and even if this is the last p-frame, we still have the corresponding i-frame in the buffer. So the new recording will not miss the current frame (yes, it has a couple of extra frames). So this is a reason for a minimal buffer size.
But no matter how large your buffer is, if your CPU cannot keep up with the filling rate, you can still have problems. Large buffers can survive fluctuations but not constant high load which your hardware cannot keep up with.

[hspaay]

@gerazo, yes, I understand the CPU constraint, so it could be a factor at play here too. I'm quite happy with zoneminder's performance actually. Zoneminder 1.36.12 is running in a proxmox container with 2 XEON E3-1220 @3.10GHZ CPU's assigned and 10GB of RAM. Configuration is 8 cameras which are set to 4FPS. Proxmox shows a system CPU load of 30-40%, not exceeding 50%, and 4-6GB memory consumed.

A couple of additional observations from experimentation:

1. The errors depend on the camera. Two cheap Chinese cameras always generate the error, regardless of their IFrame settings but a lower frame rate does make a difference. Makes me wonder if they use the I-Frame setting.
   The Hikvisions work as expected.

2. Comments in the ffmpeg encoding output configuration can cause problems. In my case the logs show an error that ffmpeg does not recognize the command '# comment'. The result was that the alarm video started fuzzy and only lasted for the duration of the motion. Removing the comments altogether solved this.