HTTP Snapshots Download Times Slow [Advanced Question]

[BGChicago]

Hey Guys,
I have a server running curl or wget requests to take/save snapshots from the NVR, triggered by alarms that the NVR sends or manually. This is essesntially like loading the ipaddres/cgi-bin/snapshot.cgi in a web browser.The reason for this is that the built in FTP upload or Email was sometimes (often actually) fail to either save the file or add to email attachment.

Now that I am using curl to download, I can see why. It normally takes about 2-3 for the image to download, <0.1 of which is actual protocol talk and login and the file transmission (jpeg download) is only 0.2s. BUT sometimes, the whole operation would take 3 to 12 seconds (and this is probably why emails are missing attachments from time to time), of which the file transmission is again only 0.2s. So I am guessing the camera is taking long time to produce that snapshot.cgi (which is jpeg).

I would like to bring these numbers down to as low as possible as often the snapshots would miss a moving object. Does anyone know how I can speed this up? I already tried changing the snapshot quality level (1-6) and that didn't seem to have much of effect - I was still getting slow responses in the 5+ seconds with the lowest quality. Files were obviously slightly larger and so was the file transmit (0.3s) with the higher quality. I also tried talking directly to the cameras instead of the NVR - it was actually slower. Any ideas?

 

[bp2008]

Go to the camera's web interface, Setting tab > Camera > Video > Snapshot. Set the Interval to 1S. Then change Snapshot Type to Event and set the Interval to 1S there too. See if that helps.

 

[BGChicago]

Go to the camera's web interface, Setting tab > Camera > Video > Snapshot. Set the Interval to 1S. Then change Snapshot Type to Event and set the Interval to 1S there too. See if that helps.

Good suggestion but it does not help.

 

[BGChicago]

Go to the camera's web interface, Setting tab > Camera > Video > Snapshot. Set the Interval to 1S. Then change Snapshot Type to Event and set the Interval to 1S there too. See if that helps.

I actually thing it performs better with the interval set as high as possible. When set to 1 sec (for event) and quality is "6=Best", it returns 500 Server Error after 15 seconds. The other interesting bit here is that I have one camera that completes everything in 0.3 seconds most of the time, but it still goes to 7+ sec sometimes. None of the other ones go under 1.7sec.

EDIT: The above is only true when going thru the NVR. I tested with a camera that I have direct connection to and setting it to 1 sec indeed speeds up the process. When going thru the NVR, 2 seconds yields better results overall. I am guessing the NVR is the bottleneck.

 

[bp2008]

Hard to say. Jpeg frames are super easy to transfer, so I am surprised they don't just have it write into memory and respond instantly with the latest image available.

 

[BGChicago]

Hard to say. Jpeg frames are super easy to transfer, so I am surprised they don't just have it write into memory and respond instantly with the latest image available.

Well, sometimes they do -I am getting them in 0.3 seconds from time to time. I just noticed that reducing the frame rate of the camera stream have impact. Still have occasional 4-5 seconds but it is better then where I started. I wonder if there is a way to monitor the NVR CPU via SSH.

 

[bp2008]

Well I have a PTZ that definitely gets sluggish to respond to PTZ commands if the frame rate and bit rate are set very high. And another PTZ (different model) where it has a faster chip onboard I guess because it is always snappy. It could be a simple matter of overloaded CPU.

 

[BGChicago]

Well I have a PTZ that definitely gets sluggish to respond to PTZ commands if the frame rate and bit rate are set very high. And another PTZ (different model) where it has a faster chip onboard I guess because it is always snappy. It could be a simple matter of overloaded CPU.

I just discovered that I can get the snapshot direct from the camera by going to

http://192.168.1.108:10080/cgi-bin/snapshot.cgi

, where 10080 is the camera port, instead of

http://192.168.1.108/cgi-bin/snapshot.cgi?channel=X

where X is channel number. That only works in a browser though. Both curl and wget fail with "connection reset by peer". By chance you'd know why?

 

[Molbo]

I just discovered that I can get the snapshot direct from the camera by going to

http://192.168.1.108:10080/cgi-bin/snapshot.cgi

, where 10080 is the camera port, instead of

http://192.168.1.108/cgi-bin/snapshot.cgi?channel=X

where X is channel number. That only works in a browser though. Both curl and wget fail with "connection reset by peer". By chance you'd know why?

Dahua cameras are using digest authentication, are you providing user name/password in CLI?
Also, it looks like curl needs a special --digest option to use digest auth.

 

[BGChicago]

Dahua cameras are using digest authentication, are you providing user name/password in CLI?
Also, it looks like curl needs a special --digest option to use digest auth.

Yes to both. Auth works when going thru

http://192.168.1.108/cgi-bin/snapshot.cgi?channel=X

 

[bp2008]

No idea why it would work in a browser but not in curl or wget.

 

[hansch]

I just discovered that I can get the snapshot direct from the camera by going to

http://192.168.1.108:10080/cgi-bin/snapshot.cgi

, where 10080 is the camera port, instead of

http://192.168.1.108/cgi-bin/snapshot.cgi?channel=X

where X is channel number. That only works in a browser though. Both curl and wget fail with "connection reset by peer". By chance you'd know why?

Did you manage to figure out why wget/curl doesn't work with the camera port?

I keep getting "Read error (Connection reset by peer) in headers."

I'm also trying to find a way to store periodic snapshots.

 

[BGChicago]

Did you manage to figure out why wget/curl doesn't work with the camera port?

I keep getting "Read error (Connection reset by peer) in headers."

I'm also trying to find a way to store periodic snapshots.

I did not. After lowering down frame rates of my cameras (to 10fps) the delay is under 3 seconds most of the time and often under 1 sec, so this became lower priority. Let me know if you need help with curl or wget as I have that working (straight from the NVR or camera direct).

One more thing I wish I could find a way to do is actually request snapshot with lower resolution. I am currently resizing them on the server (need them for a something). But if it is possible to get lower res straight from the NVR/camera, it may further speed up the snapshot capture.

 

[hansch]

I did not. After lowering down frame rates of my cameras (to 10fps) the delay is under 3 seconds most of the time and often under 1 sec, so this became lower priority. Let me know if you need help with curl or wget as I have that working (straight from the NVR or camera direct).

One more thing I wish I could find a way to do is actually request snapshot with lower resolution. I am currently resizing them on the server (need them for a something). But if it is possible to get lower res straight from the NVR/camera, it may further speed up the snapshot capture.

I'd love to know the steps you take with curl and wget and your settings.
I'm getting a lot of internal server errors when using wget and the results are not consistent. getting a lot of zero byte images.

The main thing I'm trying to achieve now is to get construction snapshots so I can join them into a timelapse video at a later stage.

 

[BGChicago]

I'd love to know the steps you take with curl and wget and your settings.
I'm getting a lot of internal server errors when using wget and the results are not consistent. getting a lot of zero byte images.

The main thing I'm trying to achieve now is to get construction snapshots so I can join them into a timelapse video at a later stage.

curl -w '%{http_code}\\n' http://192.168.1.108/cgi-bin/snapshot.cgi?channel=1 --digest -u username:password -o /path/file_name.jpeg
-w '%{http_code}\\n' - this will return http status code, 200 if succesful

wget --http-user=username --http-password=password http://192.168.1.108/cgi-bin/snapshot.cgi?channel=2 -o /path/file_name.jpeg

 

[hansch]

Thanks! I managed to script something up and added it to my Synology NAS as a cron job.
Just a simple for loop to loop through the channel numbers and name output files with channel no. and date/time.

I ended up using your wget tips with an extra parameter - "--retry-on-http-error=500". I noticed that the zero byte snapshot issues are now gone.

Did you end up changing your camera snapshot interval (in encode section) to something other than 1s?

 

[BGChicago]

Thanks! I managed to script something up and added it to my Synology NAS as a cron job.
Just a simple for loop to loop through the channel numbers and name output files with channel no. and date/time.

I ended up using your wget tips with an extra parameter - "--retry-on-http-error=500". I noticed that the zero byte snapshot issues are now gone.

Did you end up changing your camera snapshot interval (in encode section) to something other than 1s?

1 sec was bad for me thru the NVR but great for directcameras. See comments above. I think I left it at 3.

Probably what this setting does it actually buffers the snapshot every so many seconds, so when needed, it is ready. But my NVR most likely can't handle 1 second interval (individualcamera with 3 streams runing can). That's what I got of it.

Please let me know if you find better options, or ways to access camera snapshots from the camera thru the NVR via wget/curl.

 

[brianegge]

I have a script which is downloading snapshots from my eight cameras through the NVR. If I make all the http requests concurrently, most of the requests fail. Going in serial works, but if the scene is too complex or has a lot of motion, I notice I get the 500 error. It takes about 24 seconds to sweep all of the cameras. On my server which is requesting the snapshots, I don't have enough power to subscribe and decode 8 4k streams. I can do two streams, and have ffmpeg write a jpeg every second, but not more without a more powerful computer (I can do the substreams without issue).

Ideally, I could snap all the cameras every 5 seconds.

The NVR has two interfaces, the cgi interface which is fairly well documented and the XML RPC2, which is not. When you click on the camera in the NVR web interface, it makes an RPC2 call:

{"method":"SwitchPoE.getTransPort","params":{"Channel":0,"IPAddress":"10.1.1.140"},"session":"xxxxxxxxxxx","id":14,"params2":""}:

The response is the http port, which is now open, for this session:

{"id":14,"params":{"Channel":0,"ListenPort":10080},"result":true,"session":"xxxxxxxxxxxxx"}

You get the 10080 for the first camera you open, 10081 for the second, and so on. But these ports are only for the session and don't map to the camera. The camera they map to is specified in the RPC2 request. I haven't found any documentation on the RPC2 API, but it's what both the web app and iOS app use. It should be easier to reverse engineer since you can observe all the requests via a browser or wireshark. However, I haven't got so far as logging in from a script through the RPC2 interface.

I'd prefer to keep my cameras plugged into the NVR POE ports but I may move them to a POE switch on my network, which would allow direct access to the cgi interface from cURL.

 

[BGChicago]

I have a script which is downloading snapshots from my eight cameras through the NVR. If I make all the http requests concurrently, most of the requests fail. Going in serial works, but if the scene is too complex or has a lot of motion, I notice I get the 500 error. It takes about 24 seconds to sweep all of the cameras. On my server which is requesting the snapshots, I don't have enough power to subscribe and decode 8 4k streams. I can do two streams, and have ffmpeg write a jpeg every second, but not more without a more powerful computer (I can do the substreams without issue).

Ideally, I could snap all the cameras every 5 seconds.

The NVR has two interfaces, the cgi interface which is fairly well documented and the XML RPC2, which is not. When you click on the camera in the NVR web interface, it makes an RPC2 call:

{"method":"SwitchPoE.getTransPort","params":{"Channel":0,"IPAddress":"10.1.1.140"},"session":"xxxxxxxxxxx","id":14,"params2":""}:

The response is the http port, which is now open, for this session:

{"id":14,"params":{"Channel":0,"ListenPort":10080},"result":true,"session":"xxxxxxxxxxxxx"}

You get the 10080 for the first camera you open, 10081 for the second, and so on. But these ports are only for the session and don't map to the camera. The camera they map to is specified in the RPC2 request. I haven't found any documentation on the RPC2 API, but it's what both the web app and iOS app use. It should be easier to reverse engineer since you can observe all the requests via a browser or wireshark. However, I haven't got so far as logging in from a script through the RPC2 interface.

I'd prefer to keep my cameras plugged into the NVR POE ports but I may move them to a POE switch on my network, which would allow direct access to the cgi interface from cURL.

Very interesting. That explains why I can't curl directly to the camera via 10080. I also want to keep the cameras on the PoE ports but I wish there was an option to have them on same submet as NVR WAN port (e.g. disable NAT or whatever it is running on the NVR).
May be we can join forces and reverse engineer it together?

I have a script which is downloading snapshots from my eight cameras through the NVR. If I make all the http requests concurrently, most of the requests fail. Going in serial works, but if the scene is too complex or has a lot of motion, I notice I get the 500 error. It takes about 24 seconds to sweep all of the cameras. On my server which is requesting the snapshots, I don't have enough power to subscribe and decode 8 4k streams. I can do two streams, and have ffmpeg write a jpeg every second, but not more without a more powerful computer (I can do the substreams without issue).

Ideally, I could snap all the cameras every 5 seconds.

The NVR has two interfaces, the cgi interface which is fairly well documented and the XML RPC2, which is not. When you click on the camera in the NVR web interface, it makes an RPC2 call:

{"method":"SwitchPoE.getTransPort","params":{"Channel":0,"IPAddress":"10.1.1.140"},"session":"xxxxxxxxxxx","id":14,"params2":""}:

The response is the http port, which is now open, for this session:

{"id":14,"params":{"Channel":0,"ListenPort":10080},"result":true,"session":"xxxxxxxxxxxxx"}

You get the 10080 for the first camera you open, 10081 for the second, and so on. But these ports are only for the session and don't map to the camera. The camera they map to is specified in the RPC2 request. I haven't found any documentation on the RPC2 API, but it's what both the web app and iOS app use. It should be easier to reverse engineer since you can observe all the requests via a browser or wireshark. However, I haven't got so far as logging in from a script through the RPC2 interface.

I'd prefer to keep my cameras plugged into the NVR POE ports but I may move them to a POE switch on my network, which would allow direct access to the cgi interface from cURL.

See attached API document. Is that what is needed for the RPC2? Also, what documentation do you have for the CGI protocol?

 

[brianegge]

See attached API document. Is that what is needed for the RPC2?

That's what's needed for the cgi protocol. For RPC, you can view the http requests in Chrome. Some people have started work reverse engineering the RPC protocol: