Error 502 with NGINX reverse proxy and BI > 5.3.1

[Capt.Insano]

Just registered to report the same issue.

My setup:
BlueIris 5.3.3.3 behind an Nginx reverse proxy at address: https://cctv.domain.com
I use OAuth2 Proxy to control access on all other subdomains other than https://cctv.domain.com


If start with a clean browser, I can visit https://cctv.domain.com successfully.
But if I visit and login on any of my other subdomains https://example.domain.com or even my main domain Sign In and then visit https://cctv.domain.com I get: "502 Bad Gateway" error from Nginx.

In order for it to work I must clear cookies again.

@geoffmyers Did you hear anything back from BlueIris support?

 

[DarkIllusion]

I can now confirm that the cause of this particular issue is Blue Iris. I have performed comprehensive testing using multiple reverse proxies (both NGINX and Traefik) with a variety of configurations and settings, and I have tested them with multiple versions of Blue Iris to see which combinations work and which ones don't. The last version of Blue Iris that does not have this issue is 5.3.0.3. All later versions (5.3.1 and newer) experience the same issue (constant HTTP 502 errors when using a reverse proxy).

I have contacted Blue Iris support and am waiting for their reply.

Thanks! I set up the 15 day demo yesterday for a family member to try out and this issue was giving me a headache. I wasn't sure if it was because I was using
Cloudflare Argo Tunnel to set up access instead of the usual set up. It seems I only had the issue on chrome based browsers.
I downgraded from the latest version down to 5.2.XX (somewhere around there) which fixed the issue. I was then able to choose 5.3.0.3 as an upgrade option afterwards.
It seems to be running smoothly now, I hope the issue is resolved soon.
Cheers!

 

[milosm]

Was anyone able to move past this issue somehow?

Has BlueIris support given any feedback?

 

[stavrosmadk]

Today i setup cloudflare, when i make my cname to be proxied i get 502.
If i set to DNS it works. (All my other services, sites, etc work correctly)
I hope it get fixed soon.

 

[fenderman]

Today i setup cloudflare, when i make my cname to be proxied i get 502.
If i set to DNS it works. (All my other services, sites, etc work correctly)
I hope it get fixed soon.

This is not a blue iris problem. But if you think it is, it wont get fixed unless you email support. Hoping wont fix it.

 

[stavrosmadk]

This is not a blue iris problem. But if you think it is, it wont get fixed unless you email support. Hoping wont fix it.

today i made some more testing, When i rolled back to 5.3.2.11 It worked, but it doens't work on 5.3.3.16.
I will do some more testing when i get time then email support

 

[VLITKOWSKI]

Any news on that issue ? still the same case for me

 

[dacrazyazn]

i created an account here to report the same issue

I am running BI 5.3.6.2 with cloudflare->traefik reverse proxy->BI

constant 502 errors on all my Chrome browser unless all the cookies are cleared, and no other subdomains are visited. Edge browser works fine..

 

[fenderman]

i created an account here to report the same issue

I am running BI 5.3.6.2 with cloudflare->traefik reverse proxy->BI

constant 502 errors on all my Chrome browser unless all the cookies are cleared, and no other subdomains are visited. Edge browser works fine..

You need to email support.

 

[SwampdogMash]

Hi

BI Version 5.3.6.2 (04/12/2020). IIS ARR Reverse proxy and Let's Encrypt cert. Cloudflare free account with DNS proxy enabled.

Been having the same issues myself. I've looked at all the headers coming from CF and whilst there are some that differ from the working request when I DON'T use CF, there doesn't seem to be anything that would explicitly make BI not work. Doing a packet capture on the BI server shows that the BI server itself is issuing a RST packet in response to the reverse proxied CF requests. This seems odd and likely means that BI doesn't like the length, presence or otherwise of SOMETHING in the headers. I mainly tested this by sticking with a basic request - like hitting my reverse proxy from my phone on wifi and requesting

https://internalserver/favicon.ico

then taking the phone off wifi and connecting via CF (with DNS proxy turned on) and hitting "

http://externalserver/favicon.ico

". TBF, I use split DNS so actually the URLs to get the favicon are actually identical, with the internal and external names being the same but pointing to different IPs.

I have tried some basic testing of things like the X-Forwarded-For header - (in case BI was now checking for a match to the internal IP ranges), or freaking out at CF adding in an IPv6 address in the XFF. This didn't help. I tried changing the x-forwarded-proto header to http instead of https but can't remove the header so cannot test a true copy of the working request (x-forwarded-proto is not present in the working request).

I tried adding in the additional Accept-Encodings of Deflate and BR which if it was going to fix it would have meant that the wireshark trace would show BI replying but CF not liking the response. that didn't work either.... Below is a screen shot of the differences between the headers on a working request (left) and a failing request. I have removed any lines that were the same on both requests.



You can also see here that the BI is indeed sending a RST immediately after the request is received. I've attached the capture from the server. I've mailed BI support to with this info - awaiting a response back.......



To add to this.... I've also tried different browsers on the phone to no avail. I've tried accessing the reverse proxy using a pc directly from my internal network and externally and i see the same behaviour on all borwsers. I've tried logically updating every option in BI to no avail also.

 

[SwampdogMash]

All,

Think i have found the issue!

It looks to be related to the overall size of the headers sent to BI. Based on the above comparison of the headers, I had a look at the size of the working versus the non working. I then started messing with various elements of the headers to see what I could find. It looks as though if I change several of the less valuable headers (CF tracking headers that CF happily still works if they are nonsense) and make them a single digit value, PLUS the user-agent, i can get CF to reply - this of course is nothing to do with the content of those headers.. but the SIZE of the headers.

I found this by using the reverse proxy to set the following to just a single character value each to just reduce the overall HTTP header/payload.

CF-IPCountry
CF-Ray
X-Forwarded-Proto
CF-Visitor
CF-Request-ID
CF-Connecting-IP
CDN-Loop
x-arr-log-id
user-agent

At this point things started working! The header size was reduced enough for whatever it is within BI to now respond. Further testing was required...

I removed my override for user-agent. This lengthy string was then being passed to BI again and it fails to respond with a RST. I re-added my override but this time put in the original string from the failing request as a control. Request fails.

Then I reduced the size of the string by 20 characters. Still not working. Rinse and repeat 20 characters reduction at a time until I got down to just the M of Mozilla at the start of the string. The CONTENTS of the string were clearly making no difference, but the length sure did.

With just an M in the user-agent field, I then added 1 character to one of the other strings. Instant failed requests. Then to ensure it wasn't some freak choice of another header, I removed the extra character. Working again. Then I swapped which header I had added characters to. Instant fails.

If I add another Header override and reduce the header size even further, (X-Forwarded-For for instance) - I can start adding characters back into the User-agent string - pretty definitely proving the contents are not the issue, but the length of "something".

Question here is... what is the limit and what is it on... It's not the NUMBER of headers, as they are all still present in this scenario. It looks to be a limit on the overall size of the headers OR.. the overall size of the TCP payload data. Interestingly.... the "Data" section of the HTTP request seems to be 1024 bytes . Exactly 1k...for a working request..... and 1025 bytes for a failing one.... so... looks like BI has a limit on the handling of the HTTP request somewhere....

The overall size of the headers minus the GET request itself specifically doesn't have any "round" value that sounds like any kind of limit (still could be, but it would be an arbitrary one) so I would say it is unlikely JUST related to headers. It is most likely a 1k limit on the overall request size going to BI.

This explains a lot of the issues noted above - Could be a change to Cloudflare has added extra headers. Could be that changes to browsers have added extra header length or increased the size of the User-agent string adding to the overall TCP data size. Either way, BI doesn't like it!

with the headers in the state they work, I then opened and connected the UI3 console via cloudflare on my phone. All requests are <1024 bytes and work. Time to play again. Pick a common request that also affects the functionality of BI and (luckily for our problem diagnosis) is larger than all other requests ever made to load the interface... It's a call to:

/image/Index?time= LONG STRING here a total of consistently 973 bytes - which is calling the images to put on the screen for the video feeds with different times and reference numbers for each feed.

The interface is loaded, the ONLY thing hitting BI now are the requests to update the video streams. Lets add (1025 - 973) bytes to the headers and see what happens..... Broken.

case closed from my perspective i think...

side note - Can't make it work now, because the cookie size has not got so big combined between BI and CF that I can't remove enough headers to make it work! Had to log in again to fix that.

 

[SwampdogMash]

This also explain weird issues i was having in generally accessing BI over a VPN when accessing certain pages.. where the camera feeds wouldn't start but I couldn't work out the rhyme or reason behind it. It is a function of cookie size and browser user-agent string size at this point...

 

[bp2008]

Hey @SwampdogMash I will admit I am not familiar enough with low level TCP implementation details to be sure of this, but it does sound like Blue Iris is trying to impose a small limit on the http request size, and is simply closing the connection if it doesn't like what it gets. Possibly BI is even closing the connection ungracefully (suggesting maybe it wasn't intentionally closing the connection, but instead an error was occurring?). I hope you are forwarding your latest findings on to the Blue Iris support email!

 

[SwampdogMash]

I will add more to the ticket.. once i get a reply to my support request. Am guessing it's busy..... I linked this thread in the email though...

 

[johndball]

Yo, this is EXACTLY what I'm seeing too. My thread is here but I'll start posting on yours to keep it inline. Let me/us know what support says. Heck, I'll even toss in some $$$ if you had to upgrade for the support request (DM me). UI3 error - "Unable to contact Blue Iris server". My issue is further complicated by the fact that I have a gateway inspection device adding it's own "secret sauce" the packet stream so I can't do anything to get it below the max header length. You can quickly see this happening if you are using Stunnel and watch the connection stream in the output window. You'll see a connection then Stunnel reports a RESET (RST) immediately thereafter.

Edit: so I downgraded each version until I was able to get one to work. I guess I had tunnel vision in my thread and was so focused on CF and certs I didn't think to step-downgrade. This is what I found:

1. 5.3.6.1 --> no go
2. 5.3.4.3 --> no go
3. 5.3.3.16 --> no go
4. 5.3.3.8 --> no go
5. 5.3.2.11 --> no go
6. 5.3.2.8 --> no go
7. 5.3.1.6 --> no go
8. 5.3.0.3 --> WORKS

BI web options selected:

1. Enable the HTTP web server on port 81
2. Use UI3 for non-IE browsers
3. Stunnel is installed for HTTPS on port 8443
4. Advanced - require auth from all connections
5. Advanced - limit logins (single digits)
6. Advanced - Use secure sessions keys and login page
7. Advanced - Release temp ban after n minutes
8. Advanced - Log ALL connections, not just logins
9. Advanced - HTTP keep-alives
10. Advanced - Use Deflate compression

Cloudflare options selected:

1. Page rules - Always Online - off
2. Cache Level - Bypass
3. SSL TLS (Full Strict)
4. Using a Cloudflare Origin cert
5. Using Cloudflare Authenticated Origin Pulls

Stunnel options configured:

[blue-iris]​

accept = 0.0.0.0:8443​

connect = 127.0.0.1:81​

CAfile = ca-certs.pem​

cert = cloudflare.pem​

key = cloudflare.key​

debug = 7​

checkhost = origin-pull.cloudflare.net​

verifychain = yes​

​

(Some options above renamed for public posting)

 

[SwampdogMash]

I have a reply from Ken...

Hello



Please retest this with 5.3.6.3 ... a change was recently made to allow longer HTTP requests over 1K.



Thanks



Ken

 

[johndball]

Hello
Please retest this with 5.3.6.3 ... a change was recently made to allow longer HTTP requests over 1K.
Thanks

No joy with 5.3.6.3.

Firefox 83.0
Chrome 87.0.4280.88

Additionally, the Stunnel output shows:
2020.12.10 10:16:42 LOG5[8]: Service [blue-iris] connected remote server from 127.0.0.1:57710
2020.12.10 10:16:42 LOG5[8]: Connection closed: 16395 byte(s) sent to TLS, 481 byte(s) sent to socket

Rolled back to 5.3.0.3. :-(

 

[SwampdogMash]

No joy with 5.3.6.3.

Firefox 83.0
Chrome 87.0.4280.88

Additionally, the Stunnel output shows:
2020.12.10 10:16:42 LOG5[8]: Service [blue-iris] connected remote server from 127.0.0.1:57710
2020.12.10 10:16:42 LOG5[8]: Connection closed: 16395 byte(s) sent to TLS, 481 byte(s) sent to socket

Rolled back to 5.3.0.3. :-(

Same same. Just done the testing all over again. Looks like BI now responds with a valid ACK, but does not follow it up with the HTTP response when the packet size is >1k. I've tested this both by removing my cludge rules that mangle the headers to make it <1k (CF stops working), and by ADDING extra header length to a known working NON-CF request coming from my own LAN via the same reverse proxy. Same behaviour of receiving a 502 from the proxy as it doesn't get a response to the request it sent to BI when the HTTP request size is >1024 bytes.

LAN client with same headers as CF, but reduced to <1024bytes (WORKING)


Lan client with same headers as CF, but expanded to deliberately >1024 bytes (FAILING and timing out)


Lan client with standard direct headers <1024 bytes (WORKING)


Lan client with standard direct headers but cF_RAY header added to blow it over 1024bytes (FAILING)

 

[VLITKOWSKI]

I find something weird with 5.3.6 ui3 clip page :
I own a domain with 4 hosts . 3 of them using cloudflare xx.domain.com yy.domain.com zz.domain.com
blueiris.domain.com not using cloudflare proxy working well except some issues (http 405 and crash) when I show clips with ui3.hm

I find that there's a cookie in my browser at the domain level containing __cfduid
I I remove this cooking at the domain level and blueiris working perfectly

I browse xx.domain.com => _cfuid cookies is generated at domain level, and BUI is having the same weird issue

Issue started with 5.3.6 - rollback to 5.3.5.1 and no more problem with clips page and _cfuid cookies

 

[SwampdogMash]

I find something weird with 5.3.6 ui3 clip page :
I own a domain with 4 hosts . 3 of them using cloudflare xx.domain.com yy.domain.com zz.domain.com
blueiris.domain.com not using cloudflare proxy working well except some issues (http 405 and crash) when I show clips with ui3.hm

I find that there's a cookie in my browser at the domain level containing __cfduid
I I remove this cooking at the domain level and blueiris working perfectly

I browse xx.domain.com => _cfuid cookies is generated at domain level, and BUI is having the same weird issue

Issue started with 5.3.6 - rollback to 5.3.5.1 and no more problem with clips page and _cfuid cookies

I haven't seen this similar behaviour, but it may be that the CF (cloud flare) UID cookie is actually just happening to take you over the 1024 byte limit. You would have to capture on the BI box to see what impact the cookie removal has to the request size (TCP Data) of the packets received. I've rolled back to 5.3.0.3 and all seems to be working again with no issues, but no I am behind in other features and updates.