Blue Iris and CodeProject.AI ALPR

[TSAN]

You will need to manually install older modules, what version do you want, I will see if I can hunt it down.

I see. Perhaps something that could be considered in future releases to have archive repos build into CP web UI for old modules. I'm sure much easier said than done though.

Well I was going to explore the ALPR 2.9.1 to see if any different on A2000? If its a bunch of work for ya don't bother. Probably same issue.
I did try CP 2.1.8 , forget which ALPR it defaulted to. 2.5.0-RC1? It wouldn't run cuda but didn't crash on cpu either.

Also now tried CP 2.5.6 + ALPR 3.0.3 on cuda 12.3 , cudnn 9. Same crash on common.dll windows event log.

Played around all last night - About to throw in the towel for now. I did post over at CP forums as you suggested.
Its bit of bummer as I have a production environment with BI server on GTX 1650 4GB for Ui3 stream server + external server CP 2.5.6 on RTX 3050 with Cuda 12.2. Its running Yolo 5.6.2 (1.9.1) + ALPR 3.02 CUDA just fine. This A2000 was to get installed into Poweredge Esxi VM to consolidate both BI + CP on same system and free up the RTX 3050 RAM on users workstation.

 

[MikeLud1]

I see. Perhaps something that could be considered in future releases to have archive repos build into CP web UI for old modules. I'm sure much easier said than done though.

Well I was going to explore the ALPR 2.9.1 to see if any different on A2000? If its a bunch of work for ya don't bother. Probably same issue.
I did try CP 2.1.8 , forget which ALPR it defaulted to. 2.5.0-RC1? It wouldn't run cuda but didn't crash on cpu either.

Also now tried CP 2.5.6 + ALPR 3.0.3 on cuda 12.3 , cudnn 9. Same crash on common.dll windows event log.

Played around all last night - About to throw in the towel for now. I did post over at CP forums as you suggested.
Its bit of bummer as I have a production environment with BI server on GTX 1650 4GB for Ui3 stream server + external server CP 2.5.6 on RTX 3050 with Cuda 12.2. Its running Yolo 5.6.2 (1.9.1) + ALPR 3.02 CUDA just fine. This A2000 was to get installed into Poweredge Esxi VM to consolidate both BI + CP on same system and free up the RTX 3050 RAM on users workstation.

The changes to v3.0.3 from v3.0.2 were minor and if v3.0.2 worked v3.0.3 should work.

 

[TSAN]

The changes to v3.0.3 form v3.0.2 were minor and if v3.0.2 worked v3.0.3 should work.

No, misunderstanding - I have ALPR 3.0.2 working on an entirely separate CP deployment with different RTX 3050 8GB. This A2000 I'm testing on a bench is to replace it.

I cannot get ALPR 3.0.3 / A2000 to not crash regardless of trying 3 different CUDA versions 11.8 , 12.2 , 12.3.
And on all 3 different CUDA versions face + Yolo 5.6.2 does work with cuda on all of them. Something specific with ALPR / Paddle. Or something specific to this very old i7 920 bench machine I'm trying to test on. ALPR require specific CPU instruction sets this old CPU may not have?

 

[MikeLud1]

No, misunderstanding - I have ALPR 3.0.2 working on an entirely separate CP deployment with different RTX 3050 8GB. This A2000 I'm testing on a bench is to replace it.

I cannot get ALPR 3.0.3 / A2000 to not crash regardless of trying 3 different CUDA versions 11.8 , 12.2 , 12.3.
And on all 3 different CUDA versions face + Yolo 5.6.2 does work with cuda on all of them. Something specific with ALPR / Paddle. Or something specific to this very old i7 920 bench machine I'm trying to test on. ALPR require specific CPU instruction sets this old CPU may not have?

Have you tried replacing the RTX 3050 with the A2000, I think it uses all the same drivers and should work without any install needed.

 

[TSAN]

Have you tried replacing the RTX 3050 with the A2000, it uses all the same drivers and should work without any install needed.

No. Sorry I tried to be short explaining goal of this new A2000 but perhaps I was confusing.

The A2000 is actually going into a Poweredge Vmware server which presently has a GTX 1650 installed in it. This GTX 1650 is only doing BI stuff.
Technically the A2000 is going to replace this GTX 1650 card that is not doing any CP cause it doesn't have enough RAM.
Because A2000 has 12GB RAM it will now be capable of also replacing the separate CP workstation that is using RTX 3050 system. So my overall goal is getting BI + CP is consolidated into a single Server 2022 VM / A2000.

I wanted to do this lab testing on A2000 as I have to shutdown around 6 virtual servers / ESXi host afterhours and re-configure ESXi passthrough for new card.
I just wanted to see all aspects of CP working before committing to installing it into server. I'll likely be installing A2000 into Poweredge server anyway next week to minimally handle BI H264 stream encoding but can then also try CP / ALPR on that BI virtual machine.
If A2000 doesn't work on that virtual system, I will still be able to use the external CP server / RTX 3050.

I don't want to consume more of your time. Well see if CP devs have any suggestions or if A2000 works when installed in Poweredge server / VM.

 

[Steve_Texas]

No. Sorry I tried to be short explaining goal of this new A2000 but perhaps I was confusing.

The A2000 is actually going into a Poweredge Vmware server which presently has a GTX 1650 installed in it. This GTX 1650 is only doing BI stuff.
Technically the A2000 is going to replace this GTX 1650 card that is not doing any CP cause it doesn't have enough RAM.
Because A2000 has 12GB RAM it will now be capable of also replacing the separate CP workstation that is using RTX 3050 system. So my overall goal is getting BI + CP is consolidated into a single Server 2022 VM / A2000.

I wanted to do this lab testing on A2000 as I have to shutdown around 6 virtual servers / ESXi host afterhours and re-configure ESXi passthrough for new card.
I just wanted to see all aspects of CP working before committing to installing it into server. I'll likely be installing A2000 into Poweredge server anyway next week to minimally handle BI H264 stream encoding but can then also try CP / ALPR on that BI virtual machine.
If A2000 doesn't work on that virtual system, I will still be able to use the external CP server / RTX 3050.

I don't want to consume more of your time. Well see if CP devs have any suggestions or if A2000 works when installed in Poweredge server / VM.

If it makes you feel any better, I am having a similar issue on a fresh Windows install, also running on an older i7 and with an older 2060 NVIDIA card. My symptoms are the same as yours so I am very interested in what you learn here and hope you will post a follow up.

Cheers

 

[TSAN]

If it makes you feel any better, I am having a similar issue on a fresh Windows install, also running on an older i7 and with an older 2060 NVIDIA card. My symptoms are the same as yours so I am very interested in what you learn here and hope you will post a follow up.

Cheers

Thanks for sharing.
So the same ALPR "failed to start" + Win App event log has this error?

App Error - 1000

Faulting application name: python.exe, version: 3.9.6150.1013, time stamp: 0x60d9eb23
Faulting module name: common.dll, version: 0.0.0.0, time stamp: 0x6585a281
Exception code: 0xc000001d
Fault offset: 0x000000000000645a
Faulting process id: 0x1e34
Faulting application start time: 0x01da82476b8f6aaf
Faulting application path: C:\Program Files\CodeProject\AI\runtimes\bin\windows\python39\python.exe
Faulting module path: C:\Program Files\CodeProject\AI\modules\ALPR\bin\windows\python39\venv\lib\site-packages\paddle\libs\common.dll
Report Id: 1496e1bc-5b22-485c-82a4-bee2d47e2a3f
Faulting package full name:
Faulting package-relative application ID:

 

[Steve_Texas]

Thanks for sharing.
So the same ALPR "failed to start" + Win App event log has this error?

App Error - 1000

Faulting application name: python.exe, version: 3.9.6150.1013, time stamp: 0x60d9eb23
Faulting module name: common.dll, version: 0.0.0.0, time stamp: 0x6585a281
Exception code: 0xc000001d
Fault offset: 0x000000000000645a
Faulting process id: 0x1e34
Faulting application start time: 0x01da82476b8f6aaf
Faulting application path: C:\Program Files\CodeProject\AI\runtimes\bin\windows\python39\python.exe
Faulting module path: C:\Program Files\CodeProject\AI\modules\ALPR\bin\windows\python39\venv\lib\site-packages\paddle\libs\common.dll
Report Id: 1496e1bc-5b22-485c-82a4-bee2d47e2a3f
Faulting package full name:
Faulting package-relative application ID:

I see both EventID 1000 and EventID 1005

1005 is:

Windows cannot access the file for one of the following reasons: there is a problem with the network connection, the disk that the file is stored on, or the storage drivers installed on this computer; or the disk is missing. Windows closed the program Python because of this error.

Program: Python
File:

The error value is listed in the Additional Data section.
User Action
1. Open the file again. This situation might be a temporary problem that corrects itself when the program runs again.
2. If the file still cannot be accessed and
- It is on the network, your network administrator should verify that there is not a problem with the network and that the server can be contacted.
- It is on a removable disk, for example, a floppy disk or CD-ROM, verify that the disk is fully inserted into the computer.
3. Check and repair the file system by running CHKDSK. To run CHKDSK, click Start, click Run, type CMD, and then click OK. At the command prompt, type CHKDSK /F, and then press ENTER.
4. If the problem persists, restore the file from a backup copy.
5. Determine whether other files on the same disk can be opened. If not, the disk might be damaged. If it is a hard disk, contact your administrator or computer hardware vendor for further assistance.

Additional Data
Error value: 00000000
Disk type: 0

1000 is:

Faulting application name: python.exe, version: 3.9.6150.1013, time stamp: 0x60d9eb23
Faulting module name: common.dll, version: 0.0.0.0, time stamp: 0x6585a281
Exception code: 0xc000001d
Fault offset: 0x000000000000645a
Faulting process id: 0x3f8
Faulting application start time: 0x01da84586005618a
Faulting application path: C:\Program Files\CodeProject\AI\runtimes\bin\windows\python39\python.exe
Faulting module path: C:\Program Files\CodeProject\AI\modules\ALPR\bin\windows\python39\venv\lib\site-packages\paddle\libs\common.dll
Report Id: f16ed47e-bba9-408b-badb-7b7ca76cb755
Faulting package full name:
Faulting package-relative application ID:

I am going to uninstall CPAI and Python completely and start from scratch to see if that resolves my issue.

 

[TSAN]

I see both EventID 1000 and EventID 1005

Thanks for confirming. Same same.
Good luck. I've tried fresh CP install many times on CUDA 11.8 , 12.2, 12.3.

Well that shows issue crossing Nvidia Turing + Ampere cards. Yet I have another Ampere 3050 running on Ryzen 5900x running ALPR CUDA just fine.

What exact old CPU you using? My test bench PC with issue I'm using i7 920 - its like ~13 yrs old CPU. It was first gen i7. Maybe this issue is CPU related...

 

[Steve_Texas]

Same CPU - Intel(R) Core(TM) i7 CPU 920 @ 2.67GHz

However, this same PC worked with an older NVIDIA 960 card. It was not until I installed the 2060 into the machine that I started having this issue. I am about halfway through the reinstall of CPAI. Based upon your experiences, I don't have a lot of hope that this will resolve my issue, but we will see.

 

[Steve_Texas]

No joy so far...

I did notice that the LPR version is now 3.1.0

It started like this:



And ends like this about 10 minutes later:


EventIDs 1000,1005, and 1001 present in the Window Event/Application logs.

here is what 1001 looks like for this:

 

[TSAN]

No joy so far...

I did notice that the LPR version is now 3.1.0

:-(
Ya. Well there is a pattern here on this i7 920.
I'm likely going to be installing this A2000 into its permanent server this coming weekend. Poweredge r730xd using dual Xeon E5-2680v3. ESXi7 , Server 2022 VM.
My hunch is ALPR will probably work with these Xeons and this python / paddle / common.dll crash has something to do with missing CPU instructions sets on legacy i7 920. A good question for paddle developers.
The other possibility is this legacy i7 platform is using legacy PCIE 2.0. This A2000 card is going from PCIe 4.0 to 2.0 , your 2060 likely going from PCIe 3.0 to 2.0

Edit:
The other troubleshooting I tried was just getting it to run on CPU alone. Despite setting GPU to false in ALPR json config files it still appears to try to use it. (CP web ui info shows enabled) Perhaps that info doesn't update unless it successfully starts.
You could try removing the 2060 / CUDA entirely and refresh install of CP / ALPR. See if it can start on CPU alone. If it can't start with GPU removed from equation its gotta be i7 920...

 

[Steve_Texas]

:-(
Ya. Well there is a pattern here on this i7 920.
I'm likely going to be installing this A2000 into its permanent server this coming weekend. Poweredge r730xd using dual Xeon E5-2680v3. ESXi7 , Server 2022 VM.
My hunch is ALPR will probably work with these Xeons and this python / paddle / common.dll crash has something to do with missing CPU instructions sets on legacy i7 920. A good question for paddle developers.
The other possibility is this legacy i7 platform is using legacy PCIE 2.0. This A2000 card is going from PCIe 4.0 to 2.0 , your 2060 likely going from PCIe 3.0 to 2.0

Edit:
The other troubleshooting I tried was just getting it to run on CPU alone. Despite setting GPU to false in ALPR json config files it still appears to try to use it. (CP web ui info shows enabled) Perhaps that info doesn't update unless it successfully starts.
You could try removing the 2060 / CUDA entirely and refresh install of CP / ALPR. See if it can start on CPU alone. If it can't start with GPU removed from equation its gotta be i7 920...

I think you are right that this is CPU related. I attempted to analyze the .DMP file created by this issue and there is a line about "illegal instruction" related to common.dll. When searching for this, I find similar errors in StackOverflow about issues with the CPU.

Interestingly, my lab server is a PowerEdge R710 that runs BlueIris and a few VMs for me. Were it not a crazy hack job to put a GPU in it, I would try that way. I am still baffled as to why this ran with an older GPU under (essentially) the same configuration. I saw you posted on the CodeProject AI forms and I am hopeful for a good response there.


FAILED_INSTRUCTION_ADDRESS:
common!std::ctype<char>::_Getcat+ea
00007ffc`1b34645a c5fc1000 vmovups ymm0,ymmword ptr [rax]

SYMBOL_NAME: common+ea

MODULE_NAME: common

IMAGE_NAME: common.dll

FAILURE_BUCKET_ID: ILLEGAL_INSTRUCTION_c000001d_common.dll!Unknown

OS_VERSION: 10.0.19041.1

BUILDLAB_STR: vb_release

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

 

[TSAN]

I think you are right that this is CPU related. I attempted to analyze the .DMP file created by this issue and there is a line about "illegal instruction" related to common.dll. When searching for this, I find similar errors in StackOverflow about issues with the CPU.

Interestingly, my lab server is a PowerEdge R710 that runs BlueIris and a few VMs for me. Were it not a crazy hack job to put a GPU in it, I would try that way. I am still baffled as to why this ran with an older GPU under (essentially) the same configuration. I saw you posted on the CodeProject AI forms and I am hopeful for a good response there.

Ya quite familiar with the R710 GPU situation. If I remember correct even the PCIe risers will only give you 25W rather than standard 75W. So can't even use low power cards like 1650 or A2000.
The Xeon 56xx in the R710 is actually same gen CPU as the i7 920 though. So very well may experience same CPU issue on R710 too.

Ya, I'll update that CP forum post to put more attention on the i7 920 specifically.

 

[Steve_Texas]

I was not aware of that. I was able to run ALPR on the R710 CPU with no problem, other than it took too long to analyze images, especially with the CPU running at ~50% pulling the other VM loads. I (perhaps naively) think this issue can be resolved and I remain optimistic.

 

[Skinny1]

I have the new License Plate Reader module 3.1.0 that was released today installed and appears to be working.

 

[wittaj]

So we went straight from 3.0.3 to 3.1?

Mine says it is for server 2.6, but that isn't available yet?

 

[Skinny1]

So we went straight from 3.0.3 to 3.1?

Mine says it is for server 2.6, but that isn't available yet?

View attachment 191207

LPR 3.1.0 working on my Codeproject Ai version 2.5.6.

 

[war4peace]

LPR 3.2.0 working on my Codeproject Ai version 2.5.6.

3.2.0 or 3.0.2?

 

[Skinny1]

3.2.0 or 3.0.2?

My mistake, it is LPR 3.1.0.