hikvision - extract thousands of images from camera sd card?

[Alex_N]

Can you please post a screenshot of the original file/folder structure? (before moving the files to the datadir folder)
Don't worry about the script's message containing forward and backward slashes in the path, it's just a message, not the path the script uses internally.

 

[Pimvg]

Hi Alex,
Thanks for looking into this. Much appreciated!
Screenshots are attached. Please note that the number of .pic files runs all the way up (didn't include all screenshots of those).

 

[Alex_N]

The folder structure looks like the one produced by the older cameras except it doesn't use datadir folders, so you were right to place the files inside a datadir0 folder.

You said that it took 10 minutes to get the first reply, I think this was caused by those two folders (lost+found and syslog). The script makes a list of all the folders before it starts, so if those two folders contain lots of subfolders and you copied them inside the datadir0 folder you've created, that could be the cause of the delay. I would check to see what those two folders contain and when you run the script again do not copy them to the datadir0 folder.

In your screenshots it looks like all the .pic files (except the first one hiv00000.pic) have a disk size of 0KB. If that is true, they are empty and that's why the script doesn't output any images. This could indicate a problem with the memory card (corrupted files). If possible put a different memory card in the camera and let it run like this for one day then try to extract those images with the script. Copy the files to a datadir0 folder if necessary.

 

[Pimvg]

Thanks Alex,

in my first attempt, I put all the datafiles in the datadir0 folder and kept the other existing folders intact. That resulted in an error from the script, saying: Can't find event_db_index01 or index00p.bin or index01p.bin in D:\Hikv/datadir0/lost+found . So I deleted the other (empty) folders and then reran the script. The 10 min "delay" was still there and during those 10 mins one of the processor's cores is 100% active with the script. After that it starts displaying the messages about the pics.

As you correctly state, most .pic files are all 0KB but there is an .mp4 file. I didn't save any video (as far as I know) so I was thinking the camera was assembling all the images into a video-like file. The mp4 cannot be played in a standard video player so I don't know what it is.

I will now set up the camera with a different SD-card, format it and then assemble some data. Will get back once I have the results. I will also try to download a few of the images directly from the camera's web browser, just to make sure there are some images there.

 

[Pimvg]

Ok, so I formatted a new SD card and set up the camera to capture an image every minute. I was able to download the images directly from the camera as viewable jpg files.

I then took out the card from the camera and connected it to my Synology (since Windows does not recognize EXT4 formatted storage). The looks under Windows are identical to what is shown in the screenshots above, so 0KB pic files. Same when I view from the File Manager on the Synology. What is weird is that all files show the same date (today) and time (but approx two hours ahead !?!). Could this perhaps be an issue with translation of readability from the EXT4 format to the Synology and Windows environments?
If so, What would be a good alternative to copy the files from the SD-card to my Windows system?

 

[Pimvg]

Extra info.... Found and tried Ext2fsd on my Win10 laptop with the SD-card inserted. Could read the files with identical results: 0KB .pic files and a time stamp 2 hrs ahead of the actual time. I checked the time setting of the camera and that seems to be correct and in line with my Windows systems.

 

[Alex_N]

I usually use a live USB with Slax Linux, it can read and write to both ext4 and ntfs and it automatically mounts the partitions.

The .pic files size should be 0KB only if you just formatted the storage medium.

 

[Pimvg]

I formatted the memory card and then only ran the camera for 15 mins or so. Could it be that all captures are in the first file (xx0000.pic) and that the following files are just empty?
I could do another test and run the capturing for a couple of hours or a whole day, if you want.

 

[Alex_N]

Yes, it's possible, 15 minutes is not a lot and it depends on the chosen time interval.
You don't need to run the capturing for a whole day. Test on the files you have. You can exclude the .pic files that are empty when you copy the files to the datadir0 folder.

 

[Pimvg]

Did some testing. Running the script on the test-set with the 0KB .pic files deleted first. The only remaining .pic file is hiv00000.pic :

same 10 min ...processing datadir0...​

ended with an error saying: Can't find hiv00001.pic in the input folder.​

So my assumption is that the script knows which .pic files to be available​

Than ran the script on the full set of files, incl the 0KB files:

same 10 min ...processing datadir0...​

processing all .pic files one by one taking approx 1 sec per step​

ending with: All operations completed but an empty target folder​

Full details are in the attached file

 

[Alex_N]

Can you please upload your index00p.bin file?

In the meantime you can try using the python script method from my first video
When using the python script the input path must be that of the .pic file not of the folder that contains it.

 

[Pimvg]

Thanks! Attached is the requested file...
Will try to work the Python script tomorrow.

 

[Pimvg]

Gave it a quick try on the smaller test bundle and the Python script pulls the jpg's just fine! (24 captures)
Will try the same on a larger bundle tomorrow and will report the results.

Promising!!

 

[Alex_N]

I've looked at your index00p.bin file and found the problem. The structure of the file is different from every other index file I've encountered. The script couldn't find the correct header size (the header size is a number that tells the script where the information about the pictures is located inside the index00p.bin file), so it was searching the whole file for it, and since the file is quite big, that is why it was taking so long to get the first reply.

I've made some changes to the script, and I hope it will now work with your index00p.bin file.
Use the code below. The code only works with the structure of your index00p.bin file.

Spoiler: CODE

#!/usr/bin/perl -

use Date::Format;
use File::Find::Rule;
use DBI;
#use Data::Dumper qw(Dumper);

my ($mainInputDir, $outputDir) = @ARGV;

if (not defined $mainInputDir) {
  die "\nERROR: Need input directory.\nThe input directory should be the directory that contains the datadir folders.\nUsage: extract_capture.pl inputdir outputdir\n";
}

if (not defined $outputDir) {
  die "\nERROR: Need output directory.\nUsage: extract_capture.pl inputdir outputdir\n";
}

unless ((-e $mainInputDir."/datadir0/event_db_index01") or (-e $mainInputDir."/datadir0/index00p.bin") or (-e $mainInputDir."/datadir0/index01p.bin")) {
        die "\nERROR: Can't find event_db_index01 or index00p.bin or index01p.bin in $mainInputDir\\datadir0\n";
    }

print "\n************* EXTRACTION DATE AND TIME LIMITS *************\n";
print "\nDate and time format example YYYYMMDDHHMMSS: 20181225221508\n";
print "\nEnter START date and time (or enter 1 to start from the\nfirst available image or leave empty if you don't want to use\na date and time limit): ";
my $inputStartDate = <STDIN>;
chomp $inputStartDate;

$nowEndDate = time2str("%Y%m%d%H%M%S", time());

if ($inputStartDate != "") {
    print "\nEnter END date and time (or enter 1 to use current date and\ntime (this will extract until the last image), or leave empty\nif you don't want to use a date and time limit): ";
    our $inputEndDate = <STDIN>;
    chomp $inputEndDate;
    if ($inputEndDate == 1) {
        $inputEndDate = $nowEndDate;
    }
}

print "\n************* OUTPUT JPG FILES NAMING OPTIONS (DEFAULT 1) *************\n";
print "\nEnter 1 for Image_YYYY_MM_DD-HH_MM_SS-DayName.jpg\nEx: Image_2018_12_25-22_15_08-Tue.jpg\n";
print "\nEnter 2 for Image_YYYYMMDDHHMMSS-DayNumber.jpg\nEx: Image_20181225221508-2.jpg\n";
my $outputDateFormat = <STDIN>;
chomp $outputDateFormat;

print "\n************* WHEN MORE THAN ONE PICTURE IS TAKEN PER SECOND (DEFAULT 1) *************\n";
print "\nEnter 1 to auto rename and keep all pictures\n";
print "\nEnter 2 to only save the first picture\n";
my $timeDuplicates = <STDIN>;
chomp $timeDuplicates;

my @datadirArray = File::Find::Rule->directory->in($mainInputDir);
my $datadirNumber = @datadirArray;

for ($i=1; $i<$datadirNumber; $i++) {
    my $inputDir = $datadirArray[$i];
    print "Processing folder: $inputDir\n";

    if (-e $inputDir."/event_db_index01") {
        my $ary1;
        my $TBNumber;
        sleep(2);

        my $driver = "SQLite";
        my $database = $inputDir . "/event_db_index01";
        my $dsn = "DBI:$driver:dbname=$database";
        my $userid = "";
        my $password = "";
        my $dbh = DBI->connect($dsn, $userid, $password, { RaiseError => 1 }) or die $DBI::errstr;
     
        my $sth = $dbh->table_info('','main','%', 'TABLE');
        my $tables = [map{$_->[2]} @{$sth->fetchall_arrayref()}];
        #print join("\n", @$tables);
     
        if ( grep( /^event_id$/, @$tables ) ) {
            print "\n\n***** event_id found *****\n";
            my $stmt = qq(SELECT id, event_table_name from event_id;);
            my $sth = $dbh->prepare($stmt);
            my $rv = $sth->execute() or die $DBI::errstr;
                if($rv < 0) {
                    print $DBI::errstr;
                }
            $ary1 = $sth->fetchall_arrayref({});
            #print Dumper(\$ary1);    
        }
     
        else {
            $ary1 = [
                {
                  'id' => 10,
                  'event_table_name' => 'MotionDetectTB'
                },
                {
                  'id' => 12,
                  'event_table_name' => 'AlarmInputTB'
                },
                {
                  'id' => 13,
                  'event_table_name' => 'SceneChangeTB'
                },
                {
                  'id' => 24,
                  'event_table_name' => 'NormalPirTB'
                },
                {
                  'id' => 194,
                  'event_table_name' => 'faceSnapDB'
                },
                {
                  'id' => 204,
                  'event_table_name' => 'intrusionDB'
                },
                {
                  'id' => 205,
                  'event_table_name' => 'lineCrossDB'
                },
                {
                  'id' => 206,
                  'event_table_name' => 'regionEnterDB'
                },
                {
                  'id' => 207,
                  'event_table_name' => 'regionExitDB'
                },
                {
                  'id' => 208,
                  'event_table_name' => 'loiteringDB'
                },
                {
                  'id' => 209,
                  'event_table_name' => 'peopleGatherDB'
                },
                {
                  'id' => 210,
                  'event_table_name' => 'fastMoveDB'
                },
                {
                  'id' => 211,
                  'event_table_name' => 'ParkingDB'
                },
                {
                  'id' => 212,
                  'event_table_name' => 'objectLeaveDB'
                },
                {
                  'id' => 213,
                  'event_table_name' => 'objectRemoveDB'
                },
                {
                  'id' => 234,
                  'event_table_name' => 'MdWithTargetDB'
                },
                {
                  #---------- Extract first picture only ----------
                  'id' => 241,
                  'event_table_name' => 'FaceDetectTB'
                },
                {
                  'id' => 449,
                  'event_table_name' => 'IntrusionCaptureTable'
                },
                {
                  'id' => 450,
                  'event_table_name' => 'LineCrossCaptureTable'
                },
                {
                  'id' => 451,
                  'event_table_name' => 'RegionEnterCaptureTable'
                },
                {
                  'id' => 452,
                  'event_table_name' => 'RegionExitCaptureTable'
                },
                {
                  'id' => 453,
                  'event_table_name' => 'LoiteringCaptureTable'
                },
                {
                  'id' => 454,
                  'event_table_name' => 'PeopleGatherCaptureTable'
                },
                {
                  'id' => 455,
                  'event_table_name' => 'FastMoveCaptureTable'
                },
                {
                  'id' => 456,
                  'event_table_name' => 'ParkingCaptureTable'
                },
                {
                  'id' => 457,
                  'event_table_name' => 'ObjectLeaveCaptureTable'
                },
                {
                  'id' => 458,
                  'event_table_name' => 'ObjectRemoveCaptureTable'
                },
                {
                  'id' => 460,
                  'event_table_name' => 'TimingCaptureTB'
                },
                #------------------- Unknown id -------------------
                {
                  'id' => 1000,
                  'event_table_name' => 'ManualCaptureTB'
                },
                {
                  'id' => 1001,
                  'event_table_name' => 'MdWithTargetCaptureTable'
                },
                {
                  #---------- Extract first picture only ----------
                  'id' => 1002,
                  'event_table_name' => 'mixedTrafficDetectTB'
                }
            ];
        }
     
        $TBNumber = $#$ary1 + 1;
        print "\nNumber of tables in database: $TBNumber\n";
        sleep(1);
        my $tableName;
        for ($j=0; $j<$TBNumber; $j++) {
            $tableName = $ary1->[$j]{'event_table_name'};
            if ( grep( /^$tableName$/, @$tables ) ) {
                print "\n***** $tableName found *****\n";
                print "Processing table: $tableName: ";
                my $stmt = "SELECT id, trigger_time, file_no, pic_offset_0, pic_len_0 from ".$tableName;
                my $sth = $dbh->prepare($stmt) or die $DBI::errstr;
                my $rv = $sth->execute();
             
                if($rv < 0) {
                   print $DBI::errstr;
                }
                $ary = $sth->fetchall_arrayref({});

                my $TBLength = $#$ary + 1;
                print "$TBLength\n";
                sleep(1);
             
                for ($k=0; $k<$TBLength; $k++) {
                    $fileNumber = $ary->[$k]{'file_no'};
                    $picFileName = "hiv" . sprintf("%05d", $fileNumber) . ".pic";
                    open (PF, $inputDir . "/" . $picFileName) or die "ERROR: Can't find " .$picFileName. " in the input folder.\n";
                    binmode(PF);
                    $capDate = $ary->[$k]{'trigger_time'};
                    $startOffset = $ary->[$k]{'pic_offset_0'};
                    $endOffset = $startOffset + ($ary->[$k]{'pic_len_0'});
                    $formatted_start_time = time2str("%C", $capDate, -0005);
                    if ($outputDateFormat == 1) {
                        $fileDate = time2str("%Y_%m_%d-%H_%M_%S", $capDate, -0005);
                        $fileDayofWeek = time2str("%a", $capDate, -0005);
                    }
                    elsif ($outputDateFormat == 2) {
                        $fileDate = time2str("%Y%m%d%H%M%S", $capDate, -0005);
                        $fileDayofWeek = time2str("%w", $capDate, -0005);
                    }
                    else {
                        $fileDate = time2str("%Y_%m_%d-%H_%M_%S", $capDate, -0005);
                        $fileDayofWeek = time2str("%a", $capDate, -0005);
                    }
                    $limitFileDate = time2str("%Y%m%d%H%M%S", $capDate, -0005);
                                 
                    if ($inputStartDate != "" and $inputEndDate != "") {
                        if ($capDate > 0 and $limitFileDate >= $inputStartDate and $limitFileDate <= $inputEndDate) {
                            $jpegLength = ($endOffset - $startOffset);
                            $fileSize = $jpegLength / 1024;
                            $fileName = "Image_${fileDate}-${fileDayofWeek}.jpg";
                            if ($timeDuplicates != 2) {
                                my $picNumber = 1;
                                START:
                                    if (-e $outputDir."/".$fileName) {
                                    print "File Exists! Renaming...\n";
                                    $fileName = "Image_${fileDate}-${fileDayofWeek}-". sprintf("%03d", $picNumber) .".jpg";
                                    $picNumber++;
                                    goto START;
                                    }
                            }
                            unless (-e $outputDir."/".$fileName) {
                                if ($jpegLength > 0) {
                                    seek (PF, $startOffset, 0);
                                    read (PF, $singlejpeg, $jpegLength);
                                    print "PicFile: $picFileName\n";
                                    if ($singlejpeg =~ /[^\0]/) {
                                        print "POSITION (".($k+1)."): $formatted_start_time - OFFSET:($startOffset - $endOffset)\nFILE NAME: $fileName FILE SIZE: ". int($fileSize)." KB\n\n";
                                        open (OUTFILE, ">". $outputDir."/".$fileName);
                                        binmode(OUTFILE);
                                        print OUTFILE ${singlejpeg};
                                        close OUTFILE;
                                    }
                                }
                            }
                        }
                    }
                    else {
                        if ($capDate > 0) {
                            $jpegLength = ($endOffset - $startOffset);
                            $fileSize = $jpegLength / 1024;
                            $fileName = "Image_${fileDate}-${fileDayofWeek}.jpg";
                            if ($timeDuplicates != 2) {
                                my $picNumber = 1;
                                START:
                                    if (-e $outputDir."/".$fileName) {
                                    print "File Exists! Renaming...\n";
                                    $fileName = "Image_${fileDate}-${fileDayofWeek}-". sprintf("%03d", $picNumber) .".jpg";
                                    $picNumber++;
                                    goto START;
                                    }
                            }
                            unless (-e $outputDir."/".$fileName) {
                                if ($jpegLength > 0) {
                                    seek (PF, $startOffset, 0);
                                    read (PF, $singlejpeg, $jpegLength);
                                    print "PicFile: $picFileName\n";
                                    if ($singlejpeg =~ /[^\0]/) {
                                        print "POSITION (".($k+1)."): $formatted_start_time - OFFSET:($startOffset - $endOffset)\nFILE NAME: $fileName FILE SIZE: ". int($fileSize)." KB\n\n";
                                        open (OUTFILE, ">". $outputDir."/".$fileName);
                                        binmode(OUTFILE);
                                        print OUTFILE ${singlejpeg};
                                        close OUTFILE;
                                    }
                                }
                            }
                        }
                    }
                    close (PF);
                }
            }
            else {
                print "\nCan't find table: $tableName\n";
            }
        }
        $dbh->disconnect();
        print "Operation done for folder: $inputDir\n";
    }
    elsif ((-e $inputDir."/index00p.bin") or (-e $inputDir."/index01p.bin")) {
             
        $maxRecords = 4096;
        $recordSize = 80;

        open (FH,$inputDir . "/index00p.bin") or (print "\n\nCan't find index00p.bin, trying to use index01p.bin...\n" and open (FH,$inputDir . "/index01p.bin")) or die "ERROR: Can't find index00p.bin or index01p.bin in the input folder.\n";
        read (FH,$buffer,1280);
        #read (FH,$buffer,-s "index00.bin");

        ($modifyTimes, $version, $picFiles, $nextFileRecNo, $lastFileRecNo, $curFileRec, $unknown, $checksum) = unpack("Q1I1I1I1I1C1176C76I1",$buffer);
        #print "$modifyTimes, $version, $picFiles, $nextFileRecNo, $lastFileRecNo, $curFileRec, $unknown, $checksum\n";

        $currentpos = tell (FH);
        $offset = $maxRecords * $recordSize;
        $fullSize = $offset * $picFiles;

        for ($l=0; $l<$fullSize; $l++) {
                seek (FH, $l, 0); #Use seek to make sure we are at the right location, 'read' was occasionally jumping a byte
                $Headcurrentpos = tell (FH);
                read (FH,$Headbuffer,80); #Read 80 bytes for the record
                #print "************$Headcurrentpos***************\n";
                     
                ($Headfield1, $Headfield2, $HeadcapDate, $Headfield4, $Headfield5, $Headfield6, $Headfield7, $Headfield8, $Headfield9, $Headfield10, $HeadstartOffset, $HeadendOffset, $Headfield13, $Headfield14, $Headfield15, $Headfield16) = unpack("I*",$Headbuffer);
                     
                #print "\n$Headcurrentpos: $Headfield1, $Headfield2, $HeadcapDate, $Headfield4, $Headfield5, $Headfield6, $Headfield7, $Headfield8, $Headfield9, $Headfield10, $HeadstartOffset, $HeadendOffset, $Headfield13, $Headfield14, $Headfield15, $Headfield16\n";
                if ($HeadcapDate > 0 and $Headfield2 == 0 and $Headfield5 > 0 and $Headfield7 == 0 and $Headfield8 == 0 and $Headfield9 == 0 and $Headfield10 == 0 and $Headfield14 != 0 and $Headfield15 == 0 and $HeadstartOffset > 0 and $HeadendOffset > 0)
                {
                    $fullSize = 1;
                    $headerSize = $Headcurrentpos - $recordSize;
                    print "\nHeaderSize: $headerSize\nStarting...\n\n";
                    sleep (3);
                }
        }
     
        for ($m=0; $m<$picFiles; $m++) {
            my $LastCapDate = 0;
            $newOffset = $headerSize + ($offset * $m);
            seek (FH, $newOffset, 0);
            $picFileName = "hiv" . sprintf("%05d", $m) . ".pic";
            print "PicFile: $picFileName at $newOffset\n";
            open (PF, $inputDir . "/" . $picFileName) or die "ERROR: Can't find " . $picFileName . " in the input folder.\n";
            binmode(PF);
             
            for ($n=0; $n<$maxRecords; $n++) {
                $recordOffset = $newOffset + ($n * $recordSize); #get the next record location
                seek (FH, $recordOffset, 0); #Use seek to make sure we are at the right location, 'read' was occasionally jumping a byte
                $currentpos = tell (FH);
                read (FH,$buffer,80); #Read 80 bytes for the record
                #print "************$currentpos***************\n";
                     
                ($field1, $field2, $capDate, $field4, $field5, $field6, $field7, $field8, $field9, $field10, $startOffset, $endOffset, $field13, $field14, $field15, $field16) = unpack("I*",$buffer);
                #print "\n$field1, $field2, $capDate, $field4, $field5, $field6, $field7, $field8, $field9, $field10, $startOffset, $endOffset, $field13, $field14, $field15, $field16\n";
                $formatted_start_time = time2str("%C", $capDate, -0005);
                if ($outputDateFormat == 1) {
                    $fileDate = time2str("%Y_%m_%d-%H_%M_%S", $capDate, -0005);
                    $fileDayofWeek = time2str("%a", $capDate, -0005);
                }
                elsif ($outputDateFormat == 2) {
                    $fileDate = time2str("%Y%m%d%H%M%S", $capDate, -0005);
                    $fileDayofWeek = time2str("%w", $capDate, -0005);
                }
                else {
                    $fileDate = time2str("%Y_%m_%d-%H_%M_%S", $capDate, -0005);
                    $fileDayofWeek = time2str("%a", $capDate, -0005);
                }
                $limitFileDate = time2str("%Y%m%d%H%M%S", $capDate, -0005);
             
                #print "$currentpos: $field1, $field2, $capDate, $field4, $field5, $field6, $field7, $field8, $field9, $field10, $startOffset, $endOffset, $field13, $field14, $field15, $field16\n";
             
                if ($inputStartDate != "" and $inputEndDate != "") {
                    if ($capDate > 0 and $capDate > $LastCapDate and $limitFileDate >= $inputStartDate and $limitFileDate <= $inputEndDate) {
                        $jpegLength = ($endOffset - $startOffset);
                        $fileSize = $jpegLength / 1024;
                        $fileName = "Image_${fileDate}-${fileDayofWeek}.jpg";
                        if ($timeDuplicates != 2) {
                            my $picNumber = 1;
                            START:
                                if (-e $outputDir."/".$fileName) {
                                print "File Exists! Renaming...\n";
                                $fileName = "Image_${fileDate}-${fileDayofWeek}-". sprintf("%03d", $picNumber) .".jpg";
                                $picNumber++;
                                goto START;
                                }
                        }
                        unless (-e $outputDir."/".$fileName) {
                            if ($jpegLength > 0) {
                                seek (PF, $startOffset, 0);
                                read (PF, $singlejpeg, $jpegLength) or last;
                                if ($singlejpeg =~ /[^\0]/) {
                                    print "POSITION ($currentpos): $formatted_start_time - OFFSET:($startOffset - $endOffset)\nFILE NAME: $fileName FILE SIZE: ". int($fileSize)." KB\n\n";
                                    open (OUTFILE, ">". $outputDir."/".$fileName);
                                    binmode(OUTFILE);
                                    print OUTFILE ${singlejpeg};
                                    close OUTFILE;
                                }
                            }
                        }
                        if ($LastCapDate < $capDate) {
                        $LastCapDate = $capDate;
                        }
                    }
                }
                else {
                    if ($capDate > 0 and $capDate > $LastCapDate) {
                        $jpegLength = ($endOffset - $startOffset);
                        $fileSize = $jpegLength / 1024;
                        $fileName = "Image_${fileDate}-${fileDayofWeek}.jpg";
                        if ($timeDuplicates != 2) {
                            my $picNumber = 1;
                            START:
                                if (-e $outputDir."/".$fileName) {
                                print "File Exists! Renaming...\n";
                                $fileName = "Image_${fileDate}-${fileDayofWeek}-". sprintf("%03d", $picNumber) .".jpg";
                                $picNumber++;
                                goto START;
                                }
                        }
                        unless (-e $outputDir."/".$fileName) {
                            if ($jpegLength > 0) {
                                seek (PF, $startOffset, 0);
                                read (PF, $singlejpeg, $jpegLength) or last;
                                if ($singlejpeg =~ /[^\0]/) {
                                    print "POSITION ($currentpos): $formatted_start_time - OFFSET:($startOffset - $endOffset)\nFILE NAME: $fileName FILE SIZE: ". int($fileSize)." KB\n\n";
                                    open (OUTFILE, ">". $outputDir."/".$fileName);
                                    binmode(OUTFILE);
                                    print OUTFILE ${singlejpeg};
                                    close OUTFILE;
                                }
                            }
                        }
                        if ($LastCapDate < $capDate) {
                        $LastCapDate = $capDate;
                        }
                    }
                }
            }
            close (PF);
        }
        close FH;
    }
    else {
        die "\nERROR: Can't find event_db_index01 or index00p.bin or index01p.bin in $inputDir\n";
    }
}
print "\nAll operations completed.\n";

When I've first written the code that searches for the header size, I've had different index00p.bin files created by more than one camera, and by comparing all of them, I was able to create a search pattern. Now I only had the one index file from your camera.

When you format the storage medium, and the camera generates a new index file, the structure of the index file can/will change. That's why I don't know if the script will continue to work when your camera generates a new index file (it will output pictures, but they will be corrupted).

 

[Pimvg]

Alex, thanks for all the effort you are putting in!

Tried the new script against the existing test-set which contains the index file I sent. It starts working on the first .pic (xxx00000) and unpacks the jpg's successfully. After processing all files (incl the empty ones) the script displays the message: ERROR: Can't find event_db_index01 or index00p.bin or index01p.bin in D:\Hikv/jpg

Than I ran the script against a new test set that I generated yesterday, which contains many more captures and, as a result, more non-0KB .pic files. It seems to start unpacking a number of jpg's (which, by the way, are unreadable and which have much larger or much smaller sizes than expected)


So it doesn't seem to be processing any file but hiv00000.pic and finally ends with the error message as displayed above.
I have attached the index00p.bin of the new test set for your information.

So, that leaves me the option to use the Python script (suppose I would need to run on every individual .pic file) unless you would be able (and willing!!) to adapt the script to this different style index-file
In any case: Much appreciated!!

 

[Alex_N]

I've updated the code. Try it on both test sets. The more test sets, the better. It's normal for the script to process all the files, including the ones that are empty.
Those JPGs that are unreadable or have a size that is out of the ordinary are the corrupted files I was talking about and only appear when the script doesn't find the correct header value.

Spoiler: Updated code

#!/usr/bin/perl -

use Date::Format;
use File::Find::Rule;
use DBI;
#use Data::Dumper qw(Dumper);

my ($mainInputDir, $outputDir) = @ARGV;

if (not defined $mainInputDir) {
  die "\nERROR: Need input directory.\nThe input directory should be the directory that contains the datadir folders.\nUsage: extract_capture.pl inputdir outputdir\n";
}

if (not defined $outputDir) {
  die "\nERROR: Need output directory.\nUsage: extract_capture.pl inputdir outputdir\n";
}

unless ((-e $mainInputDir."/datadir0/event_db_index01") or (-e $mainInputDir."/datadir0/index00p.bin") or (-e $mainInputDir."/datadir0/index01p.bin")) {
        die "\nERROR: Can't find event_db_index01 or index00p.bin or index01p.bin in $mainInputDir\\datadir0\n";
    }

print "\n************* EXTRACTION DATE AND TIME LIMITS *************\n";
print "\nDate and time format example YYYYMMDDHHMMSS: 20181225221508\n";
print "\nEnter START date and time (or enter 1 to start from the\nfirst available image or leave empty if you don't want to use\na date and time limit): ";
my $inputStartDate = <STDIN>;
chomp $inputStartDate;

$nowEndDate = time2str("%Y%m%d%H%M%S", time());

if ($inputStartDate != "") {
    print "\nEnter END date and time (or enter 1 to use current date and\ntime (this will extract until the last image), or leave empty\nif you don't want to use a date and time limit): ";
    our $inputEndDate = <STDIN>;
    chomp $inputEndDate;
    if ($inputEndDate == 1) {
        $inputEndDate = $nowEndDate;
    }
}

print "\n************* OUTPUT JPG FILES NAMING OPTIONS (DEFAULT 1) *************\n";
print "\nEnter 1 for Image_YYYY_MM_DD-HH_MM_SS-DayName.jpg\nEx: Image_2018_12_25-22_15_08-Tue.jpg\n";
print "\nEnter 2 for Image_YYYYMMDDHHMMSS-DayNumber.jpg\nEx: Image_20181225221508-2.jpg\n";
my $outputDateFormat = <STDIN>;
chomp $outputDateFormat;

print "\n************* WHEN MORE THAN ONE PICTURE IS TAKEN PER SECOND (DEFAULT 1) *************\n";
print "\nEnter 1 to auto rename and keep all pictures\n";
print "\nEnter 2 to only save the first picture\n";
my $timeDuplicates = <STDIN>;
chomp $timeDuplicates;

my @datadirArray = File::Find::Rule->directory->in($mainInputDir);
my $datadirNumber = @datadirArray;  

for ($i=1; $i<$datadirNumber; $i++) {
    my $inputDir = $datadirArray[$i];
    print "Processing folder: $inputDir\n";

    if (-e $inputDir."/event_db_index01") {
        my $ary1;
        my $TBNumber;
        sleep(2);

        my $driver = "SQLite";
        my $database = $inputDir . "/event_db_index01";
        my $dsn = "DBI:$driver:dbname=$database";
        my $userid = "";
        my $password = "";
        my $dbh = DBI->connect($dsn, $userid, $password, { RaiseError => 1 }) or die $DBI::errstr;
       
        my $sth = $dbh->table_info('','main','%', 'TABLE');
        my $tables = [map{$_->[2]} @{$sth->fetchall_arrayref()}];
        #print join("\n", @$tables);
       
        if ( grep( /^event_id$/, @$tables ) ) {
            print "\n\n***** event_id found *****\n";
            my $stmt = qq(SELECT id, event_table_name from event_id;);
            my $sth = $dbh->prepare($stmt);
            my $rv = $sth->execute() or die $DBI::errstr;
                if($rv < 0) {
                    print $DBI::errstr;
                }
            $ary1 = $sth->fetchall_arrayref({});
            #print Dumper(\$ary1);      
        }
       
        else {
            $ary1 = [
                {
                  'id' => 10,
                  'event_table_name' => 'MotionDetectTB'
                },
                {
                  'id' => 12,
                  'event_table_name' => 'AlarmInputTB'
                },
                {
                  'id' => 13,
                  'event_table_name' => 'SceneChangeTB'
                },
                {
                  'id' => 24,
                  'event_table_name' => 'NormalPirTB'
                },
                {
                  'id' => 194,
                  'event_table_name' => 'faceSnapDB'
                },
                {
                  'id' => 204,
                  'event_table_name' => 'intrusionDB'
                },
                {
                  'id' => 205,
                  'event_table_name' => 'lineCrossDB'
                },
                {
                  'id' => 206,
                  'event_table_name' => 'regionEnterDB'
                },
                {
                  'id' => 207,
                  'event_table_name' => 'regionExitDB'
                },
                {
                  'id' => 208,
                  'event_table_name' => 'loiteringDB'
                },
                {
                  'id' => 209,
                  'event_table_name' => 'peopleGatherDB'
                },
                {
                  'id' => 210,
                  'event_table_name' => 'fastMoveDB'
                },
                {
                  'id' => 211,
                  'event_table_name' => 'ParkingDB'
                },
                {
                  'id' => 212,
                  'event_table_name' => 'objectLeaveDB'
                },
                {
                  'id' => 213,
                  'event_table_name' => 'objectRemoveDB'
                },
                {
                  'id' => 234,
                  'event_table_name' => 'MdWithTargetDB'
                },
                {
                  #---------- Extract first picture only ----------
                  'id' => 241,
                  'event_table_name' => 'FaceDetectTB'
                },
                {
                  'id' => 449,
                  'event_table_name' => 'IntrusionCaptureTable'
                },
                {
                  'id' => 450,
                  'event_table_name' => 'LineCrossCaptureTable'
                },
                {
                  'id' => 451,
                  'event_table_name' => 'RegionEnterCaptureTable'
                },
                {
                  'id' => 452,
                  'event_table_name' => 'RegionExitCaptureTable'
                },
                {
                  'id' => 453,
                  'event_table_name' => 'LoiteringCaptureTable'
                },
                {
                  'id' => 454,
                  'event_table_name' => 'PeopleGatherCaptureTable'
                },
                {
                  'id' => 455,
                  'event_table_name' => 'FastMoveCaptureTable'
                },
                {
                  'id' => 456,
                  'event_table_name' => 'ParkingCaptureTable'
                },
                {
                  'id' => 457,
                  'event_table_name' => 'ObjectLeaveCaptureTable'
                },
                {
                  'id' => 458,
                  'event_table_name' => 'ObjectRemoveCaptureTable'
                },
                {
                  'id' => 460,
                  'event_table_name' => 'TimingCaptureTB'
                },
                #------------------- Unknown id -------------------
                {
                  'id' => 1000,
                  'event_table_name' => 'ManualCaptureTB'
                },
                {
                  'id' => 1001,
                  'event_table_name' => 'MdWithTargetCaptureTable'
                },
                {
                  #---------- Extract first picture only ----------
                  'id' => 1002,
                  'event_table_name' => 'mixedTrafficDetectTB'
                }
            ];
        }
       
        $TBNumber = $#$ary1 + 1;
        print "\nNumber of tables in database: $TBNumber\n";
        sleep(1);
        my $tableName;
        for ($j=0; $j<$TBNumber; $j++) {
            $tableName = $ary1->[$j]{'event_table_name'};
            if ( grep( /^$tableName$/, @$tables ) ) {
                print "\n***** $tableName found *****\n";
                print "Processing table: $tableName: ";
                my $stmt = "SELECT id, trigger_time, file_no, pic_offset_0, pic_len_0 from ".$tableName;
                my $sth = $dbh->prepare($stmt) or die $DBI::errstr;
                my $rv = $sth->execute();
               
                if($rv < 0) {
                   print $DBI::errstr;
                }
                $ary = $sth->fetchall_arrayref({});

                my $TBLength = $#$ary + 1;
                print "$TBLength\n";
                sleep(1);
               
                for ($k=0; $k<$TBLength; $k++) {
                    $fileNumber = $ary->[$k]{'file_no'};
                    $picFileName = "hiv" . sprintf("%05d", $fileNumber) . ".pic";
                    open (PF, $inputDir . "/" . $picFileName) or die "ERROR: Can't find " .$picFileName. " in the input folder.\n";
                    binmode(PF);
                    $capDate = $ary->[$k]{'trigger_time'};
                    $startOffset = $ary->[$k]{'pic_offset_0'};
                    $endOffset = $startOffset + ($ary->[$k]{'pic_len_0'});
                    $formatted_start_time = time2str("%C", $capDate, -0005);
                    if ($outputDateFormat == 1) {
                        $fileDate = time2str("%Y_%m_%d-%H_%M_%S", $capDate, -0005);
                        $fileDayofWeek = time2str("%a", $capDate, -0005);
                    }
                    elsif ($outputDateFormat == 2) {
                        $fileDate = time2str("%Y%m%d%H%M%S", $capDate, -0005);
                        $fileDayofWeek = time2str("%w", $capDate, -0005);
                    }
                    else {
                        $fileDate = time2str("%Y_%m_%d-%H_%M_%S", $capDate, -0005);
                        $fileDayofWeek = time2str("%a", $capDate, -0005);
                    }
                    $limitFileDate = time2str("%Y%m%d%H%M%S", $capDate, -0005);
                                   
                    if ($inputStartDate != "" and $inputEndDate != "") {
                        if ($capDate > 0 and $limitFileDate >= $inputStartDate and $limitFileDate <= $inputEndDate) {
                            $jpegLength = ($endOffset - $startOffset);
                            $fileSize = $jpegLength / 1024;
                            $fileName = "Image_${fileDate}-${fileDayofWeek}.jpg";
                            if ($timeDuplicates != 2) {
                                my $picNumber = 1;
                                START:
                                    if (-e $outputDir."/".$fileName) {
                                    print "File Exists! Renaming...\n";
                                    $fileName = "Image_${fileDate}-${fileDayofWeek}-". sprintf("%03d", $picNumber) .".jpg";
                                    $picNumber++;
                                    goto START;
                                    }
                            }
                            unless (-e $outputDir."/".$fileName) {
                                if ($jpegLength > 0) {
                                    seek (PF, $startOffset, 0);
                                    read (PF, $singlejpeg, $jpegLength);
                                    print "PicFile: $picFileName\n";
                                    if ($singlejpeg =~ /[^\0]/) {
                                        print "POSITION (".($k+1)."): $formatted_start_time - OFFSET:($startOffset - $endOffset)\nFILE NAME: $fileName FILE SIZE: ". int($fileSize)." KB\n\n";
                                        open (OUTFILE, ">". $outputDir."/".$fileName);
                                        binmode(OUTFILE);
                                        print OUTFILE ${singlejpeg};
                                        close OUTFILE;
                                    }
                                }
                            }
                        }
                    }
                    else {
                        if ($capDate > 0) {
                            $jpegLength = ($endOffset - $startOffset);
                            $fileSize = $jpegLength / 1024;
                            $fileName = "Image_${fileDate}-${fileDayofWeek}.jpg";
                            if ($timeDuplicates != 2) {
                                my $picNumber = 1;
                                START:
                                    if (-e $outputDir."/".$fileName) {
                                    print "File Exists! Renaming...\n";
                                    $fileName = "Image_${fileDate}-${fileDayofWeek}-". sprintf("%03d", $picNumber) .".jpg";
                                    $picNumber++;
                                    goto START;
                                    }
                            }
                            unless (-e $outputDir."/".$fileName) {
                                if ($jpegLength > 0) {
                                    seek (PF, $startOffset, 0);
                                    read (PF, $singlejpeg, $jpegLength);
                                    print "PicFile: $picFileName\n";
                                    if ($singlejpeg =~ /[^\0]/) {
                                        print "POSITION (".($k+1)."): $formatted_start_time - OFFSET:($startOffset - $endOffset)\nFILE NAME: $fileName FILE SIZE: ". int($fileSize)." KB\n\n";
                                        open (OUTFILE, ">". $outputDir."/".$fileName);
                                        binmode(OUTFILE);
                                        print OUTFILE ${singlejpeg};
                                        close OUTFILE;
                                    }
                                }
                            }
                        }
                    }
                    close (PF);
                }
            }
            else {
                print "\nCan't find table: $tableName\n";
            }
        }
        $dbh->disconnect();
        print "Operation done for folder: $inputDir\n";
    }
    elsif ((-e $inputDir."/index00p.bin") or (-e $inputDir."/index01p.bin")) {  
               
        $maxRecords = 4096;
        $recordSize = 80;

        open (FH,$inputDir . "/index00p.bin") or (print "\n\nCan't find index00p.bin, trying to use index01p.bin...\n" and open (FH,$inputDir . "/index01p.bin")) or die "ERROR: Can't find index00p.bin or index01p.bin in the input folder.\n";
        read (FH,$buffer,1280);
        #read (FH,$buffer,-s "index00.bin");

        ($modifyTimes, $version, $picFiles, $nextFileRecNo, $lastFileRecNo, $curFileRec, $unknown, $checksum) = unpack("Q1I1I1I1I1C1176C76I1",$buffer);
        #print "$modifyTimes, $version, $picFiles, $nextFileRecNo, $lastFileRecNo, $curFileRec, $unknown, $checksum\n";

        $currentpos = tell (FH);
        $offset = $maxRecords * $recordSize;
        $fullSize = $offset * $picFiles;

        for ($l=0; $l<$fullSize; $l++) {
                seek (FH, $l, 0); #Use seek to make sure we are at the right location, 'read' was occasionally jumping a byte
                $Headcurrentpos = tell (FH);
                read (FH,$Headbuffer,80); #Read 80 bytes for the record
                #print "************$Headcurrentpos***************\n";
                       
                ($Headfield1, $Headfield2, $HeadcapDate, $Headfield4, $Headfield5, $Headfield6, $Headfield7, $Headfield8, $Headfield9, $Headfield10, $HeadstartOffset, $HeadendOffset, $Headfield13, $Headfield14, $Headfield15, $Headfield16) = unpack("I*",$Headbuffer);
                       
                #print "\n$Headcurrentpos: $Headfield1, $Headfield2, $HeadcapDate, $Headfield4, $Headfield5, $Headfield6, $Headfield7, $Headfield8, $Headfield9, $Headfield10, $HeadstartOffset, $HeadendOffset, $Headfield13, $Headfield14, $Headfield15, $Headfield16\n";
                if ($HeadcapDate > 0 and $Headfield1 != 0 and $Headfield2 == 0 and $Headfield5 > 0 and $Headfield7 == 0 and $Headfield8 == 0 and $Headfield9 == 0 and $Headfield10 == 0 and $Headfield14 != 0 and $Headfield15 == 0 and $HeadstartOffset > 0 and $HeadendOffset > 0)
                {
                    $fullSize = 1;
                    $headerSize = $Headcurrentpos - $recordSize;
                    print "\nHeaderSize: $headerSize\nStarting...\n\n";
                    sleep (3);
                }
        }
       
        for ($m=0; $m<$picFiles; $m++) {
            my $LastCapDate = 0;
            $newOffset = $headerSize + ($offset * $m);
            seek (FH, $newOffset, 0);
            $picFileName = "hiv" . sprintf("%05d", $m) . ".pic";
            print "PicFile: $picFileName at $newOffset\n";
            open (PF, $inputDir . "/" . $picFileName) or die "ERROR: Can't find " . $picFileName . " in the input folder.\n";
            binmode(PF);
               
            for ($n=0; $n<$maxRecords; $n++) {
                $recordOffset = $newOffset + ($n * $recordSize); #get the next record location
                seek (FH, $recordOffset, 0); #Use seek to make sure we are at the right location, 'read' was occasionally jumping a byte
                $currentpos = tell (FH);
                read (FH,$buffer,80); #Read 80 bytes for the record
                #print "************$currentpos***************\n";
                       
                ($field1, $field2, $capDate, $field4, $field5, $field6, $field7, $field8, $field9, $field10, $startOffset, $endOffset, $field13, $field14, $field15, $field16) = unpack("I*",$buffer);
                #print "\n$field1, $field2, $capDate, $field4, $field5, $field6, $field7, $field8, $field9, $field10, $startOffset, $endOffset, $field13, $field14, $field15, $field16\n";
                $formatted_start_time = time2str("%C", $capDate, -0005);
                if ($outputDateFormat == 1) {
                    $fileDate = time2str("%Y_%m_%d-%H_%M_%S", $capDate, -0005);
                    $fileDayofWeek = time2str("%a", $capDate, -0005);
                }
                elsif ($outputDateFormat == 2) {
                    $fileDate = time2str("%Y%m%d%H%M%S", $capDate, -0005);
                    $fileDayofWeek = time2str("%w", $capDate, -0005);
                }
                else {
                    $fileDate = time2str("%Y_%m_%d-%H_%M_%S", $capDate, -0005);
                    $fileDayofWeek = time2str("%a", $capDate, -0005);
                }
                $limitFileDate = time2str("%Y%m%d%H%M%S", $capDate, -0005);
               
                #print "$currentpos: $field1, $field2, $capDate, $field4, $field5, $field6, $field7, $field8, $field9, $field10, $startOffset, $endOffset, $field13, $field14, $field15, $field16\n";
               
                if ($inputStartDate != "" and $inputEndDate != "") {
                    if ($capDate > 0 and $capDate > $LastCapDate and $limitFileDate >= $inputStartDate and $limitFileDate <= $inputEndDate) {
                        $jpegLength = ($endOffset - $startOffset);
                        $fileSize = $jpegLength / 1024;
                        $fileName = "Image_${fileDate}-${fileDayofWeek}.jpg";
                        if ($timeDuplicates != 2) {
                            my $picNumber = 1;
                            START:
                                if (-e $outputDir."/".$fileName) {
                                print "File Exists! Renaming...\n";
                                $fileName = "Image_${fileDate}-${fileDayofWeek}-". sprintf("%03d", $picNumber) .".jpg";
                                $picNumber++;
                                goto START;
                                }
                        }
                        unless (-e $outputDir."/".$fileName) {
                            if ($jpegLength > 0) {
                                seek (PF, $startOffset, 0);
                                read (PF, $singlejpeg, $jpegLength) or last;
                                if ($singlejpeg =~ /[^\0]/) {
                                    print "POSITION ($currentpos): $formatted_start_time - OFFSET:($startOffset - $endOffset)\nFILE NAME: $fileName FILE SIZE: ". int($fileSize)." KB\n\n";
                                    open (OUTFILE, ">". $outputDir."/".$fileName);
                                    binmode(OUTFILE);
                                    print OUTFILE ${singlejpeg};
                                    close OUTFILE;
                                }
                            }
                        }
                        if ($LastCapDate < $capDate) {
                        $LastCapDate = $capDate;
                        }
                    }
                }
                else {
                    if ($capDate > 0 and $capDate > $LastCapDate) {
                        $jpegLength = ($endOffset - $startOffset);
                        $fileSize = $jpegLength / 1024;
                        $fileName = "Image_${fileDate}-${fileDayofWeek}.jpg";
                        if ($timeDuplicates != 2) {
                            my $picNumber = 1;
                            START:
                                if (-e $outputDir."/".$fileName) {
                                print "File Exists! Renaming...\n";
                                $fileName = "Image_${fileDate}-${fileDayofWeek}-". sprintf("%03d", $picNumber) .".jpg";
                                $picNumber++;
                                goto START;
                                }
                        }
                        unless (-e $outputDir."/".$fileName) {
                            if ($jpegLength > 0) {
                                seek (PF, $startOffset, 0);
                                read (PF, $singlejpeg, $jpegLength) or last;
                                if ($singlejpeg =~ /[^\0]/) {
                                    print "POSITION ($currentpos): $formatted_start_time - OFFSET:($startOffset - $endOffset)\nFILE NAME: $fileName FILE SIZE: ". int($fileSize)." KB\n\n";
                                    open (OUTFILE, ">". $outputDir."/".$fileName);
                                    binmode(OUTFILE);
                                    print OUTFILE ${singlejpeg};
                                    close OUTFILE;
                                }
                            }
                        }
                        if ($LastCapDate < $capDate) {
                        $LastCapDate = $capDate;
                        }
                    }
                }
            }
            close (PF);
        }
        close FH;
    }
    else {  
        die "\nERROR: Can't find event_db_index01 or index00p.bin or index01p.bin in $inputDir\n";
    }
}
print "\nAll operations completed.\n";

 

[Pimvg]

Alex,
you continue to amaze me! Your response is crisp and you provide adjusted coding with lightning speed. Impressive.

I ran the updated code against the three test-sets that I created earlier. For this I used two different SD-cards which were formatted in the camera before use. The three sets contain a various number of images (60+, 600+ and 16.000+). JPGs were created flawlessly and are all readable.

So, even though I cannot do the real proof until I have the other camera back (it is now running a time-lapse capturing on a construction site of my neighbour's new house), which is expected to be by the end of the year. It will then contain some 250.000 images so your script will be a great help as it beats downloading those images in sets of 200.

So again, a big Thank You! for all your help, guidance, and effort. I owe you one!

Btw: All three runs end with the "ERROR: Can't find event_db_index01 or index00p.bin or index01p.bin in D:\Hikv/jpg" but that's only cosmetic, in my view.

 

[Alex_N]

It's possible that this custom code will not work with that other camera if it's a different model. You should first try to run the original code from my GitHub page.

I don't know why it gives you that error, but D:\Hikv\jpg sounds like the output folder. Did you copy the datadir folder to D:\Hikv\jpg ? Your image output folder should not be inside the source folder.

 

[Pimvg]

The other camera is the same model and has the same firmware level as the one that I used for testing. The only difference is the size of the SD-card. So hopeful

The source data is in D:\Hikv\datadir0
The output is in D:\Hikv\jpg
Is that ok or should the jpg dir be somewhere else?
Again: This is not an issue for me and just cosmetic.

 

[Alex_N]

The script checks all the subfolders of the input folder for index files. If you place the output JPG folder inside the input folder, the script will check it. However, since there are no index files in the JPG output folder, it will give you an error. Therefore, do not place the JPG output folder inside the input folder.
Ex: Input folder: D:\Hikv should contain the datadir subfolders
Image output folder: D:\jpg