Latest Git snapshot raises "Range Check error"

[aehimself]

I have a project which connects to a MySQL database. After refreshing Zeos to the currently latest version on GIT, one of the queries started throwing range check errors in ZVariant.pas, line 2464:

function TZSoftVariantManager.Convert(const Value: TZVariant; NewType: TZVariantType): TZVariant;
[...]
{$IF defined (RangeCheckEnabled) and defined(WITH_UINT64_C1118_ERROR)}{$R-}{$IFEND}
vtUInteger: Result.VUInteger := GetAsUInteger(Value);
{$IF defined (RangeCheckEnabled) and defined(WITH_UINT64_C1118_ERROR)}{$R+}{$IFEND}

I am setting a filter on the query, and then call refresh. Call stack is:

MyZQueryWrapper.SetFilter
MyZQueryWrapper.Refresh
Data.DB.TDataSet.Refresh
ZAbstractRODataSet.TZAbstractRODataSet.InternalRefresh
ZAbstractRODataSet.TZAbstractRODataSet.Locate
ZAbstractRODataSet.TZAbstractRODataSet.InternalLocate
ZDataSetUtils.PrepareValuesForComparison
ZVariant.TZSoftVariantManager.Convert

I am using UInt64-s for IDs and UNIX timestamps, table structure is:

CREATE TABLE `capacityinfo` (
`CapacityInfoID` bigint(20) unsigned NOT NULL AUTO_INCREMENT COMMENT 'Main ID field for table CapacityInfo',
`Created` bigint(20) unsigned NOT NULL COMMENT 'Record creation date',
`Modified` bigint(20) unsigned NOT NULL COMMENT 'Record last modification date',
`ClientID` bigint(20) unsigned NOT NULL COMMENT 'External ID field for table Clients',
`DateTime` bigint(20) unsigned NOT NULL,
`DriveLetter` varchar(50) CHARACTER SET utf16 COLLATE utf16_general_ci NOT NULL,
`DriveSize` int(11) NOT NULL,
`DriveFree` int(11) NOT NULL,
PRIMARY KEY (`CapacityInfoID`),
KEY `FK_CapacityInfo_Clients_idx` (`ClientID`),
CONSTRAINT `FK_CapacityInfo_Clients` FOREIGN KEY (`ClientID`) REFERENCES `clients` (`clientid`)
) ENGINE=InnoDB DEFAULT CHARSET=utf16 ROW_FORMAT=DYNAMIC

Is it safe to just turn off RangeCheckEnabled; or it's an issue I should attempt to look into? Is it possible that the issue is in Zeos, as it was working with a previous build? As I am using the component from DCUs, I have no idea which version broke.

[marsupilami]

Hello aehimself

duh - I suppose this is an error in Zeos. The git repo was a bit out of date. I pushed the latest updates there now. Could you please check if the error still happens? Which compiler do you use for this project? If you could create a small demo application, we could try to check different revisons of Zeos to find out, which revision introduces the error. Also I am not sure if Zeos will understands "bigint unsigned" correctly and if Delphi has a field type that works here...

Best regards,

Jan

[aehimself]

Refreshed, recompiled, different call stack; same error.

I think the issue will be a little bit above, see:

Capture.PNG

As for bigint, according to Oracle:
BIGINT 0 .. 2^64-1 (https://dev.mysql.com/doc/refman/8.0/en ... types.html)
Delphi's UInt64 is exactly the same:
Uint64 0 .. 18446744073709551615 (0 .. 2^64-1)

The field itself is UInt64, but stored values are rather small. The largest I can see is 1587992495, which is easily within standard Integer range.

I'll attempt to debug it; but the strange thing is that like 10 more tables have similar layouts and they do work without issues...

The whole reason I started to debug is that I noticed something really strange lately:

Capture2.PNG

...while in the database the UNIX timestamps are indeed correct. It is just sent out as garbage.

[aehimself]

What the everliving f*ck...?

Capture3.PNG

I mean, even these numbers (barely but they do) fit into a UInt64, but honestly... if I can not read them out from the database, how did they get in?!

I guess Zeos's only issue is that they don't really handle close-to-max UInt64 values. My issue lies somewhere else.
Those should be UNIX timestamps by the way.

https://i.redd.it/q1d1cb2o3j9z.jpg

Edit: oh, boy...
[2020.04.27 15:36:15.738] SQL trace: '2020-04-27 15:04:15' cat: Execute, proto: mysql, msg: Statement 15 : UPDATE CapacityInfo SET DateTime = Created WHERE DateTime > 1587992496
[2020.04.27 15:36:15.892] Tab 1, MyConnection: Query affected 132 row(s) in 163 milliseconds

Edit-edit: after the above quick fix, everything works like a charm. They were not sent out garbage. They were stored garbage...

[aehimself]

And the other way around, too.

[2020.04.27 15:44:14.262] SQL trace: '2020-04-27 15:04:14' cat: Execute, proto: mysql, msg: Statement 23 : UPDATE CapacityInfo SET DateTime = Created WHERE DateTime < 1562583615
[2020.04.27 15:44:14.398] Tab 0: Query affected 188 row(s) in 143 milliseconds

[EgonHugeist]

Hello aehimself,
i'm not able to reproduce it on my side. Could you write a small test? Changing the stack often shows different alltime errors...

[aehimself]

MySQL 8.0.18, client version: 8.0.18, database component version: 7.3.0-alpha

CREATE TABLE `temp` (
`f` bigint(20) unsigned DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf16;

Preparation:

Code: Select all

  q.SQL.Text := 'SELECT * FROM temp';
  q.Open;
  Try
   q.Append;
   q.FieldByName('f').AsLargeInt := UInt64.MaxValue;
   q.ApplyUpdates;
  Finally
   q.Close;
  End;

Confirm that value is in there, correctly:

Capture4.PNG

Issue:

Code: Select all

  q.SQL.Text := 'SELECT * FROM temp';
  q.Open;
  Try
   q.Refresh; // Range check error pops up here
  Finally
   q.Close;
  End;

q is a simple TZQuery component. Basically, if you are standing on a record, which contains a number higher than Integer.MaxValue and call refresh; the error will appear when the record is attempted to be located. As I see from the code, TZSoftVariantManager is not prepared for any other simple types than Integer:

Code: Select all

  case Value.VType of
    vtNull:     Result := 0;
    vtBoolean:  Result := Ord(Value.VBoolean);
    vtInteger:  Result := Value.VInteger;
    vtUInteger: Result := Value.VUInteger;
    vtCurrency: Result := PInt64(@Value.VCurrency)^ div 10000;
    vtBigDecimal:BCD2UInt64(Value.VBigDecimal, Result);
    {$IFNDEF UNICODE}vtString,{$ENDIF}
    {$IFNDEF NO_ANSISTRING}vtAnsiString,{$ENDIF}
    {$IFNDEF NO_UTF8STRING}vtUTF8String,{$ENDIF}
    vtRawByteString: Result := RawToUInt64Def(Pointer(Value.VRawByteString), 0);
    {$IFDEF UNICODE}vtString,{$ENDIF}
    vtUnicodeString: Result := UnicodeToUInt64Def(Value.VUnicodeString, 0);
    vtCharRec: if (Value.VCharRec.CP = zCP_UTF16)
      then Result := UnicodeToUInt64Def(PWideChar(Value.VCharRec.P), PWideChar(Value.VCharRec.P)+Value.VCharRec.Len, 0)
      else Result := RawToUInt64Def(PAnsiChar(Value.VCharRec.P), PAnsiChar(Value.VCharRec.P)+Value.VCharRec.Len, 0);
    vtPointer:  Result := {%H-}NativeUInt(Value.VPointer);
    else Result := {$IFDEF USE_FAST_TRUNC}ZFastCode.{$ENDIF}Trunc(GetAsDouble(Value));
  end;

[marsupilami]

MySQL 8.0.18, client version: 8.0.18, database component version: 7.3.0-alpha

Just out of curiosity: I assume this is a 64 bits application? Or did you get a 32 bits MySQL client library somewhere? I ask this question because I failed to locate a 32 bits library with the current version. So if there is a way to download a current 32 bits mysql client library somewhere I am interested in getting it...

[marsupilami]

I tried to create a test case - based on your description.Unfortunately the test passes in my test environment:
Zeos 7.3 rev. 6483, MySQL 8.0.19, libmariadb from MariaDB Connector C 3.1.7, Delphi 2010.

I will checkin the test nonetheless - maybe the error happens on newer Delphi versions or helps someone else.

Any ideas?

[marsupilami]

And one more note: I ran that new test through the different compilers that we have for testing. I get range check errors on Delphi versions 2009 and XE 2. Also on FPC 3.0 and 3.2.
The error doesn't happen on Delphi 7 and Delphi 2010. But that might be a problem of the configuration for these compilers. Maybe range checking is just disabled on these compilers and the problem will show there too once we enable range checking.

[aehimself]

Just out of curiosity: I assume this is a 64 bits application?

Yes, this is the output of my SQL Client, which is 64 bit. Indeed the latest 32 bit MySQL driver is 6.1.11.0 - the test was done with that one.

I get range check errors on Delphi versions 2009 and XE 2. Also on FPC 3.0 and 3.2.

Yes, forgot to mention - I'm using Delphi 10.3.3.

[aehimself]

Jan, I see you added the test case to the repository, thank you for that. I am sorry that I am uncapable of debugging / fixing it myself - to be honest I dislike Variants and kind of afraid of them; I avoid them whenever I can. And exactly because of this - I have no experience.

I also would like to apologize the inappropriate and incompletely short posts from yesterday - kid decided to examine my eye with her sharp nails, causing some serious damage in it during the process. Fortunately things are getting better already, but I still have bad days

[EgonHugeist]

Hello aehimself,
i've resolved your rangecheck regression see:https://sourceforge.net/p/zeoslib/code-0/6486/
puh your git was very very old, propably ... time for svn?

[marsupilami]

I also would like to apologize the inappropriate and incompletely short posts from yesterday - kid decided to examine my eye with her sharp nails, causing some serious damage in it during the process. Fortunately things are getting better already, but I still have bad days

Don't worry - that kind of things happen. I hope you will get better soon

puh your git was very very old, propably ... time for svn?

lol - some would say it is time for Zeos to move to git - Anyway - I may be wrong and I forgot to update the git repo for some time. But it always was quite close to the SVN. Not more than a mont of delay
But it should work now. Maybe it is time to automate the svn->git conversion, so it stays up to date. Maybe make it another job on Jenkins...

[marsupilami]

Maybe it is time to automate the svn->git conversion, so it stays up to date. Maybe make it another job on Jenkins...

Done. If everything works as expected, the Github repository will be updated daily at 0:30 now by a Jenkins job. We also could have it check the Zeos svn every xxx minutes and have it run then.
This all is still considered to be experimental. If things go wrong I might have to delete the GIT repository and start over new.