D2010 + Firebird + Varchar doubles FieldSize?

[louis]

Hallo,
I initiated a new project with D2010 and Firebird.

I notice that D2010 doubles the size of TWideString Field if the Firebird column is VarChar.

I have a column of VarChar(20) but a TWideStringField with a Size = 40!

This cause problems because DBEdit permit a 40 character entry but the database raises a post error because the size of column is 20!

If I verify the property of database with Flamerobin I get the "Default character set = UTF8" and "ODS Version = 11.2"

Why I have this problems? Can I Solve without reduce to half size of all TWideString Persistent Fields on all tables of the project?

Thanks.

Louis

[marsupilami]

Hello Louis,

which Version of Zeos do you use?
Best regards,

Jan

[louis]

which Version of Zeos do you use?

I use version 7.1.4-stable because version 7.2 is in beta.

Thanks.

[marsupilami]

Hello Louis,

I can confirm that bug. I will have a look at this later on and keep you informed.
With best regards,

Jan

[louis]

I will have a look at this later on and keep you informed.

Ok, thanks

[marsupilami]

Hello Louis,

sooo - I checked on that one. And I remember that I have seen it in the IDE. Then I tried it in a real program. There it didn't happen. And after recompiling the packages everything seems to be fine now. But I am using the latest 7.1-version from the SVN. Could you please test the current SVN version and make sure that you have no old dcu files anywhere? The link to the current 7.1-SVN version is:
http://svn.code.sf.net/p/zeoslib/code-0/trunk zeoslib-code-0/branches/7.1-patches/

Please let me know about your findings.

With best regards,

Jan

[louis]

Please let me know about your findings.

The problem happens even.

Try this:

• a) Create a new Datamodule;
  b) put in a ZConnection and configure for a existing database with a table that contains a field of type Varchar;
  c) put in a ZTable and point the table at point b);
  d) Right Click on ZTable and open Fields Editor then "Add All Fields"

Now all sizes of the varchar fields have created corresponding fields of type TWideStringField but the size are Doubled respect of field size in the database. I.E. if size of varchar field is 20 the widestringfield is of size of 40!

What is difference on stability from 7.1.4-stable version versus the svn version?

Thanks.

[EgonHugeist]

@ALL

NOT A BUG!!! I wrote it serveral times in between.

Note a 4Byte UTF8 encoded char (Chinese/Japanese f.e) needs 2 WideChars space.

UTF8 Characterset requires 3/4Byte per Char (chinese overlongs up to 6Byte/char) so a 4/3Byte /FireBird reserved amount of bytes for one UTF16 char) will never fit in a simple (2Byte) WideChar!

Call it a bug what ever you want. But tomorrow a chinese, korean or japanese guy writes he will get a String truncation for his environment.

Please read www.unicode.org, study encodings and you will understand why Varchar(10) * 3or4Byte per Character(30 or 40Byte) = 20 WideChar Characters.

[louis]

NOT A BUG!!! I wrote it serveral times in between.

Please read http://www.unicode.org, study encodings and you will understand why Varchar(10) * 3or4Byte per Character(30 or 40Byte) = 20 WideChar Characters.

Ok, but If I have a varcar(10) field in database and a TWideStringField of 20 characters, the TDBEdit permit me write 20 characters! and when i try post to write data in database table, Firebird raises exception, not truncates the text of TDBEdit but raises this exception:

SQL Error: Dynamic SQL Error SQL error code = -303 arithmetic exceptio, numeric overflow, or string truncation string right truncation. Error Code: -303. Incompatible comuln/host variable data type.

and not permits to save data in database! I Tried with D2010 + ztable + FB 2.5.4.
There is a character set issue?

The settings of Firebird database:
Dialect = 3;
Default Character set = UTF8.

Thanks.

[marsupilami]

Note a 4Byte UTF8 encoded char (Chinese/Japanese f.e) needs 2 WideChars space.

UTF8 Characterset requires 3/4Byte per Char (chinese overlongs up to 6Byte/char) so a 4/3Byte /FireBird reserved amount of bytes for one UTF16 char) will never fit in a simple (2Byte) WideChar!

Call it a bug what ever you want. But tomorrow a chinese, korean or japanese guy writes he will get a String truncation for his environment.

Hello Michael,

I have to disagree. TWideStringField has two properties - Size and DataSize. Size is the size in characters

Size is the maximum number of characters in the string.

Ok - I have to admit, this holds true for ANSI Strings. But - TWideStringField inherits from TAnsiString and there unfortunately is no separate information for ftWideString in the Wiki. [1]
Then there is the DataSize property [2]. The Embarcadero Wiki says:

Indicates the amount of memory needed to store a field component's value.

Check DataSize to determine the number of bytes required to store a field component's value. Use DataSize to determine the required size of a buffer for working with the field's value in native format. For example, use DataSize to determine the buffer size needed by the GetData and SetData methods.

So - this is the correct property for your aim, I think. The data Size for a TWideStringField schould be 4 * Size because the encoding of one character can take up to four bytes.

And as far as I know, Zeos handles things like this for TStringField already. I know that pretty well because the source of ZeosLib contains an exception for TStringField in Freepascal to not do so there because Freepascal doesn't allow to set the DataSize property for TStringFields.

So - in my book having a Size that is different from x for a varchar(x) is a bug. And I don't know why but that bug doesn't happen on my computer. But I will check it again.

1: http://docwiki.embarcadero.com/Librarie ... Field.Size
2: http://docwiki.embarcadero.com/Librarie ... d.DataSize

[EgonHugeist]

Hello Jan, Louis.

EgonHugeist wrote:
Note a 4Byte UTF8 encoded char (Chinese/Japanese f.e) needs 2 WideChars space.

UTF8 Characterset requires 3/4Byte per Char (chinese overlongs up to 6Byte/char) so a 4/3Byte /FireBird reserved amount of bytes for one UTF16 char) will never fit in a simple (2Byte) WideChar!

Call it a bug what ever you want. But tomorrow a chinese, korean or japanese guy writes he will get a String truncation for his environment.


Hello Michael,

I have to disagree. TWideStringField has two properties - Size and DataSize. Size is the size in characters

Embarcadero Wiki wrote:
Size is the maximum number of characters in the string.

Ok - I have to admit, this holds true for ANSI Strings. But - TWideStringField inherits from TAnsiString and there unfortunately is no separate information for ftWideString in the Wiki. [1]
Then there is the DataSize property [2]. The Embarcadero Wiki says:

Embarcadero Wiki wrote:
Indicates the amount of memory needed to store a field component's value.

Check DataSize to determine the number of bytes required to store a field component's value. Use DataSize to determine the required size of a buffer for working with the field's value in native format. For example, use DataSize to determine the buffer size needed by the GetData and SetData methods.

So - this is the correct property for your aim, I think. The data Size for a TWideStringField schould be 4 * Size because the encoding of one character can take up to four bytes.

Exactly this is the point!
There is a bug(MPOV) inside the TString/TWideString fields.

Note

Code: Select all

  property DataSize: Integer read GetDataSize;
 
  // DataSize is byte count
function TWideStringField.GetDataSize: Integer;
begin
  Result := (Size + 1) * 2;
end;

  and in addition think about these liunes:

function TWideStringField.GetValue(var Value: UnicodeString): Boolean;
var
  Temp: array of Char;
  PBuf: PChar;
  Buffer: array[0..dsMaxStringSize div 2] of Char;
begin
  if DataSize > SizeOf(Buffer) then
  begin
    SetLength(Temp, (DataSize div 2) + 1);
    PBuf := PChar(Temp);
  end
  else
    PBuf := @Buffer;
  Result := GetData(PBuf, False);
  if Result then
    Value := PBuf;
end;

As you can see DataSize just aligns to max 2Byte Characters depending on property Size. But UTF8 allows 1-6 Byte / character, UCS2 2-6Byte per character.

I would agree IF TWideStringField.GetValue whouldn't use a static buffer or the buff-size would be calculated * 2.

What happens IF TField move a String -> GetData calls GetValue which calls a CopyData defined as StrCopy(PChar(Dest), PChar(Source))
VARCHAR(10) x 4Bytes / Char (40Byte total + trailing #0#0) ????

---> A pretty nice Stack overrun would be result.

@Louis
Could you try(untested -> no time) to limit the String by limiting the DisplayWidth?

@Jan

And as far as I know, Zeos handles things like this for TStringField already. I know that pretty well because the source of ZeosLib contains an exception for TStringField in Freepascal to not do so there because Freepascal doesn't allow to set the DataSize property for TStringFields.

So - in my book having a Size that is different from x for a varchar(x) is a bug. And I don't know why but that bug doesn't happen on my computer. But I will check it again.

? Zeos throws a Exception ? You meen the SQLDB Components still have this issue i think. AFAIK did i introduce a Buffer-Size equal to (count of Chars) x (Count of Bytes per charset of the DB). And SETTING DataSize is impossible on Delphi too.

Well second suggestion would be: Switch on my TZFields which i've allready started. What you guys are thinking why other DB-Components allways have it's own TString/TWideStringField descendants?

Cheers, Michael

[louis]

@Louis
Could you try(untested -> no time) to limit the String by limiting the DisplayWidth?

The property DisplayWidth is intended for display on DBGrid/DBLookup. Not for editing/saving data.
From Embarcadero Help:

Description
Specifies the number of characters that should be used to display a field's value by a cooperating data-aware control.

Set DisplayWidth to control how many characters are used to calculate the column width for the field when it is displayed by a TDBLookupListBox object or a descendant of TCustomDBGrid. The width of columns is determined by the average character width. If the data-aware control that displays the field is not using a fixed pitch font, DisplayWidth may indicate more or fewer characters than will fit in the column for any given string.

Note: The DisplayWidth property has no effect if there is an explicit Columns value for the field in a TDBGrid object.

I tried, like suggests you and I confirm: change displayWidth not solve the problem but the space occupied on column of DBGrid/DBLookup is reduced to half.


Thanks.

[EgonHugeist]

Hi All,

my code was the "HEAP"-Safe code which avoids a possible overrun.

But never discuss with the users ...

so i did add a TZDataSet.Options enum

Code: Select all

doAlignMaxRequiredWideStringFieldSize

which turns the logic around. Adding this option means reserve a "perfect fit" buffer, whereas missing the option (default) should align TWideStringField.Size to Count of Characters declared in your DataBase.

Patch done R3661 \testing-7.2 (svn)

Note i didn't had the time to test it please report success or other issues related to this thread..
It might be possible i need to fix the meta-informations too.

Just waitung for the next Ticket of a broken HEAP-mem..

Cheers, Michael

[louis]

so i did add a TZDataSet.Options enum

Code: Select all

doAlignMaxRequiredWideStringFieldSize

which turns the logic around. Adding this option means reserve a "perfect fit" buffer, whereas missing the option (default) should align TWideStringField.Size to Count of Characters declared in your DataBase.

Patch done R3661 \testing-7.2 (svn)

Note i didn't had the time to test it please report success or other issues related to this thread..

Not success I tried with doAlignMaxRequiredWideStringFieldSize and doNoAlignDisplayWidth but is the same:
a) The size is boubled;
b) The DislpaySize is doubled;
c) The Error on Post is the same: -303.

Thanks.

[marsupilami]

Hello Louis,

did you update (checkout again) your copy of Zeos from the SVN repository?
Best regards,

Jan