[patch_done] About Unicode and TStringField

[mariuszekpl]

When I look in delphi 2009 in db.pas i see:

TStringField = class(TField)
protected
function GetAsString: string; override;
public
property Value: AnsiString read GetAsAnsiString write SetAsAnsiString;

And function GetAsString return GetAsAnsiString

In default TStringField i can't see full unicode
when i use .value or .asString

May by we should replace TStringField with TZStringField and add in TZStringField support for unicode ?

[mariuszekpl]

1) Add new class for string field

Code: Select all

   TZStringField = class (TWideStringField)
   private
    buf : PByte;
   public
    constructor Create(AOwner: TComponent); override;
    destructor Destroy; override;
    class procedure CheckTypeSize(Value: Integer); override;
    function GetAsString: string; override;
    function GetAsVariant: Variant; override;
    function GetValue(var Value: string): Boolean;
    procedure SetAsString(const Value: string); override;
    property Value: UnicodeString read GetAsWideString write SetAsWideString;
   end;

{ TZStringField }

class procedure TZStringField.CheckTypeSize(Value: Integer);
begin
  if (Value < 0) then
    DatabaseError(SInvalidFieldSize);
end;

constructor TZStringField.Create(AOwner: TComponent);
begin
  inherited Create(AOwner);
end;

destructor TZStringField.Destroy;
begin
  if Assigned(Buf) then
    FreeMem(Buf);
  inherited;
end;

function TZStringField.GetAsString: string;
begin
  if not GetValue(Result) then
    Result := '';
end;

function TZStringField.GetAsVariant: Variant;
var
  S: string;
begin
  if GetValue(S) then
    Result := S
  else
    Result := Null;
end;


function TZStringField.GetValue(var Value: string): Boolean;
begin
  if not Assigned(Buf) then
    GetMem(Buf, (Size + 1) * SizeOf(PChar));

  Result := GetData(Buf);
  if Result then
    Value := String(PChar(Buf))

end;

procedure TZStringField.SetAsString(const Value: string);
begin
  SetData(Pointer(Value), False);
end;

2) override function TZAbstractRODataset.GetFieldClass

Code: Select all

{$IFDEF ZEOS_FULL_UNICODE}
function TZAbstractRODataset.GetFieldClass(FieldType: TFieldType): TFieldClass;
begin
  if (FieldType in  [ftString,ftFixedChar,ftWideString])  then
    result := TZStringField
  else
    result := inherited ;
end;
{$ENDIF}

[mariuszekpl]

3) Add support unicode for TZSQLiteResultSet

Code: Select all

  TZSQLiteResultSet = class(TZAbstractResultSet)
...
...
public 
  {$IFDEF ZEOS_FULL_UNICODE}
    function GetUnicodeString(ColumnIndex: Integer): WideString; override;
  {$ENDIF}


{$IFDEF ZEOS_FULL_UNICODE}
function TZSQLiteResultSet.GetUnicodeString(ColumnIndex: Integer): WideString;
var
  Buffer: PAnsiChar;
begin
  Buffer := GetPChar(ColumnIndex);
  if Buffer <> nil then
    Result := UTF8ToUnicodeString(StrPas(Buffer))
  else
    Result := '';
end;
{$ENDIF}

[mariuszekpl]

4) function ConvertSQLiteTypeToSQLType return stUnicodeString for stString field

Code: Select all

  if Result = stString then
    result := stUnicodeString;

[mariuszekpl]

Code: Select all

procedure TForm2.Button4Click(Sender: TObject);
begin
  ZConnection1.Database := 'c:\dane-ŁŁŁĘĘĘĘ-ЫФЫФДЛДЛДЛДкенке.db';
  ZConnection1.Connected := true;

  ZQuery1.SQL.Text:='drop table прапрпра3;' ;
  ZQuery1.ExecSQL;

  ZQuery1.SQL.Text:='create table прапрпра3(id integer NOT NULL PRIMARY KEY,daneЙЦЦУ varchar(11) );' ;
  ZQuery1.ExecSQL;

  ZQuery1.SQL.Text:='INSERT into прапрпра3(id ,daneЙЦЦУ) values ( 101,''dfg1 вапвап'') ';
  ZQuery1.ExecSQL;

  ZQuery1.SQL.Text:='INSERT into прапрпра3(id ,daneЙЦЦУ) values ( 102,''dfg2 вапвап'') ';
  ZQuery1.ExecSQL;

  ZQuery1.SQL.Text:='INSERT into прапрпра3(id ,daneЙЦЦУ) values ( 103,''dfg3 вапвап'') ';
  ZQuery1.ExecSQL;

  ZQuery1.SQL.Text:='select * from прапрпра3;' ;
  zquery1.Open;
  caption := ZQuery1.FieldByName('daneЙЦЦУ').AsString;

   DataSource1.DataSet := zquery1;
end;

[mdaems]

Do I understand well this is a D2009 shortcoming?
Or does D2009 provide a new field_type for unicode-strings?
In that case we shouldn't add our own field type but make sure zeoslib decides to use the D2009 replacement for the (Anis) TStringfield. Isn't it better to use TWideStringField? That one uses a Unicodestring as Value?

Code: Select all

  TWideStringField = class(TStringField)
  protected
    class procedure CheckTypeSize(Value: Integer); override;
    procedure CopyData(Source, Dest: Pointer); override;
    function GetAsAnsiString: AnsiString; override;
    function GetAsString: string; override;
    function GetAsVariant: Variant; override;
    function GetAsWideString: UnicodeString; override;
    function GetDataSize: Integer; override;
    procedure GetText(var Text: string; DisplayText: Boolean); override;
    function GetValue(var Value: UnicodeString): Boolean;
    procedure SetAsAnsiString(const Value: AnsiString); override;
    procedure SetAsString(const Value: string); override;
    procedure SetVarValue(const Value: Variant); override;
    procedure SetAsWideString(const Value: UnicodeString); override;
  public
    constructor Create(AOwner: TComponent); override;
    property Value: UnicodeString read GetAsWideString write SetAsWideString;
  end;

[mariuszekpl]

I can't understund why TstringField.GetAsString return ansistring
but we can't use TstringField if we want add unicode suport.

In new delphi string == widestring we don't need two types: TstringField and TWideStringField

We can use common type for all string field.

[mariuszekpl]

We have two roads:

1) Use one common TZStringField for Unicode and Ansi strings (like IBX)

or

2) User can select what he want , if he has data with unicode he should inform ZEOS about it , and now zeos use internal UNICODE , if we want ANSI zeos internal use ANSI

In my opinion first way is better, we stop thinking about difference STRING / UNICODE STRING, but i can make patch with both

In my opinion internal ZEOS should use Unocode to storage internal data
(in d2009) because it is default string format in delphi 2009

[mdaems]

mariuszekpl,

In a first approach (get D2009 basically supported quickly with almost no impact on the other compilers) I would like a patch like this:

Code: Select all

Index: ZDatasetUtils.pas
===================================================================
--- ZDatasetUtils.pas	(revision 535)
+++ ZDatasetUtils.pas	(working copy)
@@ -301,7 +301,7 @@
     stFloat, stDouble, stBigDecimal:
       Result := ftFloat;
     stString:
-      Result := ftString;
+      Result := {$IFDEF ZEOS_FULL_UNICODE}ftWideString{$ELSE}ftString{$ENDIF};
     stBytes:
       Result := ftBytes;
     stDate:

But I need you to check what extra changes are needed to get this working. (Test suite shows me only changing this causes more errors on string fields)
Could you please have a look at it?

In a later stadium we can talk about adding TZXXXFieldTypes. For the moment Zeoslib only contained TDataset descendants. I'm not sure how adding TField descendants would affect the use of zeoslib datasets in general.

Mark

[mariuszekpl]

Your patch work
BUT
this change is not enough.

We can use TWideString field to display data and it works with unicode and ansiString but this is not enough.

In TZSQLType we have stString, stUnicodeString
If we want use unicode we shoud use stUnicodeString in all place where is stString

[mdaems]

Applied your (a little modified) patch. At least for mysql this passes the test suite without problems, even with Delphi 2009.

SVN rev. 543

[bravecobra]

This patch hasn't been applied to the testing branch as such? I see a

Code: Select all

Result := {$IFDEF DELPHI12_UP}ftWideString{$ELSE}ftString{$ENDIF};

, which no longer offers the choice to the user.
I have code with persistent fields, which now gives an error (ftWideString <> ftString). Shouldn't we be building something that can make the user choose which one to default to.

Does this currently mean I have to replace all my TStringField declarations as TWideStringField??

[mdaems]

Brave,

I don't think so. But I have no real Delphi 2009 experience. So just test and prove I'm right or wrong. As far as I understand the new situations is comparable to the default D2009 field type choice. But if you have an existing application with TStringFields it seems to me like the behaviour would still be the same as before. I think the code you mention above only applies when defining new fields using the IDE or at runtime when opening the query.

Mark