Firebird max.record length 32Kb

[pbueger]

A simple SELECT * FROM MY_TABLE
Having 2 record size bigger then 32Kb generates an exception because of capacity.

This SELECT works fine when runned out of Zeos component (IBExpert, ISQL,...).

Component used is TZReadOnlyQuery.

[mrLion]

Not only in FireBird. In PostgreSQL also. I`m detect point of error. But dont understand why use 32k limit.

[mrLion]

Error is probably still strategic, contained in the module "ZDBCCashe.pas" in function "constructor TZRowAccessor.Create (ColumnsInfo: TObjectList);"
Yes, and of course version ZEOS 7.0 - latest release from svn repository

Code: Select all

constructor TZRowAccessor.Create(ColumnsInfo: TObjectList);
var
  I: Integer;
  Current: TZColumnInfo;
begin
  FBuffer := nil;
  FColumnCount := ColumnsInfo.Count;
  FColumnsSize := 0;
  SetLength(FColumnNames, FColumnCount);
  SetLength(FColumnCases, FColumnCount);
  SetLength(FColumnTypes, FColumnCount);
  SetLength(FColumnLengths, FColumnCount);
  SetLength(FColumnOffsets, FColumnCount);
  SetLength(FColumnDefaultExpressions, FColumnCount);
  FHasBlobs := False;

  for I := 0 to FColumnCount - 1 do
  begin
    Current := TZColumnInfo(ColumnsInfo[I]);
    FColumnNames[I] := Current.ColumnName;
    FColumnCases[I] := Current.CaseSensitive;
    FColumnTypes[I] := Current.ColumnType;
    FColumnLengths[I] := GetColumnSize(Current);
    FColumnOffsets[I] := FColumnsSize;
    FColumnDefaultExpressions[I] := Current.DefaultExpression;
    Inc(FColumnsSize, FColumnLengths[I] + 1);
    // 32768 is the length of a TByteArray. (HeidiSQL patch)
    if FColumnsSize > 32767 then
    begin
      raise EZSQLException.Create(SRowBufferWidthExceeded);
    end;
    FHasBlobs := FHasBlobs
      or (FColumnTypes[I] in [stAsciiStream, stUnicodeStream, stBinaryStream]);
  end;
  FRowSize := FColumnsSize + RowHeaderSize;
end;

As it became clear from the above package, absolutely everything SQL-query data (or rather one row of the table) stored in one (!!!!????) TByteArray array, which can not be larger than 32 kilobytes!
In this case, it turns out that if the query results nazodyatsya several fields, the sum of sizes which are larger than 32 kb then the library will generate an error.
Well, if I have such a table PostgreSQL 65,535 fields? PostgreSQL allows it, and ZEOS - will collapse.
In my case the query return of about 125 fields Varchar (255) ...
I think solution to the problem would have been refining the mechanism of formation data table row.

[mdaems]

mrLion,
Please feel free to optimize the TZRowAccessor class design. All improvements compilable on the supported compilers and not breaking the test suite are very welcome. (7.X version, not 6.6)

Your analysis is correct, except for the

absolutely everything SQL-query data (or rather one row of the table) stored in one (!!!!????) TByteArray array, which can not be larger than 32 kilobytes!

part. BLOBs are not stored inside this TByteArray.

Another really usefull change in this domain would be a more optimal memory allocation algorythm, allocating memory by more than 1 TByteArray at a time.
The easiest example where this becomes usefull is a program which loops through a table and adds some data to the end of one big string. In that case you'll see a very huge memory fragmentation occurs. After every TByteArray read the growing string will be reallocated and put after the TByteArray because there's no more place to grow it.

So if you refactor, please also consider this issue as well.

Mark

[mrLion]

mdaems,

Your analysis is correct, except for the

I agree with that, but below the text I say that chosen field of type Varchar 255 ... what to do if the problem is exactly like that.
I think that the "temporary and crooked" decision of the forcible removal of a patch from the source (and try to do in my project). Here is the way the question: What is justified type selection TByteArray?
You can also use a dynamic array....oh ... My apologies, it's Pascal! I write in C + +, so it did not immediately thought ...
Well, in any case, it is unclear why the use of the limited size of 32 kb array, because even with "segmented" model of memory segment size - 64 kb. And given the fact that now uses the "flat"model (indicated by as much as you need) a similar approach to the economy - even beyond my comprehension.
By the way, and how to return the data interface library database itself? After all, none of them are not written in Delphi. All of them (well, at least that I know of) written in C. And there is hardly referred to the use of such TByteArray. In any case, EMS Postgre Manager, written using ZEOS 6.x performs these queries correctly.
What does this mean that if you remove the code from the error response, then ZEOS will work properly? But in this case where in the data will be stored? (Somewhere in a pile of garbage?:))
I see what I can offer, but can not guarantee the outcome, since the main effort still at work.

[mdaems]

I have no idea what justifies the use of a TByteArray. It's like that since I started using zeoslib.


Don't laugh about Pascal! Dynamic arrays do exist in pascal too nowadays. It's all but a primitive language and readable at least.

Actual communication with the dll's just happens using standard methods as used in Pascal all the time. It's just a matter of knowing what kind of data should be passed to the library functions and construct a binary identical memory structure. And TByteArrays are quite right to do that als long as the dll doesn't need data chunks bigger than a maximally sized TByteArray.
If you want to check the details, have a look at the actual interface implementation in the Plain layer of zeoslib, eg for mysql, which dll is written in plain C.