First CVR mod!

[Ford_Prefect]

Forum for the python based editor I wrote.
http://alphacentauri2.info/index.php?topic=7866.0

Source code:
https://github.com/uaEquals42/CVR-Colorizer

[PlotinusRedux]

Ford--

After ruling out a lot of possible formats, I'm convinced the 11 bits are an index into a hard-coded array of normals, so I'm now reverse engineering the code to locate it.  I haven't found it yet, but I've found some general information on the file format from the possible error messages you might be interested in.

The entire file is a series of chunks, consisting of:

4 byte Chunk Type
4 byte Chunk Size (including the Type above)
(Chunk Size - 8 bytes) bytes of data
Next Chunk...

Each chunk is immediately followed by another chunk with no alignment.

Some chunk types are containers, whose data consists of child chunks.  For instance, the Palette Container chunk can contain a Palette Name chunk, a Palette Entry chunk, and 0+ Material chunks.

You had wondered in one place how the code knew if some data was there or not--it's just a matter of it if that chunk type appears in the chain of chunks or not.  A number of the chunks are optional and/or can be present multiple times.

The chunk types are:

00525643  Main Container (Contains the entire file)
01000000  Version
02000000  Database Name

03000000  Palette Container
01010000  Palette Name
01020000  Palette Entry (0x1B02 of data)
01030000  Material (0x3004 of data)

04000000  Scene Container
04010000  Scene Name
04020000  Object Counter
04030000  Frame Counter

04040000  Object Container
04040100  Object Name

04040200  Geometry Container
04040201  Voxel Object (0x2A Header + Voxel data)
04040202  Voxel Object Size

04040300  Animation Container
04040301  Object Flag
04040302  Object Location
04040303  Object Matrix

04040400  Link Tree Container
04040401  Link Flags
04040402  Object Parent
04040403  Object Pivot

04050000  Aux Object Container
04050001  Aux Object ID
04050002  Aux Object (0x2A Header + Aux data)

So, for example, the beginning of the file is interpreted:

00000000  00525643 = Main Container Chunk
00000004  00001A76 = Main Container Chunk Size = Size of file
00000008  01000000 = Version Chunk
0000000C  0000000C = Version Chunk Size 
00000010  41200000 = Version
00000014  02000000 = Database name
00000018  00000015 = Database name Size
0000001C  "ACAC-WEP.max"
00000029  03000000 = Palette Container
0000002D 00004B2D = Palette Container Size
00000031  01010000 = Palette Name
...

[Ford_Prefect]

Thats pretty cool.

I'm glad someone who knows how to read assembly is working on this.   

If you figure the normals and where the heck the rest of the color palette is at, I'll pause my JAC project and work on an updated mod tool.

I was thinking of writing code that would try to calculate the normals by grouping voxels by their slope (11 bit key) and then calculating the slope from those groupings.  But you finding the lookup values would be allot more percise. (and there wouldn't be any missing values)  I don't envy you trying to figure out 3d code in assembly. 

[PlotinusRedux]

What were you missing from the palette specifically, so I can be on the look out for it?

The files I'm looking at (sp-marined.cvr and vr03.cvr) have 0A and F5 at the start of the palette info, which I take to mean the valid palette indexes run from 0A to F5.  It's followed by 256 3 byte runs of which the first 10 and last 10 are all 0's, those being the ones outside the 0A to F5 range.  Were you seeing color indexes in the voxel data outside that range?

That is followed by 24 tables of 256 bytes I assume have something to do with palette stepping--they don't have a separate chunk type id and are included in the palette entry chunk's size.  I haven't looked into exactly how they work yet.

The material chunks are also in the Palette container and have 24 tables of 256 words it looks like--how the material actually works I haven't looked at yet.

[PlotinusRedux]

I've been staring at numbers too long.

2^11 = 2048 normal indexes * 3 bytes per normal = 6144 bytes for the normal table

There are 6144 bytes left in the palette entry chunk after the last actual color entry.  But, they're in 256 * 24 blocks with a run of 20 0's at the end of each block, exactly the number of unused colors indexes in the palette.  So I know what has to be palette stepping of some sort rather than the normal tables.  But still, that coincidence of both being 6144 bytes is taunting me....

[Ford_Prefect]

Color pallete.  What I did to get it to work mostly is weird...   

Code: [Select]

0A F5       ????? A = 10.  F5 = 245   10+245 = 255 
                For reading in the values, I had to move 10 forward before starting to count.
                245 for 245 values used?
    zz F5 00 00 00 FC 00 FC FC 00 FC FC 00 FC FC 00 FC FC 00 FC FC 00 FC FC 00 FC FC 00 FC FC 00 FC....
    zz F5 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 AC 80 50 A0 7C .... Yeah....
    Ok.  I've learned this:
        zz*3 (usually 1*3 or 10*3) tells how far to right to being start counting up from 0 for color numbers.
        F5 (245) is the length of this sections data. 
        so values greater than 245 - zz. are fetched by some other method. 
        Colors are stored as R G B values.  1 byte each.
Basically, there are allot of 00 00 00 or other junk data at the start of the pallete that you skip using the first number and the second number tells you how many colors there are that you can use.   The problem is, there are color codes greater than 245 that are being used.  If I try to go off the end and keep in reading values past the 245 rgb values, I get values that are incorrect.  (For example, the terraformer isn't yellow if I try that)

I had no idea about a materials system being in the file.  Maybe its getting all the color values for those items from that instead. 

EDIT: Fixed the stupid mistake I made with the numbers. 

[PlotinusRedux]

Ok, here's how I think it works.  (BTW, I think you meant F5 = 245, 10+245=255, etc.)

0A F5
The 0A is the first palette index actually used, and the F5 is the last palette index actually used.

The physical palette itself starts immediately after the 0A F5, not 3*10 bytes later, and is always exactly 256*3 bytes long.  It's just the first 10 entries aren't used, so they're 0's.  But you still need to count them--so the first actual color @3*10 is palette index 10, not palette index 0.  By treating it as palette index 0, you're offsetting your entire palette by 10, and that's why values like F5 appear to overrun your palette.

So in terms of your comments--
zz tells the first valid color index, not how far to the right to start counting--you always start counting right after the F5.

Separate from that, I think I've figure out how the palette stepping for shading works.

Immediately after the palette entries themselves, there are 24 sets of 256 bytes.  Consider them as an array Shades[24][256] of byte.  The first index is the light level, with 0 being darkest, 23 being lightest, and 11 being neutral lighting.  The 2nd index is the palette index of the color your shading minus zz from your comments above (usually 10, occasionally 1).  The byte pointed to by those indexes is the palette index for that color at that level of lighting.

So take sp-marined.cvr for example.  It has zz = 0x0A = 10.  Almost all the voxels in it have palette index 0x46=70.  So to find that color's neutral shaded color, Shades[11][70-10=60], which has the value 0x45=69.  So voxels with palette index 0x46 actually use palette index 0x45 at neutral lighting.  At Shades[12][60] we have 0x46, so it shows it's true color at just slightly above neutral lighting.

[Ford_Prefect]

Opps.  I meant 246.   
The problem is that 255 - 10 = 245.  And then they go ahead and use 247-255 as a reference number on some of the voxels. 

So if i'm getting it right, the large block of data after the color palette is pre-computed shading data.  Telling it what colors to use when a surface is bright or dark. 

[PlotinusRedux]

Opps.  I meant 246.   
The problem is that 255 - 10 = 245.  And then they go ahead and use 247-255 as a reference number on some of the voxels. 

You're missing what I'm saying--255 is a valid palette index, and all the palettes have 256 entries--even RGB(0,0,0) still counts as an entry, even though it's never used.

The 10 means for that file, none of the palette indexes in the voxels should be less than 10 (except 0, which means translucent).  It doesn't mean to offset the palette by 10.

In CVR.py, change:

Code: [Select]

pos=pos + 1 + CVRfile[pos-1]*3  #10 was the right number for ACP which had 145 for its number. 155-145 = 10
# Color is wrong on select for the knob parts... as well as the drill arm
# It isn't getting the right colors when it is # 145 or higher
inttmp = pos + 3*(245-CVRfile[pos-1]-1)  #it won't be reading in the right values for 155-x... yeah.
colorcount = 0;
#TODO: I think this might be where it is adding one color to many sometimes.... I'm gonna have to pin it down sometime.
while pos <= inttmp:to:

Code: [Select]

pos=pos + 1
inttmp = pos + 3*255
colorcount = 0;
while pos <= inttmp:assuming pos is starting 1 byte past the 0x0A (at the 0xF5).  Unless you using the shading information I found, you can completely ignore both the 0x0A and the 0xF5.

On your other question--yeah, the 24*256 bytes right after the palette is definitely a shading array.  And I've found in the code now where the 11 bit normal index is used to look up a shading value between 0 and 23--which fits with the 24 shading tables.  Now I just need to work out, the normal index is added to an offset computed from the viewing and light angles--once I work out exactly how that offset is calculated I should be able to translate the normal indexes into actual normal values.

[Ford_Prefect]

I tried your suggested code change.

Vwntu.cvr now is yellow.
Acolonypod.cvr has incorrect colors.
Vw12 cvr has incorrect colors
etc.

A side note:  There are at least 2 different version numbers for the cvr files in SMACX.  Didn't know if you noticed that.  There could be a differences between files because of that.

[PlotinusRedux]

So, this has all the colors correct by looking them up in the shading tables in all the files I've tested, including the ones you listed above:

Code: [Select]

# Palette colors
pos += 16
logging.debug("palette num" + str(CVRfile[pos-1]))
pos += 1

# Physical Palette
inttmp = pos + 3 * 255
colorcount = 0;
while pos <= inttmp:
self.physical_palette[colorcount] = str(CVRfile[pos]) + "," + str(CVRfile[pos+1]) +"," + str(CVRfile[pos+2])
colorcount+=1
pos+=3

# Shader Palette
# 12 below is the shade level: 0 = dark 23 = light
pos += 12 * 256
inttmp = pos + 255
colorcount = 0;
while pos <= inttmp:
self.dict_colors[colorcount] = self.physical_palette[CVRfile[pos]]
colorcount += 1
pos += 1

# Body Parts 3D data
...

And added self.physical_palette = {} to __init__()

The only issue is I'm missing some ambient light information that may be encoded in the normal table, or in a header somewhere, so for some of the files all the colors look dim.  That could be offset a bit by adding a slider to adjust the hard-coded 12 shading level above.

[PlotinusRedux]

If this isn't the computation to retrieve the normals, it's close to it.  It's actually quite clever, using exactly 11 bits to store the angles between Z and the X-axis and Y and the X-axis.  Oddly, they seem have normalized to Y > 0 rather than the usual Z > 0, so Y is always positive--they may have internally used a system with the Z and Y axes.

Code: [Select]

void DecodeNormal(DWORD n, double* x, double* y, double* z)
{
    // A1(0-23)*89+A2(0-88)
    // Normally the highest number would be 23 * 89 + 88 = 2136,
    // but because in the case of A1=0, we know Y=1.0, so X=Z=0.0, so A2 = 0,
    // we compressed the range 0-88 down to 0
    // In the case of A1=0, sin(A1)=0, so x & y will be 0 for any value
    // from 0 to 88 below
    n = n + 88;

    // extract the original angles
    // Angle1 = Z to X-axis
    // Angle2 = Y to X-axis
    DWORD dwAngle1 = 88 - n % 89;
    DWORD dwAngle2 = n / 89;

    // convert the angles to radians
    double dAngle1 = dwAngle1 / 88.0 * 2.0 * PI;
    double dAngle2 = dwAngle2 / 23.0 * PI / 2.0;

    // convert the angles to coords in the range [0, -1]
    // Using standard 3d axes, for cavier axes reorder to x, y, z below
    *z = sin(dAngle1) * sin(dAngle2);
    *x = cos(dAngle1) * sin(dAngle2);
    *y = cos(dAngle2);
}

I've attached a program I'm using to test.  At the command line, cvrtest <direction> <normal>, where <direction> is the 1st byte of the voxel and <normal> is the 2nd byte.  The output is x y z.

The program itself doesn't actually decode the normals, it just looks them up in a table, but from what it was doing to the light vector as an offset into that table, the above is how I think they would be decoded, and seems to be working in testing.  I modified your python to show direction and normal instead of painting to input to cvrtest--I'm not a python coder, I'd have to look up how to do the trig with it.

[Ford_Prefect]

Honestly, I'm not much of a python programmer as well.   Writing that program sucked.  I'm seriously thinking of remaking it in another language.  (And since you are figuring out the normals, it could be done in 3d  ) 

Anyways...
Keep up the great work! 

[PlotinusRedux]

I edited the code above to remove a line that did nothing and add a comment about the axes.

Now that I think about it, of course they're using non-standard 3d axes.  They're using Y for left-right, and Z for up-down, and X for front-back, which is why you found their coordinates in front-back (normal Z, their X), left-right (normal X, their Y), top-down (normal Y, their Z) order in the file and wrote it as ZZ XX YY--converting it to standard 3d axes in your interpretation just as I've down in the code above.

That's going to be confusing working with their code, since everything will be in ZXY order instead of XYZ, including their matrices.

It's giving plausible normals for everything I've sight-checked--the next step will be to build a 3d renderer using them and comparing that to cvrplay to make sure they match.

It looks like their renderer makes a pass through the voxels, for each one each one writing to the Z buffer a 16-bit depth, a 1-byte lighting value from 0-23 (looked up from the tables based on viewXlight(converted to the same format as the normals in the file) + normal), and a 1 byte palette code (if the depth of the current voxel < depth value already in the Z buffer, of course).  Then possibly another pass comes through and adds to all the lighting values an ambient light value from 0-23, with the total lighting value clamped at 23.  So only specular and ambient light, no diffuse.  Then finally it reads through the Z-buffer and uses the lighting value * 0x100 + palette code to look up the physical palette code in the shading tables stored in the file, and reads the RGB values from the physical palette table using the physical palette code.

That's why you had apparently missing palette colors, btw--the palette codes in the Voxels are never directly used as indexes into the physical palette--only into the shading tables, which themselves contain the indexes into the physical palette and never use the missing colors.

I still need to interpret the material block I saw in sp-marined.cvr to see how that plays into it all--most of the files I've looked at don't have a material block, but for those that do we should know what it does.

[Ford_Prefect]

Could you post/attach the normal lookuptable?