First CVR mod!

[PlotinusRedux]

You meaning the lighting table the normals are used as in index into?  I had expected to find an actual normal look-up table (i.e., the indexes would give you an X, Y, Z normal), but instead there's just the table that gives the lighting value using the normal + viewXlightsource as an index.

I'll try to post it tonight after work.

Ultimately I'd want to replace that table look up with actual lighting calculations based on the extracted X, Y, Z normal from my code fragment above, in order to get shading levels finer than the current 0-23 system, to go along with a 32-bit color conversion also replacing the 8-bit palette indexes.

[Ford_Prefect]

You meaning the lighting table the normals are used as in index into?

No...

What I meant to ask is how is cvrtest coming up with x,y,z ... and then I realized that you already had (I'm feeling rather stupid right now. ).   

[PlotinusRedux]

Oh, yeah, it's just using that function I posted, where the "n" parameter being passed in = (( D & 7 ) << 8 ) | N if you label the 3 bytes of each voxel as D, N, P, and with "PI" just being a #define I made for the value of pi.

If you want to do the opposite and *encode* a normal--i.e., convert a standard X, Y, Z normal into their format to store in a .cvr file to modify the existing normals, make a new mesh, or whatever, it should be:

Code: [Select]

#define PI 3.141592653589793238463

DWORD EncodeNormal(double x, double y, double z)
{
if (y < 0.0)
{
x = -x;
y = -y;
z = -z;
}

double dAngle1 = atan2(z, x);
if (dAngle1 < 0.0)
dAngle1 += 2 * PI;

DWORD dwAngle1 = (DWORD)round(dAngle1 / (2.0 * PI) * 89.0) % 89;
dwAngle1 = 88 - dwAngle1;

double dAngle2 = acos(y);

DWORD dwAngle2 = (DWORD)round(dAngle2 / (PI / 2.0) * 23.0);

DWORD dwRet = dwAngle2 * 89 + dwAngle1;

// This works because in the case of y = 1.0, x & z will be 0,
// so 23 * 89 is the highest number we need
dwRet = (dwRet >= 89) ? dwRet - 88 : 0;

return dwRet;
}

Which is what I saw the code basically doing to the lighting vector except for the part about subtracting 88 at the end since internally it doesn't need to fit it into 11 bits, and from which I derived DecodeNormal() as it's inverse.

[PlotinusRedux]

Ford--

I ran any analysis of all the tags in all the .cvr files to get an idea of the tree structure and see what tags we actually need to worry about.  It's attached here.  It looks like we can ignore the following tags because they aren't actually used in any of the SMACX .cvr files:

   CVRCHUNK_VOXEL_OBJECT_SIZE      = 0x04040202,
   CVRCHUNK_LINK_TREE_CONTAINER   = 0x04040400,
   CVRCHUNK_LINK_FLAGS         = 0x04040401,
   CVRCHUNK_OBJECT_PARENT         = 0x04040402,
   CVRCHUNK_OBJECT_PIVOT         = 0x04040403,
   CVRCHUNK_AUX_OBJECT_CONTAINER   = 0x04050000,
   CVRCHUNK_AUX_OBJECT_ID         = 0x04050001,
   CVRCHUNK_AUX_OBJECT         = 0x04050002

I believe I've figured out the material chunk, which was the last tag we hadn't deciphered.  It appears to be a 16-bit color (555) version of the shading tables to complement the 8-bit paletted version I identified earlier, and consists of a 4 byte integer followed by 24*256 16 bit colors.  It gives a bit finer shading than the 8-bit table, which you can see in CVRPlay.exe by switching the "Mode" between "High Color" and "Index Color".  It's not a huge difference, but it does look slightly better, and would be a superior input to a Gaussian shader.  The CVRPlay code can handle 555 or 565 16-bit color, and there's probably a bit somewhere in the file saying which to use, but all the ones I sight-checked the color value tops out at 0x7FFF, pointing to 555 since the highest bit is always 0.

[Ford_Prefect]

It's attached here. 

  Ummm.   Where?

Great to know that there is a higher quality color source I can use.  That will come in real handy when I make a cvr viewer for JAC.  Sounds like the only information needed for exporting is the positioning data for voxel meshes.  (See the terraformer model, Vwntu.cvr, which has the arm, drill, body as separate pieces)

[PlotinusRedux]

D'oh, it's attached now.

The general file format seems to go:

VW14.cvr
CVRCHUNK_FILE_CONTAINER 00000000 0003dce8
    CVRCHUNK_VERSION 00000008 0000000c
    CVRCHUNK_DB_NAME "SMACX_FINAL.max"  00000014 00000017
    CVRCHUNK_PALETTE_CONTAINER 0000002B 00004b2d
        CVRCHUNK_PALETTE_NAME "cavcols"  00000033 0000000f
        CVRCHUNK_PALETTE_DATA 00000042 00001b0a
        CVRCHUNK_MATERIAL 00001B4C 0000300c
    CVRCHUNK_SCENE_CONTAINER 00004B58 00039190
        CVRCHUNK_SCENE_NAME "SMACX_FINAL.max"  00004B60 00000017
        CVRCHUNK_OBJECT_COUNTER (40)  00004B77 0000000c
        CVRCHUNK_FRAME_COUNTER (15)  00004B83 0000000c
        CVRCHUNK_OBJECT_CONTAINER 00004B8F 00002fd3
            CVRCHUNK_OBJECT_NAME "Barrel Brace 01"  00004B97 00000017
            CVRCHUNK_GEOMETRY_CONTAINER 00004BAE 00002cb5
                CVRCHUNK_VOXEL_OBJECT 00004BB6 00002cad
            CVRCHUNK_ANIMATION_CONTAINER 00007863 000002ff
                CVRCHUNK_OBJECT_FLAG 0000786B 00000017
                CVRCHUNK_OBJECT_LOCATION 00007882 000000bc
                CVRCHUNK_OBJECT_MATRIX 0000793E 00000224
        CVRCHUNK_OBJECT_CONTAINER 00007B62 00000cad
            CVRCHUNK_OBJECT_NAME "Barrel02"  00007B6A 00000010
            CVRCHUNK_GEOMETRY_CONTAINER 00007B7A 00000996
                CVRCHUNK_VOXEL_OBJECT 00007B82 0000098e
            CVRCHUNK_ANIMATION_CONTAINER 00008510 000002ff
                CVRCHUNK_OBJECT_FLAG 00008518 00000017
                CVRCHUNK_OBJECT_LOCATION 0000852F 000000bc
                CVRCHUNK_OBJECT_MATRIX 000085EB 00000224
        CVRCHUNK_OBJECT_CONTAINER 0000880F 00000b09
            CVRCHUNK_OBJECT_NAME "Barrel03"  00008817 00000010
            CVRCHUNK_GEOMETRY_CONTAINER 00008827 000007f2
                CVRCHUNK_VOXEL_OBJECT 0000882F 000007ea
            CVRCHUNK_ANIMATION_CONTAINER 00009019 000002ff
                CVRCHUNK_OBJECT_FLAG 00009021 00000017
                CVRCHUNK_OBJECT_LOCATION 00009038 000000bc
                CVRCHUNK_OBJECT_MATRIX 000090F4 00000224
...

FLAG, LOCATION, and MATRIX are included for each mesh separately, and are arrays of length CVRCHUNK_FRAME_COUNTER.  So if you take the first location of each array and apply the first matrix of each array, that should align the meshes properly.  I don't know what all is in the flags--it may be as simple as a 1 to show or hide the mesh in each frame.

I'm working on a full file parser in C++ I'll post when it's done sometime this weekend.  You'd mentioned maybe re-rewriting your editor in a language other than python--were you thinking C++, Java, or something else?

[Ford_Prefect]

I was thinking doing it in Java since JAC is in Java (I'll be able to reuse the code).   In fact, I already created a voxel viewer in Java for reading the export files CVRColorizer produces (I didn't release it because it couldn't show the normals). So that means I  just need to write a cvr file reader and a gui.

[Ford_Prefect]

Ack.  I just realized the toolkit i used wont work in a canvas element. (Can't use swing).  So I might need to redo the renderer as well.  Oh well.  I'll save analysis of the different toolkits for a latter date.

[PlotinusRedux]

Ford--I've attached a write up of the .cvr file format, just for our use.  Once we get everything confirmed and nailed down, you can update wiki entry, using my document or your own write up as you wish--this is just a WIP while we nail down the last parts of the file.

New since my last report are:

First 4 bytes of Voxel header is a flag--
flag & 2 means a jump header follows and each voxel group will have another jump header, otherwise there is no jump header (answering a question you had asked in your cvr file write up);
flag & 8 does something in the code, but I haven't figured out what exactly yet;
flag & 1 is true in only 1 file, VLIGHTS.CVR, and it means instead of each jump group having a jump header, each group has a voxel header without a jump header.  I.e., the 32 bytes at the start of the voxel data repeat inside the voxel data the way the jump headers do in other files.

The last 4 bytes of the jump header is actually a flag.  It may be 0 in all files, but if it is ever 1, it means an extra 4-byte integer will follow the header giving the byte offset of the next jump header.

Offset 8 in the voxel header is a 4 byte float giving the "units per 3d pixel".

The CVRCHUNK_OBJECT_LOCATION array in the animation area is an array[FrameCount] of VECTOR3's, VECTOR3 being defined as 4-byte float Z, 4-byte float X, 4-byte float Y.

The CVRCHUNK_OBJECT_MATRIX array in the animation area is an array[FrameCount] of 3x3 Matrices of 4-byte float, in order to multiple against [Z,X,Y] VECTOR3's.

Aligning multiple objects (meshes) should be a matter of placing their origin at CVRCHUNK_OBJECT_LOCATION[0] * CVRCHUNK_OBJECT_MATRIX[0].

I'm not happy with definition of the voxel jump headers.  They contain 4 SHORTVECTOR3's (call them v1, v2, v3, and v4), and v1 + v4 does seem to give the start position as you use in CVRColorizer.  But that doesn't explain why they are separate vectors, or what 2 and 3 are for.  I'm looking at some code that uses all 4, that seems to be determining the boundaries of each voxel group, but I haven't reverse compiled it well enough to really understand what it's doing yet.  Oddly, the first thing it does is compute 2 vectors A and B as A = v1 + v4 and B as v2 - v4, and it passes (B, v3) to the same routine it passes from the voxel header itself (Position, Scale).  But v3 does not seem to be a scale, and v2-v4 doesn't seem to be a meaningful position.  If it really is just computing boundaries, it's replaceable with the method you used of actually rendering the voxels to find the boundaries, and changing v2 and v3 seems to have no effect on cvrplay.exe, but it bugs me that I don't understand what they are.

Let me know you have any thoughts on what v2 and v3 might be.

[Ford_Prefect]

I'll find a stopping spot in my current work on JAC and start working on CVR editor 2.

Let me know you have any thoughts on what v2 and v3 might be.

  No clue.  If I come up with an idea, I'll tell you.

[PlotinusRedux]

How committed to Java are you, Ford?  I just have a fundamental dislike for interpreted languages and things that happen "behind the coder's back" like Java's garbage collection, and I've never seen a really smooth Java game under Windows.  C++ using OpenGL can easily be coded to cross-compile for Linux if that's your concern.

[Ford_Prefect]

How committed to Java are you, Ford?

It's mostly because I'm used to it and I thought I would be doing most of the coding. 

I've never seen a really smooth Java game under Windows

I have.  Blocks that matter, Far Sky, Minecraft.


A side note:  I'm pretty sure it would be legally questionable in several countries if you (and the other exe decompilers) were to contribute code.  I'm wanting the work I'm doing to be completely legal in case EA ever notices it.  Which means following http://en.wikipedia.org/wiki/Clean_room_design.

[PlotinusRedux]

Interesting, I hadn't heard of that clean room thing before.

As long as any mod/rewrite is (1) free and (2) requires SMACX to be installed, I'd be very surprised if Firaxis would have an issue with it--they've always encouraged modding, and if you still have to buy their game, no financial harm is done to them.

For a rewrite to be completely free of any liability to Firaxis (or whoever owns the copyright today), though, I think you'd have to not use their artwork.  Game rules are explicitly not copyright-able, but images definitely are--even more so than code.

[Ford_Prefect]

Yeah.  I'm not worried about Frixas, but EA owns the game and they are not known for being nice.  So I'm doing it by the book.  Its why writing JAC takes so much effort is because instead of just programing it to be the same, I'm making it read all the config files and dialog, etc from SMAC.

Also, I've almost got a cvr viewer ready.  I think its reading in all the data correctly, I just need to hook it up to a renderer and see if it actually produces the models correctly.