I need help with Pattern language. Noob question.

[pmelnik]

Hello!
First of all, I what to say thank say thanks to creator of this editor. This is best hex editor I have find.
I feel the power, but falls with create my pattern code. :)

My task is find in binary data something that starts with 0xa015 (16b LE)(id) next 2 byte is length and next is (16b data * length) and colorize that and only that.

So i try to create the pattern code:

`
struct hdt {
u16 id;
if (id == 0xA015) {
u16 len;
u16 data[len];
}
}

hdt data[while (!std::mem::eof())] @ 0x0

`

And I've got pattern data every 2 byte and beautiful color, but not there i need it to be. It's even better than other hex editors, but I understand that it can do more. :)
If it were python, i have to open file for read, and in for loop find(if) needed byte seq, then save it to list, tuple or whatever.
I check the existed patterns and documentation, try find something in google but didn't understand the logic of this script language.
My main misunderstanding is how to process data without creating struct.
I can't find docs and tutorials other than official.

It was first step, on next step, that i need to change code to find more than 1000 different named ID's. But first things fist.

Can I import "Highlighting rules"? I find this rules in project file, but didn't find import/export function.

Thank you

[david-c14]

You could try something like this:

import std.mem;

struct hdt {
  $ = std::mem::find_sequence(0, 0x15, 0x0a);
  le u16 id;
  le u16 len;
  le u16 data[len];
};

hdt i @0;

The structure is placed at 0, searches for the address of the first occurrence of the byte sequence 0x15 0xa0 and gets the address. Sets the cursor to that address and then places the id, len, and data array accordingly.

If you don't want to set the cursor, then you could place the three items in the structure explicitly instead.

import std.mem;

struct hdt {
  auto addr = std::mem::find_sequence(0, 0x15, 0x0a);
  le u16 id @addr;
  le u16 len @addr + 2;
  le u16 data[len] @addr + 4;
};

hdt i @0;

The first approach will give you an hdt pattern which starts at 0 and has a length that takes it to the end of the data array.

The second approach will give you an hdt patter which starts at 0 and ends at 0, but the component parts are still in the correct places.

[paxcut]

The last code I posted does have a problem in that it creates an empty element at the end of the array. This can be solved by moving the break statement to the end of Finder. That presents also a problem if the input file has no matching sequences so I print an error in that case, but it still will create an empty element but only in the case that there are no matching sequences.
The code could be further optimized so that it doesn't search sequences twice for each entry. Create a global variable to store the found address and prime it before placing the array.

import std.mem;
import std.io;

s128 address = 0;

struct Hdt {
    u16 id;
    u16 len;
    u16 data[len];
};

struct Finder {
    $ = address;
    Hdt hdt;
    address = std::mem::find_sequence_in_range(0,$, sizeof($), 0x15, 0xA0);   
    if (address < 0)
        break;
};

address = std::mem::find_sequence_in_range(0,0, sizeof($), 0x15, 0xA0);   
if (address < 0) {
    std::print("Error: input file doesn't contain valid id entries");
    return;
} 

Finder data[while (true)] @ 0x0;

[paxcut]

So I implemented the final optimizations and updated the last code posted to the final (hopefully) version which now creates no empty arrays ever and only searches for valid sequences once per found entry.

[pmelnik]

Same error. Even on part of binary file... I've check more than twice.

There is no way find sequence with "IF" operator instead of "find_sequence"?
Seems, like I can't convert it to what I need at final.

[paxcut]

That's not the same error that was showing when a matching sequence is part of the data because it doesn't search in the data part of found matches anymore. If a matching sequence appears that is not an id and is not in the data of a matching sequence then there has to be some condition that will invalidate it. Otherwise the format is defined incorrectly and it is impossible to parse.

[paxcut]

Your original specification stated:

My task is find in binary data something that starts with 0xa015 (16b LE)(id) next 2 byte is length and next is (16b data * length) and colorize that and only that.

that seems to imply that an input file like
C3 C3 15 A0 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3 C3
will never be encountered. Otherwise you need to define how every case that one finds 0xA015 is to be handled. Without a full specification of the problem finding solutions is almost impossible.

[pmelnik]

I didn't see different error. Only this.

Thank you, anyway!
I understand that it is not that easy as I thought.

Is it possible to import "Highlighting rules"?

[paxcut]

The same error message may be caused by separate issues. In this case the problem is that there is not sufficient information to define patterns that would apply to all possible inputs. Maybe if you explained how you would accomplish the goal you are pursuing using python or other languages we may be able to help you. I am not 100% sure but I think that highlighting rules are not part of the pattern language and cannot be imported into pattern. They are a feature of the hex editor and they must be defined in the hex editor ui.

[paxcut]

Changing Hdt to

struct Hdt {
    u16 id;
    u16 len;
    if ($+len < sizeof($))
        u16 data[len];
};

will prevent that error from ever occurring.

[pmelnik]

Great! Now it is work as expected. Thanks.

Binary file is the engine firmware. I need to know were is laying maps. I now some map parameters and some info from similar, but not identical map. So i is good to highlight info that we know to find maps. ID is parameter is used as axis. Data is parameter value on axis. Mostly there is two axis. I have 1090 parameters in this binary. Not all will be used as axis, but i didn't know which has to be.

I can write some algorithm in python to find something like axis in bin. But in imHex I also can edit map. So I spend a whole day to adjust imHex to my need.

That's all.