Decode/Encode Messages - Little Endian/Big Endian

[HpLightcorner]

Hey!

Just some Key Words: CAN-DBC File, Bit and Byte Order, Bitstruct, Encoding, Decoding

I came across some issues while using the cncoding and decoding features. I got some data from an ECU in Intel format and the values where totally wrong. After debugging canmatrix I found out that format string for Bitstruct is inverting the Bit-Order when the DBC file is using Intel format. My understanding of the DBC setting (0 for Motorola, 1 for Intel) is that I set the Byte Order here, not the Bit order? Also my test equipment with a dedicated Vector Lib is interpreting this as the Byte Order as far as I tested it till now.

So, acutally I am not sure if I found a bug or a feature. Bitstruct allows to reverse both, the byte and bit order. So if it's a bug i would start a pull request on my changes, otherwise i hope someone can explain me why it is useful to cange the bit-order. Actually i cannot see any advantage here when interpreting data, it's all about Byte Order, or isn't it?

Thanks!

[HpLightcorner]

Just to inform you what i did to get correct data from my interface:

• Introduced a local byte-order in bitstruct (suffix instead of prefix at the signal) for pack and unpack function

def _parse_format(fmt):
    if fmt[-1] in [">", "<"]:
        byte_order = fmt[-1]
        fmt = fmt[:-1]
    else:
        byte_order = ">"

    parsed_infos = re.findall(r'([<>]?)([a-zA-Z])(\d+)([<>]?)', fmt)

    # Use big endian as default and use the endianness of the previous
    # value if none is given for the current value.
    infos = []
    endianness = ">"
    local_byteorder = ""

    for info in parsed_infos:
        if info[0] != "":
            endianness = info[0]

        if info[3] != "":
            local_byteorder = info[3]

        infos.append((info[1], int(info[2]), endianness, local_byteorder))

    return infos, byte_order

• Changed the signal bitstruct_format string in canmatrix file by simply using the new local byte-order instead of bit-endianess (suffix instead of prefix)

    def bitstruct_format(self):
        """Get the bit struct format for this signal

        :return: str        """
        endian = '<' if self.is_little_endian else '>'
        if self.is_float:
            bit_type = 'f'
        else:
            bit_type = 's' if self.is_signed else 'u'

        return bit_type + str(self.signalsize) + endian

Maybe it's not the best idea to add some code to bitstruct.... But it's just a first attempt to verify functionality. Looking forward to get some feedback if I got something totally wrong.

[ebroecker]

You are completely right:
Dbc talks about byte order not bit order (there is only one wiki-article in this project: https://github.com/ebroecker/canmatrix/wiki/signal-Byteorder)

you found a Bug.

by the way, the decoding/encoding-feature is quite new in canmatrix.

a pull-request will be very wellcome (please against developement-branch)!

Thanx in advance.

[HpLightcorner]

Okay - i just did some tests now with a Intel and a Motorola source with the changes i did in camatrix.py and bitstruct.py - looks good so far.

I will fix the bug and start a pull request, however, what's your suggestion? Should i contribute to bitstruct to add the needed feature or should i keep all the changes in camatrix?

[altendky]

In case it's of any use, here's my unpacking code.

https://github.com/altendky/st/blob/c75f436f503adcb4708131a65b059c8062484ae5/epyqlib/canneo.py#L598

Lots of mess in that file that needs to be rewritten but the unpacking does handle big and little endian last I checked as well as overlapping signals (we have some signals that contain multiple other signals. Basically a bunch of bits also available as a single int). I'm still using my branch of bitstruct but I don't think I'm using the changes I made.

[ebroecker]

@HpLightcorner
maybe you could do a "best of" your solution and altendky's solution.

Anyway I could imagine, that contributing to bitstruct is not so easy/fast. So I have no problem with adding this functionality direct to canmatrix.

A further approach could be to reverse the byte-array, recalc the startbit and use the motorola-decoding of bitstruct.

[HpLightcorner]

I will go through the code the next days - my tests yesterday were all succesfull. As soon as I am ready I will start the pull request against development so we can start to get into discussion about the best solution. Thanks @altendky for your approach, I will take it under consideration to get the best solution!

[ebroecker]

@HpLightcorner:
Do you plan any activities for this issue?
Else I'd have to fix this issue myself :-)

[HpLightcorner]

@ebroecker - sorry for the delay :)

Still planning to contribute my solution, i just had some busy weeks... I will clean up my code and make a pull request till end of the week!

[HpLightcorner]

@ebroecker - I still like the idea of modifying the bitstruct implementation by adding a byte order per value. There are just some minor changes necessary and the implementation with format strings is simple. In addition we could also add further improvements to the bitstruct implementation if necessary like half precision floats? I removed the global byte order as the can file allows to implement a byte order per value anyway.

I added bitstruct directly to my local canmatrix repository, renaming it canmatrix_bitstruct. Contributing is a quite new "activity" for me, so what's the best way now? Should i just add my modified file of bitstruct?

[HpLightcorner]

@ebroecker and again... I was not able to find out if test_codec.py is used? If so, I would add some tests here.

[ebroecker]

First of all, its nice to hear, that you are still planning to help canmatrix, thanx in advance.

I think your approach is good:
put a copy of bitstruct in canmatrix with a name like "canmatrix_bitstruct".
Do the relevant changes here.
so yes: just add your modified version

Maybe afterwards it would be good to contribute your bitstruct-changes to bitstruct also.

till now test_codec is only executed manually. So just add/modify your tests there...

[ebroecker]

by the way, did you see https://github.com/eerimoq/cantools
It can do several decodings just using bitstruct?
I stumbled over it while looking at bitstruct (https://github.com/eerimoq/bitstruct)
https://github.com/eerimoq/bitstruct/blob/master/bitstruct.py#L313 seems to support byte-order?

[altendky]

eerimoq/bitstruct#1

There's my attempt to get a request accepted for, I think, the same thing. I'm pretty sure I was wrong about it being needed. But it does sound like I should check what those cantools have.

[HpLightcorner]

okay, plenty of options now...

@ebroecker I had a look at cantools, also a good approach. To be honest, I have to go into details reguarding multiplexing if staying with the original bitstruct is the preferred way. cantools is decoding little and big endian indepent, so it's possible to stay with original bitstruct syntax - I think this would also be possible for the canmatrix approach, but for me just with further investigation... I have never used multiplexed signals before, so befor i crash the functionality I will have figure it out how this works

@altendky and @ebroecker I also hab a look at the bitstruct discussion. I changed somehow the syntax of the fmt string, there is now a seperator between each formating info which allows a byte order and bit order per value, something like

|p4||s5<|p1|f16 and so on...

For floats, signed and unsigned it's necessary to add bit and byte order information.

This will allow to encode and decode little and big endian at the same time, which will keep the canmatrix and the adapted bitstruct code quite clean - at least I hope so.

[HpLightcorner]

I uploaded my changes to my forked repo, I would appreciate it to get some feedback from your side!
cef22fa

Just so that I got everything correct now - a short example - taken from the test.dbc file with some minor changes:

BO_ 291 testFrame1: 8 testBU
SG_ someTestSignal : 3|11@0+ (1,0) [0.0|500.0] "specialCharUnit�$" recBU
SG_ Signal : 20|3@1+ (1,0) [0.0|6.0] "someUnit" recBU

someTestSignal now has gain 1 and offset 0.

Assume:
data = { 'Signal': 2, 'someTestSignal': 101, }

We should get:
0x01, 0x94, 0x04, 0, 0, 0, 0, 0 for intel on both signals (1+)
0x05, 0x0C, 0x04, 0, 0, 0, 0, 0 for motorola on someTestSignal and Intel on Signal

Just to make sure that i got the 11 bit special case now correct...

[ebroecker]

I wrote a comment to:
cef22fa

Your could use https://github.com/ebroecker/SocketCandecodeSignals for double-check values - at least nobody complained that SocketCandecodeSignals caluclates values wrong :-)

[HpLightcorner]

@ebroecker did some checks now with my data, I will add some tests based on your link and commit them as well, pull request can be expected at the end of the week :)

[HpLightcorner]

Hey!

Still missing the test files, but i commited today some improvements after I had a chance to test my Software with an inverter. Acutally the LSB part now is working properly, Motorla tests are open.

So, I will try to manage adding the missing test stuff the next days.

[ebroecker]

Hi - what is your current state - do you have something to merge?

[HpLightcorner]

Hi!

Sry for not starting a pull request till now, i actually finished my academic studies so I forgot about the issue. I still could start a pull request from my forked development branch. Has someone found an alternative solution?

[ebroecker]

Hi HpLightcorner,

nice to hear from you!
Your pull request would be welcome.
We have still no other solution..

[ebroecker]

could it be that meanwhile bitstruct supports what we need?

eerimoq/bitstruct@24a5d3a

[HpLightcorner]

Hey!

I wrote some test cases the last week - floats are now working for me but somehow i got errors with integers now - start bit is vice versa and shifted. At the moment i am not sure if this which is the correct way, so i organized some additional information. I will get some can traces from a commercial available device and will use it in for test cases. Has someone else proofed dbc files and can traces as well?

When bitstruct now Supports the features, is it better to stay with it?

[HpLightcorner]

Okay, looked through bitstruct. Would work now if there is no change within a can message.

For example: two floats in one message, One Motorla, one intel won't work as far as i can tell now.

Not sure if this is a case we must consider?

[ebroecker]

cantools do decode the whole message like this.

https://github.com/eerimoq/cantools/blob/master/cantools/db/message.py#L86-L108

maybe one can 'lend' some ideas

[ebroecker]

there is a dbc from c library to de/encode can frames:

https://github.com/ebroecker/SocketCandecodeSignals/blob/master/ccl_test.dbc

001#8d00100100820100
002#0C00057003CD1F83
you can find decoded output for these messages here:
https://github.com/ebroecker/SocketCandecodeSignals

Maybe this could be a good startingpoint for testing...

[mgiaco]

Any news here? We also found that bug right now.

[HpLightcorner]

To be honest, i am using cantools now for decoding/encoding with DBC files. I tried to find a proper solution using canmatrix, but it took me a lot of time to find all related parts in the code. So, to get my project up and running cantools was the fastest solution. I also tried to get the cantools encoding/decoding to work with canmatrix, but again, did not have the time to finish till now. Sorry!

[mgiaco]

Okay many thanks.

[mgiaco]

@ebroecker please help here because that is a real problem and I am not sure if I can help here. I can only help testing.

[ebroecker]

Seems @HpLightcorner has No Time anymore for cleaning up this issue.

As I have to implement a solution now:
@altenkey: I tend to switch to your solution. Is it OK, to use your code you linked above?

[ebroecker]

fixed with merge of #224