Unknown Message Types #58

spheenik · 2015-09-13T16:49:44Z

Thank you for reporting an unknown message type.

Please note: In case clarity detects an unknown message, it reports this with the warning, but will be able to skip over the message and successfully parse the replay. This means that if you have an exception, there is another problem, and I suggest you open a new issue.

Decoud · 2015-09-17T21:00:37Z

Match ID: 1782181262
Running clarity-examples tick package. Happens on many ticks:

tick 57378, synthetic true, had 0 messages
unknown embedded message of kind SOURCE2/47. Please report this in the correspon
ding issue: #58
unknown embedded message of kind SOURCE2/47. Please report this in the correspon
ding issue: #58

Decoud · 2015-09-18T14:49:41Z

Same replay, I don't have the ticks but can get them if you want:

unknown embedded message of kind SOURCE2/497. Please report this in the correspo
nding issue: #58
unknown embedded message of kind SOURCE2/128. Please report this in the correspo
nding issue: #58

spheenik · 2015-09-18T19:06:25Z

uploaded a new snapshot, which has all those...

Decoud · 2015-09-21T19:11:36Z

unknown embedded message of kind SOURCE2/510. Please report this in the correspo
nding issue: #58

spheenik · 2015-09-21T19:14:45Z

new snapshot uploaded

Decoud · 2015-09-22T23:53:55Z

What type of messages are these things?

unknown embedded message of kind SOURCE2/62. Please report this in the correspon
ding issue: #58

howardchung · 2015-09-23T00:19:28Z

My guess is that @spheenik tried to remove as many of the messages from the generated Java classes as possible to reduce code size and speed up compile times.

spheenik · 2015-09-23T05:17:20Z

@howardchung: correct.

@Decoud:
62 is svc_UserMessage...
In S1, UserMessages had their own container, in S2, they were, until now, encoded as Embedded Messages. The latest protobufs on SteamKit do not have the message definition (yet). Someone has to extract it from the client. I'll look into it tonight...

manveru · 2015-09-23T06:14:27Z

@spheenik latest protobufs are always available at https://github.com/SteamDatabase/GameTracking/tree/master/Protobufs/dota

howardchung · 2015-09-23T06:16:36Z

+1 for GameTracking :)

spheenik · 2015-09-23T06:39:21Z

@manveru: thx!

So, their netmessages say that it's CSVCMsg_HLTVStatus. I'll release a new protobuf snapshot tonight...

spheenik · 2015-09-23T17:20:28Z

new snapshot uploading as I type...

Decoud · 2015-10-11T02:55:45Z

SOURCE2/554

howardchung · 2015-10-11T03:29:38Z

I think this is the new CombatLogDataHLTV. @spheenik added support for it last night, try updating your snapshot?

Decoud · 2015-10-12T01:54:35Z

Works fine, my bad. Should have checked for the update first. Will have to take a closer look at some of those combat log changes.

spheenik · 2015-10-12T04:08:19Z

@Decoud: have a look at the comments in issue #61 for info on what changed.

1zb · 2015-10-19T02:58:41Z

kind 534 or 544

file name: frankfurt_major\1856290165_1750727379.dem
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
league id: 3671
total time taken: 0.158s

file name: frankfurt_major\1858282173_1016472045.dem
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
league id: 3671
total time taken: 0.535s

file name: frankfurt_major\1859146913_1174254792.dem
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
league id: 3671
total time taken: 0.582s

file name: frankfurt_major\1860271085_922706473.dem
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
league id: 3671
total time taken: 0.601s

file name: frankfurt_major\1861004655_314559210.dem
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
league id: 3671
total time taken: 0.511s

file name: frankfurt_major\1861405896_2145933448.dem
unknown embedded message of kind SOURCE2/544. Please report this in the corresponding issue: #58
unknown embedded message of kind SOURCE2/544. Please report this in the corresponding issue: #58
league id: 3671
total time taken: 0.321s

file name: frankfurt_major\1861422181_1394172680.dem
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
unknown embedded message of kind SOURCE2/534. Please report this in the corresponding issue: #58
league id: 3671
total time taken: 0.213s

spheenik · 2015-10-19T04:20:19Z

Hi, thx for reporting!

Deployed a new snapshot of clarity protobuf that has 534,

The 544 is the new combatlog - I already included it a week ago. So it seems your snapshots are not up to date. mvn -U ... should do the trick.

spheenik · 2015-10-19T04:24:22Z

Bullshit - 544 was new too - added.

howardchung · 2016-02-18T06:40:25Z

unknown embedded message of kind SOURCE2/557. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

According to proto definition appears to be some kind of match metadata. Type is bytes so it's probably binary encoded in some way.

spheenik · 2016-02-18T11:01:55Z

I have a fix, but I cannot test it.
Can you provide a D/L link to a replay?

howardchung · 2016-02-18T17:33:16Z

https://s3-us-west-2.amazonaws.com/manta.dotabuff/2159568145.dem

spheenik · 2016-02-18T17:41:39Z

Thx, will push a fix tonight.

spheenik · 2016-02-18T18:14:03Z

Fix is out.

howardchung · 2016-03-25T03:34:50Z

unknown embedded message of kind SOURCE2/558. Please report this in the corresponding issue: #58

(lower priority, I don't think this crashes the parser)

TodorovicF · 2016-03-30T07:55:30Z

I also got the same error as @howardchung

unknown embedded message of kind SOURCE2/558. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

It seemed to have ran alright, no errors, though it does seem like it took longer to complete.

spheenik · 2016-04-02T14:45:45Z

added 558 (S2DotaGcCommon.CMsgDOTAMatch) which contains a lot of detailed information about the match.

PatheticNeatness · 2016-04-10T11:38:07Z

Can you help me with

18:36:10.003 [clarity-runner] WARN  s.c.p.reader.InputSourceProcessor - unknown usermessage message of kind SOURCE1/15. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
18:36:10.003 [clarity-runner] WARN  s.c.p.reader.InputSourceProcessor - unknown usermessage message of kind SOURCE1/27. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

spheenik · 2016-04-10T17:00:51Z

added those 2 with this commit and pushed a new snapshot.

TodorovicF · 2016-06-29T01:58:42Z

I've been getting crashes lately with this code:

unknown top level message of kind SOURCE2/1171. Please report this in the corresponding issue: #58

Exception in thread "clarity-runner" java.lang.ClassCastException: java.io.IOException cannot be cast to java.lang.RuntimeException
    at skadistats.clarity.event.Event.raise(Event.java:25)
    at skadistats.clarity.processor.runner.AbstractRunner.runWith(AbstractRunner.java:78)
    at skadistats.clarity.processor.runner.ControllableRunner.access$1401(ControllableRunner.java:13)
    at skadistats.clarity.processor.runner.ControllableRunner$3.run(ControllableRunner.java:208)
    at java.lang.Thread.run(Unknown Source)

Whoeza · 2022-12-03T15:20:15Z

So you have to find the definition somewhere, (but where?) and update clarity-protobuf. I see that the repository has not been updated for 2 years, how is that possible? I see that this issue thread is quite active... Does it have to do with Maven Central? Like, clarity-protobuf being updated on Maven and not on GitHub?

Can I help keeping clarity-protobuf up to date?

spheenik · 2022-12-03T15:46:55Z

There is a project on Github that extracts the protobuf definitions from the game regularly.

https://github.com/SteamDatabase/GameTracking-Dota2

My approach is not to simply copy their definitions completely, since many of them are not needed for parsing demos, and sometimes they remove fields also, and I keep those fields in my definitions, since I want older replays to also be able to be read.

It's true I didn't update for a long time, and should do so. But since the "core definitions" do not change, and there are mostly new UserMessages that will not be of interest for replay parsing anyway, I just figured it's priority B.

Whoeza · 2022-12-03T16:04:00Z

There is a project on Github that extracts the protobuf definitions from the game regularly.

https://github.com/SteamDatabase/GameTracking-Dota2

My approach is not to simply copy their definitions completely, since many of them are not needed for parsing demos, and sometimes they remove fields also, and I keep those fields in my definitions, since I want older replays to also be able to be read.

I see, your approach has some careful steps to take. I propose you and/or I could make a page in the wiki on this repository (or clarity-protobufs) in GitHub and explain the steps. So if I or someone else who wanted to help keeping clarity-protobufs updated, could follow those steps and produce pull requests.

How does it sound to you?

It's true I didn't update for a long time, and should do so. But since the "core definitions" do not change, and there are mostly new UserMessages that will not be of interest for replay parsing anyway, I just figured it's priority B.

I followed your advice from time ago and started from clarity-examples and clarity-analyzer to be able to understand how to parse game events. Until I learn to do it, I wanted to use Clarity Analyzer to read through a replay quickly, to understand what variables I can parse with clarity. But these unknown message types make Analyzer crash. The next best thing is to use clarity-examples to dump the entire replay, which is not great for a beginner like me, but I make do with what I have.

I'd be glad to help in the little free time I have.
PS: when submitting in this issue 58, is it enough to post just one match ID?

spheenik · 2022-12-03T16:50:59Z

Unknown message ids should not make analyzer crash. The crash is most surely related to something else.

Whoeza · 2022-12-03T17:44:42Z

Unknown message ids should not make analyzer crash. The crash is most surely related to something else.

Ah, sorry. I did not understand it. That's what you meant when you said the warning and the exception are not related to each other.

Can I help with the crash as well? Should I open a new issue for the crash to separate it from this thread?

spheenik · 2022-12-03T18:15:51Z

The analyzer crash is most certainly because it uses an old version of clarity.
It should work if it uses 2.7.5.

Whoeza · 2022-12-03T18:48:54Z

The analyzer crash is most certainly because it uses an old version of clarity. It should work if it uses 2.7.5.

Oh thank you, it was that... it works now.

Let me know if I can help with the other stuff we talked about, glad if I can.

darrensapalo · 2023-01-11T06:48:33Z

Detailing the above details here: faq

I too am experiencing the above mentioned issue:

unknown embedded message of kind SOURCE2/154. 
Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

Using git repo: clarity-examples @ commit
Clarity version: 2.7.5 commit

with replay ID: 6963982486.dem, but as @spheenik explained above, he still needs to (1) extract the new protobuf definitions and (2) tweak the definitions to enable backwards compatible replay parsing.

Let me know if I misunderstood anything 🙇

Whoeza · 2023-01-11T12:46:16Z

Detailing the above details here: faq

I too am experiencing the above mentioned issue:
unknown embedded message of kind SOURCE2/154. 
Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58
Using git repo: clarity-examples @ commit Clarity version: 2.7.5 commit

with replay ID: 6963982486.dem, but as @spheenik explained above, he still needs to (1) extract the new protobuf definitions and (2) tweak the definitions to enable backwards compatible replay parsing.

Let me know if I misunderstood anything 🙇

Thank you for starting the wiki! I did not know if we were allowed to do it.
About your understanding, that's what I got as well.

spheenik · 2023-01-14T20:43:10Z

New protobufs have been released.

Added
SOURCE2/114 (CUserMessageRequestState)
SOURCE2/154 (CUserMessageServerFrameTime)
SOURCE2/496 (CDOTAUserMsg_SwapVerify)
SOURCE2/499 (CDOTAUserMsg_ItemAlert)
SOURCE2/531 (CDOTAUserMsg_PlayerMMR)
SOURCE2/562 (CDOTAUserMsg_KillcamDamageTaken)
SOURCE2/608 (CDOTAUserMsg_AghsStatusAlert)
SOURCE2/616 (CDOTAUserMsg_DuelOpponentKilled)
SOURCE2/617 (CDOTAUserMsg_DuelAccepted)

My testing capability was limited, since downloading replays through the client seems buggy atm. (Opendota also does not work).

Regarding the Wiki Thing:
Doing a FAQ there is probably really a good thing. @darrensapalo Mind if I change your entry to properly explain this?

spheenik · 2023-01-14T20:44:34Z

Oh, and, the protobufs contain several extra combatlog properties. If you want to use them, I released clarity 2.7.6. Pushed both artifacts to central five minutes ago, so might be an hour or so before you see them.

darrensapalo · 2023-01-15T21:15:36Z

@spheenik Please do! I'll follow your lead on the docs. Thanks spheenik!

p0d3x · 2024-02-08T15:08:24Z

unknown top level message of kind CSGO_S2/16

also no decoder for class CCSPlayerController at 19/11! since the update today

lochetti · 2024-04-12T20:01:51Z

Not sure how to be more helpful, but I got a
WARN clarity.runner - unknown top level message of kind CSGO_S2/16. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

I was trying to parse the first map of this game https://www.hltv.org/matches/2370928/liquid-vs-faze-iem-chengdu-2024
Using the version 3.0.3

BusyLivingDota · 2024-04-14T08:15:07Z

unknown top level message of kind CSGO_S2/16. Please report this in the corresponding issue: #58
on tick 152 of the cs2 replay:
https://www.hltv.org/matches/2370727/faze-vs-natus-vincere-pgl-cs2-major-copenhagen-2024
map-3-Inferno

spheenik · 2024-04-17T06:48:17Z

Thank you guys, CSGO_S2/16 noted, I'll put updating the protobufs to my not so small list of todos.

spheenik · 2024-04-27T12:44:41Z

CSGO_S2/16 and CSGO_S2/17 fixed in protobuf release 5.2.

thwonghin · 2024-06-11T04:32:04Z

Got an error parsing furia-vs-ninjas-in-pyjamas-m1-ancient.dem from this match
https://www.hltv.org/matches/2372732/furia-vs-ninjas-in-pyjamas-yalla-compass-2024

unknown embedded message of kind CSGO_S2/76

spheenik · 2024-06-11T16:37:43Z

Thanks for reporting, as long as the parsing process didn't crash, it's fine.
I'll come around to updating the protobufs some time.

darrensapalo · 2024-06-13T16:20:58Z

Thanks for reporting, as long as the parsing process didn't crash, it's fine. I'll come around to updating the protobufs some time.

Hey Martin @spheenik! When you do plan to do it- Would it be possible for you to record your process? I'd like to learn and document the process in the FAQ so others can potentially contribute in the protobuf parsing. You don't need to explain everything step by step- just dropping specific keywords will be enough for me to do my own research to be able to do it myself.

If this is inconvenient to do- no worries. I just want to express my gratitude for your work! Thank you!

ndorigatti · 2024-08-12T23:00:04Z

Got an error parsing furia-vs-ninjas-in-pyjamas-m1-ancient.dem from this match https://www.hltv.org/matches/2372732/furia-vs-ninjas-in-pyjamas-yalla-compass-2024
unknown embedded message of kind CSGO_S2/76

got the same error, but the parser crashes. I'm just trying to seek to "getLastTick" in order to get scoreboard

spheenik · 2024-09-14T16:28:50Z

CSGO_S2/76, CSGO_S2/141 and CSGO_S2/154 have been added in clarity-protobuf 5.3.

spheenik · 2024-09-14T16:40:26Z

@ndorigatti I just tested your replay, and it does not crash here in getLastTick() with clarity 3.1.0.

ndorigatti · 2024-09-14T17:03:09Z

@ndorigatti I just tested your replay, and it does not crash here in getLastTick() with clarity 3.1.0.

Yeah it was not in that furia replay, was one of mine... I will try again, but don't know how to share that 300mb replay

spheenik · 2024-09-16T19:07:55Z

@ndorigatti I have a hunch that self recorded replays do not properly encode the last tick.
Anyway, if you manage to upload it somewhere, open a new issue for it, because this issue here is for unknown message types.
Cheers.

thurd · 2024-09-23T14:12:54Z

Dota 2. Match ID: 7888568761

I get the following message:
unknown embedded message of kind DOTA_S2/624. Please report this in the corresponding issue: https://github.com/skadistats/clarity/issues/58

Clarity seems to be up to date: api("com.skadistats:clarity:3.1.1")

spheenik · 2024-09-23T15:04:21Z

@thurd fixed in clarity-protobuf 5.4

any626 · 2024-10-18T06:51:09Z

2024-10-17T23:49:17.997-07:00 WARN 38708 --- [ clarity-runner] clarity.runner : unknown embedded message of kind DEADLOCK/353. Please report this in the corresponding issue: #58
2024-10-17T23:49:18.184-07:00 WARN 38708 --- [ clarity-runner] clarity.runner : unknown embedded message of kind DEADLOCK/348. Please report this in the corresponding issue: #58
2024-10-17T23:49:18.371-07:00 WARN 38708 --- [ clarity-runner] clarity.runner : unknown embedded message of kind DEADLOCK/351. Please report this in the corresponding issue: #58
2024-10-17T23:49:19.860-07:00 WARN 38708 --- [ clarity-runner] clarity.runner : unknown embedded message of kind DEADLOCK/349. Please report this in the corresponding issue: #58
2024-10-17T23:49:20.608-07:00 WARN 38708 --- [ clarity-runner] clarity.runner : unknown embedded message of kind DEADLOCK/352. Please report this in the corresponding issue: #58
2024-10-17T23:49:24.878-07:00 WARN 38708 --- [ clarity-runner] clarity.runner : unknown embedded message of kind DEADLOCK/350. Please report this in the corresponding issue: #58

darrensapalo mentioned this issue Jan 11, 2023

nextByte: insufficient buffer (24 of 23) dotabuff/manta#147

Open

ndorigatti mentioned this issue Sep 17, 2024

CS2 - Issue Parsing Last tick (not only last) #321

Closed

Unknown Message Types #58

Unknown Message Types #58

Comments

spheenik commented Sep 13, 2015 • edited Loading

Decoud commented Sep 17, 2015

Decoud commented Sep 18, 2015

spheenik commented Sep 18, 2015

Decoud commented Sep 21, 2015

spheenik commented Sep 21, 2015

Decoud commented Sep 22, 2015

howardchung commented Sep 23, 2015

spheenik commented Sep 23, 2015

manveru commented Sep 23, 2015

howardchung commented Sep 23, 2015

spheenik commented Sep 23, 2015

spheenik commented Sep 23, 2015

Decoud commented Oct 11, 2015

howardchung commented Oct 11, 2015

Decoud commented Oct 12, 2015

spheenik commented Oct 12, 2015

1zb commented Oct 19, 2015

spheenik commented Oct 19, 2015

spheenik commented Oct 19, 2015

howardchung commented Feb 18, 2016

spheenik commented Feb 18, 2016

howardchung commented Feb 18, 2016

spheenik commented Feb 18, 2016

spheenik commented Feb 18, 2016

howardchung commented Mar 25, 2016

TodorovicF commented Mar 30, 2016

spheenik commented Apr 2, 2016

PatheticNeatness commented Apr 10, 2016

spheenik commented Apr 10, 2016

TodorovicF commented Jun 29, 2016

Whoeza commented Dec 3, 2022

spheenik commented Dec 3, 2022

Whoeza commented Dec 3, 2022 • edited Loading

spheenik commented Dec 3, 2022

Whoeza commented Dec 3, 2022

spheenik commented Dec 3, 2022

Whoeza commented Dec 3, 2022

darrensapalo commented Jan 11, 2023 • edited Loading

Whoeza commented Jan 11, 2023

spheenik commented Jan 14, 2023

spheenik commented Jan 14, 2023

darrensapalo commented Jan 15, 2023

p0d3x commented Feb 8, 2024

lochetti commented Apr 12, 2024

BusyLivingDota commented Apr 14, 2024

spheenik commented Apr 17, 2024 • edited Loading

spheenik commented Apr 27, 2024

thwonghin commented Jun 11, 2024

spheenik commented Jun 11, 2024

darrensapalo commented Jun 13, 2024

ndorigatti commented Aug 12, 2024

spheenik commented Sep 14, 2024

spheenik commented Sep 14, 2024

ndorigatti commented Sep 14, 2024

spheenik commented Sep 16, 2024

thurd commented Sep 23, 2024 • edited Loading

spheenik commented Sep 23, 2024

any626 commented Oct 18, 2024

spheenik commented Sep 13, 2015 •

edited

Loading

Whoeza commented Dec 3, 2022 •

edited

Loading

darrensapalo commented Jan 11, 2023 •

edited

Loading

spheenik commented Apr 17, 2024 •

edited

Loading

thurd commented Sep 23, 2024 •

edited

Loading