proper support for Verilog line directive

[Nic30]

Currently line directive is used only by preprocessor. Preprocessor builds a large string from all the input files and this string is an input of the parser.
Problem is that the original code positions are usually lost because preprocessor does not set line directive and Verilog parser ignores it.

Solution:

• Mark all code segments with proper line directive https://github.com/Nic30/hdlConvertor/blob/master/src/verilogPreproc/verilogPreprocContainer.cpp#L90.
• Add lexer and parser rules for line directives to SystemVerilog grammar https://github.com/Nic30/hdlConvertor/blob/master/grammars/sv2017Lexer.g4 https://github.com/Nic30/hdlConvertor/blob/master/grammars/sv2017Parser.g4
• instead sharing just comment parser https://github.com/Nic30/hdlConvertor/blob/master/include/hdlConvertor/svConvertor/moduleParser.h#L16 between all parsers create a SVParserContext which will contain commentParser and actual file and offset which will be used for all positions of objects and all exceptions.

[Nic30]

Solves #88 and #89

[Thomasb81]

I understand that the 2 first bullets are working together.

But your last bullet seems to be an alternative proposition. Which is not very clear to me:
How do you maintain a meaningful link of a commentParser or whatever object with initial filename and line inside created by the preprocessor and the SVparser ?

[Nic30]

I would like to create a list of <line in preprocessed str, original file, original line> from line directives somehow. Then each time the position is required I would use this list to resolve correct original file and original line.

[Nic30]

Maybe we probably do not need the line directives in preprocessed string, but we need some sort of map which keeps the info about orig. file. This info is actually an argument of line directive.

[Thomasb81]

Well, it is sometime use full to dump prepossessed code.
Having `line directive in this dump should allow tool interoperability... I mean reprocess the dump by another tool.

[Nic30]

@Thomasb81, yes. Let's make it configurable. In pure preprocessor we would like to put line directives. If preproc. is run before parser lets produce a list of line offsets so we do not have to parse the file again.

Also maybe we would need something more powerful than just list, because of thinks like unconnected_drive, but it is only idea because unconnected_drive has to be around module declaration, so it is actually better to process it in parser.

[Nic30]

It is possible that I will be able to implement this feature this weekend is anyone working on this?

[Nic30]

It seems that this functionality is connected with intensive code removal and refactoring, I will do it during weekend.

[Nic30]

Now 8c519e4 the preprocessor exports correct file/line map for input string. This map is available in https://github.com/Nic30/hdlConvertor/blob/verilog_pp_line_directive/src/convertor.cpp#L139 verilog_pp::VerilogPreprocOutBuffer but it is not currently used.

Next step is to use this map when resolving position of the object or error.

[Thomasb81]

Now 8c519e4 the preprocessor exports correct file/line map for input string. This map is available in https://github.com/Nic30/hdlConvertor/blob/verilog_pp_line_directive/src/convertor.cpp#L139 verilog_pp::VerilogPreprocOutBuffer but it is not currently used.

Next step is to use this map when resolving position of the object or error.

As mention in #166 with #191, this structure is inaccurate. We probably need data structure that allow to reference a line interval and attach a property.
ie preprocessed line 0 to 10 come from file1.vh, 10 to 15 file1.vh, default : my module.v
ie keyword line 200 to 240 compatible verilog2001 default systemVerilog 2017

Later we can split the preprocessor result and parse the different piece with different properties.

[Thomasb81]

Why have we remove the usage of antlr4::TokenStreamRewriter ?

762791e

The initial idea in mind was to annotate each token to have the possibility to keep fine grain the origin of each token, from preprocessor up to SV parser.

By only keeping track of line I think you can't identify that original source code is behind a macro...

[Thomasb81]

antlr4::TokenStreamRewriter does not preserve original token reference. So keeping this object finally does not matter so much.