a lot of sources, transforms

[Rentu]

I have a lot of log collection, and the simplest way to manage them is to use the filename as a source. However, this may lead to too many sources and impact performance, so we do some merging. But in the transform stage, different logs may have different conditions for multi-line matching or different rules for timestamp extraction. This could lead to a large number of transforms, possibly over a thousand. What are some good solutions for such a scenario? Our daily log collection volume is more than 30TB, and the logging scenarios are complex, with inconsistent encoding and a large number of log collections, making the cleansing rules complex.