How to bulk process a lot of SQL in python

[TatsuyaMori63]

Hi all,

I try sqllineage by using python to get table level lineages from 100,000 of SQL job log.
It takes long time over 10 hours to complete it in my Google Colab. Does anyone have good idea to reduce time for bulk processing?

My source code is here. I put SQLs into dataframe and pass them to function.

import sqlparse
from sqllineage.runner import LineageRunner

def get_lineage(query_text):
    results = []
    try:
        for statement in sqlparse.split(query_text):
            result = LineageRunner(statement, dialect="ansi")
            source_tables = [tbl.schema.raw_name.lower() + '.' + tbl.raw_name.lower() for tbl in result.source_tables]
            target_tables = [tbl.schema.raw_name.lower() + '.' + tbl.raw_name.lower() for tbl in result.target_tables]
            results.append([source_tables, target_tables])
    except:
        results = []
    return json.dumps(results)

df['ref_tables'] = df['query_text'].apply(get_lineage)

[entilo]

threads may helps

[reata]

This is more of a usage thing, so I moved it to discussion.

To my knowledge, pandas dataframe apply is running sequentially for user defined function. You can refer to #353 for high level ideas of leveraging parallelism.