2023-11-13 - KDB Project Call Meeting Minutes

[alvin-c-shih]

Date

Monday, November 13, 2023 - 9:30am ET / 2:30pm UK

// Second Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
James McLeod	FINOS	@mcleo-d
Alvin Shih	Morgan Stanley	@alvin-c-shih
Jo Shinonome	ExeQution Analytics	@jshinonome
Peter Gyorok	KX Systems	@gyorokpeter
Rian Ó Cuinneagáin	KX Systems	@rianoc
Gary Davies	Data Intellect	@drgdavies
Aaron Davies	Morgan Stanley	@adavies42
Connor Gervin	KX Systems	@Connor-Gervin

Meeting notices

• FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.

• All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.

• FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact [email protected] with any questions.

• FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

• (5 min) Convene, roll call, welcome new people.
• Display FINOS Antitrust Policy summary slide
• Review Meeting Notices (see above)
• (5 min) Approve previous meeting minutes.
  • Approved by acclamation.
• (10 min) Takeaways from OSFF New York?
  • Data Lakehouse presentation by DataBricks showed a way to expose federated data under a consistent view for analysis / reporting.
  • KX extends this notion with the Data Timehouse:
    • https://kx.com/data-timehouse/
    • https://kx.com/blog/the-data-timehouse/
    • https://kx.com/news/kx-delivers-industry-first-data-timehouse-on-microsoft-azure/
• (10 min) Takeaways from Vectorize NY:
  • Retrieval-Augmented Generation (RAG) might not necessarily be decided by speed.
    • Erin Stanton shared that question answering results were better with small, curated document corpus.
    • Answer quality also affected by: chunk size, overlaps, embedding model, and LLM.
    • Would be beneficial for KX to have SMEs who write about how to achieve the best quality results with scalability as a secondary concern.
  • GitHub copilot quite bad at q programming - dictionaries especially.
    • Expose APIs that wrap q.
    • PyKX select functionality still exposes qsql.
      • Roadmap includes more Pythonic API with object-based where clauses.
      • Citadel QF wrapper for PyKX has already headed in that direction.

PyKX

res=pykx.q.qsql.select(
         table=trade, columns={'avgpx': 'qty wavg price'},
         where=['date=2023.02.01','sym in `IBM`MSFT'], by={'sym':'sym'})

Citadel QF:

res=trade.select(
         columns={'avgpx': qf.col('qty').wavg('price')},
         where=[qf.col('date') == datetime.date(2023,2,1),
                qf.col('sym').isin(['IBM','MSFT'])],
         by={'sym': qf.col('sym')})

• • • • Rust wrapper on Polars might also be a source of inspiration:
        • https://docs.rs/polars/latest/polars/
    • Someone mentioned BuaBook (which has come up in previous FINOS calls).

• (10 min) Walkthrough of Feature/unzip #98 .

  • Prepends a header of structs for .Q.gz to decompress.
  • Extracts into memory to avoid hassles of managing disk space.
  • Parallel unzipping with peach.
  • Fixed handling of empty archive. (Received one from a vendor feed.)
    • File format: headers, file, headers, file, ... , central directory
    • Empty zipfile has "end of central directory" block with zero entries.
  • Might be nice to implement using qbinparse.
  • Other utilities related to file ingestion that might be of interest:
    • https://github.com/adavies42/qist/tree/master/bin
      • awq -awk(1) for q
      • jqq -jq(1) for q
    • CSVKit from Ferenc Bodon
      • https://www.kdnuggets.com/2020/07/powerful-csv-processing-kdb.html

• (5 min) technical-documentation/ ideas continued

  • "Awesome XYZ" seems to be a popular name for GitHub repos hosting curated lists of resources:
    • https://github.com/EthicalML/awesome-production-machine-learning
  • Found "Awesome K" but not much momentum:
    • https://github.com/mkmik/awesome-k
  • Still looking for a one stop shop for HDB querying gotchas.
    • Maybe PyKX could produce warnings.

• (5 min) AOB, Q&A, Adjourn.

  • Rust integration with q?
    • https://github.com/KxSystems/kxkdb
    • Two lines of Rust code to allow parallel deserialization that's faster than PyKX.
    • Curious to see if anyone has written q plugins in Rust.
    • Financial services companies might not support Rust development.
      • Full ecosystem would require: code scanning, CI/CD, integration with other apps / infrastructure.

WebEx info

Webex

• https://finos.webex.com/finos/j.php?MTID=me61719ca3c3295a7cc76762d78392bc3

Dial-in and Project Materials

• US: +1-415-655-0003 US Toll
• UK: +44-20319-88141 UK Toll
• Access code: 2704 417 0390
• GitHub Repos:
  • https://github.com/finos/kdb
  • https://github.com/finos/kdb-studio
• Wiki Page:
  • https://finosfoundation.atlassian.net/wiki/spaces/DT/pages/329383945/kdb+Project
• KDB Metrics on FINOS
  • https://metrics.finos.org/goto/033b15c80f0d08cab70f40f4358932f6

[mcleo-d]

James McLeod / FINOS

[gyorokpeter]

Peter Gyorok/KX

[adavies42]

Aaron Davies/Morgan Stanley

[jshinonome]

Jo 🦊

[drgdavies]

Gary Davies/Data Intellect

[rianoc]

Rian Ó Cuinneagáin / KX

[jshinonome]

# an example for polars, polars is lighting fast, same level as kdb
morning = (
    taxi.filter(pl.col("pickup_time") >= time(7), pl.col("pickup_time") < time(9))
    .select(["weekday", "trip_distance"])
    .group_by(pl.col("weekday"))
    .sum()
    .rename({"trip_distance": "morning_rush_taxi"})
)

[alvin-c-shih]

Alvin Shih / Morgan Stanley