Skip to content

Improve schemeshard operations build time #10633

Open
@ijon

Description

@ijon

Compiling schemeshard and schemeshard operations takes significant amount of time.
(8394 seconds, which is 2.5 hours on a single core or 8.7 minutes on a 16 cores)

And while schemeshard consists of several parts including the core, schema operations support mechanics, split-merge subsystem, background compaction subsystem, long running schema processes ("meta-operations") like database backup/restore, and so on, the real concern this issue is trying to address (beyond the total schemeshard building time) is the build time of the single schema operation.

Schemeshard includes implementations of dozens of scheme operations (78 to be precise)...

TODO...

  • move TCompactionPolicy from flat_scheme_op.proto to some core .proto
  • and completely replace TCompactionPolicy with something else to use in public api -- current specification is unusable for that
  • break ydb/core/base dependency on flat_scheme_op.proto
  • localdb.h -- remove header dependency on flat_scheme_op.proto
  • schemeshard_impl.h -- split operations iface into schemeshard__operation_iface.h
  • schemeshard_impl.h -- remove use by operations
  • separate schemeshard core and operations "api"
    • make non intrusive separation
    • get rid of static_cast to TSchemeShard* in operations code
    • real api, may be with intrusive changes in operations
  • make anything coming from schemeshard__operation_iface.h forward declared
  • move static methods out of TSchemeShard (into some lib/utils module)
  • schemeshard_tx_infly.h -- remove dependency on flat_scheme_op.proto and flat_tx_scheme.proto
  • schemeshard_identifiers.h -- remove dependency on flat_scheme_op.proto
  • schemeshard__operation_memory_changes.h -- make clean interface without leaking specific types
  • schemeshard_info_types.h -- separate different types into separate files
  • schemeshard__operation_side_effects.h -- make clean interface without leaking specific types
  • schemeshard__operation_db_changes.h -- also clean from specific types
  • schemeshard.h, schemeshard_private.h -- separate event interface into client, core, operations etc interfaces and include relevant parts only where necessary
  • schemeshard__operation_common.h -- remove dependency on ydb/core/base/hive.h
  • schemeshard__operation_common.h -- remove dependency on schemeshard_private.h
  • schemeshard_utils.h
    • do not include from schemeshard_impl.h and other heavy headers
    • move TShardDeleter out of schemeshard_utils.h
    • move TSelfPinger out of schemeshard_utils.h
    • may be dismantle even
  • tests -- reduce influence of a single change on single ut_-module rebuild time
  • ...

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions