Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SearchV2 merger does not preserve meta DB ordering for some system attributes #3160

Closed
cthulhu-rider opened this issue Feb 19, 2025 · 1 comment
Assignees
Labels
bug Something isn't working I4 No visible changes neofs-storage Storage node application issues S4 Routine U2 Seriously planned
Milestone

Comments

@cthulhu-rider
Copy link
Contributor

meta DB stores and cmps following attributes as plain bytes:

  • owner
  • payload SHA-256
  • payload TZ
  • split ID
  • all OIDs (first and parent)

on match, they go to the result in protocol string fmt. On merge multiple results from several sources (shards, nodes), merge func cmps them lexicographically. If string fmt does not preserve byte order, the order can break on merge

Expected Behavior

keep order returned from the meta DBs

Current Behavior

the order can break

Possible Solution

  1. check which ones are problematic. Being base58btc strings, owner and OIDs are definitely are. Check the others
  2. fix cmp in the merge func

Context

@cthulhu-rider cthulhu-rider added bug Something isn't working neofs-storage Storage node application issues labels Feb 19, 2025
@roman-khimov roman-khimov added U2 Seriously planned S4 Routine I4 No visible changes labels Feb 19, 2025
@roman-khimov roman-khimov added this to the v0.45.0 milestone Feb 19, 2025
@cthulhu-rider cthulhu-rider self-assigned this Feb 19, 2025
cthulhu-rider added a commit that referenced this issue Feb 19, 2025
Test cursors deeper. Deduplicate a lot of code also.

Refs #3160. Refs #3058.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit that referenced this issue Feb 19, 2025
Test cursors deeper. Many tests fail revealing implementation bugs,
mainly ordering. Fixes coming soon.

This deduplicates a lot of code also.

Refs #3160. Refs #3058.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit that referenced this issue Feb 20, 2025
Test cursors deeper. Many tests fail revealing implementation bugs,
mainly ordering. Fixes coming soon.

This deduplicates a lot of code also.

Refs #3160. Refs #3058.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit that referenced this issue Feb 21, 2025
Test cursors deeper. Many tests fail revealing implementation bugs,
mainly ordering. Fixes coming soon.

This deduplicates a lot of code also.

Refs #3160. Refs #3058.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit that referenced this issue Feb 21, 2025
Test cursors deeper. Many tests fail revealing implementation bugs,
mainly ordering. Fixes coming soon.

This deduplicates a lot of code also.

Refs #3160. Refs #3058.

Signed-off-by: Leonard Lyubich <[email protected]>
cthulhu-rider added a commit that referenced this issue Feb 21, 2025
Test cursors deeper. Many tests fail revealing implementation bugs,
mainly ordering. Fixes coming soon.

This deduplicates a lot of code also.

Refs #3160. Refs #3058.

Signed-off-by: Leonard Lyubich <[email protected]>
@cthulhu-rider
Copy link
Contributor Author

cthulhu-rider commented Feb 21, 2025

payload SHA-256 and TZ hashes are HEX strings, so the order is preserved

ive got following results for others

func checkOrder[T fmt.Stringer](name string, rnd func() T, cmp func(a, b T) int) {
	s := make([]T, 1000000)
	for i := range s {
		s[i] = rnd()
	}
	slices.SortFunc(s, cmp)
	ss := make([]string, len(s))
	for i := range s {
		ss[i] = s[i].String()
	}
	fmt.Println(name, "order kept:", slices.IsSorted(ss))
}

func TestOrder(t *testing.T) {
	checkOrder("OID", oidtest.ID, func(a, b oid.ID) int { return bytes.Compare(a[:], b[:]) })
	checkOrder("User", usertest.ID, func(a, b user.ID) int { return bytes.Compare(a[:], b[:]) })
	checkOrder("UUID", uuid.New, func(a, b uuid.UUID) int { return bytes.Compare(a[:], b[:]) })
}

UUIDs (split ID) are HEX-encoded, too, so OK. OIDs are expectedly broken

the one that surprises me a bit is user ID: it's base58btc-encoded as well as OID, but order is kept. Mb the fact that it has fixed prefix affect smth, but i wouldnt rely on this for now. At the same time, in practice I couldnt find two user IDs with disorder

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working I4 No visible changes neofs-storage Storage node application issues S4 Routine U2 Seriously planned
Projects
None yet
Development

No branches or pull requests

2 participants