Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[IO] Add Redis Connector #289

Open
wants to merge 7 commits into
base: experimental
Choose a base branch
from
Open

Conversation

gypleon
Copy link

@gypleon gypleon commented Jun 26, 2017

Test cases

- Preparation

  1. Flush Redis data using "/data/opt/course/redis/flushall-cluster.sh"
  2. Check configuration file "$HUSKY_ROOT/test/test-redis-config.ini"

- Test 1: CMake

  1. Follow the standard Cmake commands of Husky.
  2. make "Master".
  3. make "TestRedisOutputFormat".
  4. make "TestRedisInputOutputFormat".
  5. make "WordCountRedis".

Expectation: running well.
Notes:

- Test 2: enwiki from MongoDB to Redis:

  1. Run "./Master -C test-redis-config.ini"
  2. Run "time /data/opt/tmp/exec.sh ./TestRedisOutputFormat -C test-redis-config.ini"

Expectation: running well. 4825150 records to be imported. around 1m30s.
Notes: MUST wait for 10 minutes before Test 3, because for the first time data were loaded into Redis cluster, it takes time for master-slave synchronization.

- Test 3: enwiki from Redis to Redis:

  1. Run "./Master -C test-redis-config.ini"
  2. Run "time /data/opt/tmp/exec.sh ./TestRedisInputOutputFormat -C test-redis-config.ini"

Expectation: running well. 4825150 records to be imported. around 1m.
Notes: On Husky-worker side, "waiting for keys" logs may be printed as expected, which means that some workers are waiting for keys from the next batch.

- Test 4: enwiki WordCount:

  1. Run "./Master -C test-redis-config.ini"
  2. Run "time /data/opt/tmp/exec.sh ./WordCountRedis -C test-redis-config.ini"

Expectation: running well. 4825150 records to be imported. around 50s.
Notes: Can be performed consecutively without restart Husky-Master.

@gypleon
Copy link
Author

gypleon commented Jun 26, 2017

OMG!!! I merged these commits for several hours... but they come out again... maybe I made a ridiculous mistake that override the merge operations.... OMG.... actually, "[IO] Add Redis Connector. Fixed merge conflicts." (38ce3c2) is supposed to be the ONLY ONE left.

I am currently searching for safely merging solutions.

@gypleon gypleon force-pushed the dev branch 3 times, most recently from f322ea9 to 95228f4 Compare June 27, 2017 17:06
@gypleon
Copy link
Author

gypleon commented Jun 27, 2017

Hi Legend,
I think the commit problem has been fixed. Please review and leave me comments.
Thanks.

@kygx-legend
Copy link
Member

This PR is including other commits in master branch which are not merged into experimental branch. Could you redo the rebase?

@gypleon
Copy link
Author

gypleon commented Jun 28, 2017

No problem.

@kygx-legend kygx-legend changed the title Add Redis Connector. Add Redis Connector Jun 28, 2017
@kygx-legend kygx-legend changed the title Add Redis Connector [IO] Add Redis Connector Jun 28, 2017
@kygx-legend
Copy link
Member

This branch is updated to master now. Please rebase again!

@kygx-legend
Copy link
Member

And you can run ./scripts/clang-format.py -o check to see what does not match the coding style and run ./scripts/clang-format.py -o replace to fix those errors automatically. Then use lint.py to double check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants