Langjian/distributed json #366

jianyinglang · 2018-12-06T20:26:08Z

No description provided.

…unittest

…d_json

sayantan-nervana · 2018-12-21T00:38:55Z

examples/mnist/mnist_softmax_distributed.py

+  global_step = tf.contrib.framework.get_or_create_global_step()
+  opt = tf.train.GradientDescentOptimizer(0.5)
+  # Add MPI Distributed Optimizer
+  #import pdb; pdb.set_trace()


this pub line can be removed

sayantan-nervana · 2018-12-21T00:39:52Z

examples/mnist/mnist_softmax_distributed.py

+  parser.add_argument('--log_dir',type=str, default='/tmp/tensorflow/mnist/logs/mnist_with_summaries',
+                      help='Summaries log directory')
+  FLAGS, unparsed = parser.parse_known_args()
+  tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)


Can you please add the command line to run this as a comment in the end of the file

Added an example run command for distributed runs

sayantan-nervana · 2018-12-21T00:40:19Z

src/ngraph_encapsulate_op.cc

@@ -39,6 +39,10 @@

 #include "ngraph/runtime/interpreter/int_backend.hpp"

+#ifdef NGRAPH_DISTRIBUTED
+#include <mpi.h>>


>>. Is this a typo?

sayantan-nervana · 2018-12-21T00:40:56Z

src/ngraph_rewrite_pass.cc

@@ -29,6 +29,10 @@

 #include <iomanip>

+#ifdef NGRAPH_DISTRIBUTED
+#include <mpi.h>>


>> is this a typo?

Thanks a lot for catching this!

sayantan-nervana · 2018-12-21T18:35:25Z

examples/mnist/mnist_softmax_distributed.py

+        help='Summaries log directory')
+    FLAGS, unparsed = parser.parse_known_args()
+    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
+# run comand for this distributed script


Suggested change

# run comand for this distributed script

# run command for this distributed script

Thanks for catching this. Fixed it.

…aSystems/ngraph-tf into langjian/distributed_json

…d_json

avijit-nervana · 2019-01-09T04:34:36Z

build_ngtf.py

@@ -512,6 +512,7 @@ def main():
        "-DNGRAPH_TUNE_ARCH=" + target_arch,
        "-DNGRAPH_ARTIFACTS_DIR=" + artifacts_location,
        "-DUNIT_TEST_ENABLE=ON",
+        "-DNGRAPH_DISTRIBUTED_ENABLE=TRUE",


The default should be false i.e., for regular builds no need to enable distributed. Need to add a command line option to the build_ngtf.py (e.g., --enable_distributed) which when specified will add this flag. See the --debug_build as an example of how to extend the cmake options.

Added the option of --distributed_build for distributed build

avijit-nervana · 2019-01-09T04:35:59Z

examples/mnist/mnist_softmax_distributed.py

+See extensive documentation at
+https://www.tensorflow.org/get_started/mnist/beginners
+"""
+from __future__ import absolute_import


Please provide a summary of modifications made to this file and a reference to the source of this file.

Listed the changes I made from the source file and added the link to the reference source of this file

avijit-nervana · 2019-01-09T04:36:27Z

src/ngraph_encapsulate_op.cc

@@ -36,6 +36,10 @@

 #include "ngraph/runtime/interpreter/int_backend.hpp"

+#ifdef NGRAPH_DISTRIBUTED


It's better to use the if defined(NGRAPH_DISTRIBUTED)

Changed to #if defined NGRAPH_DISTRIBUTED

avijit-nervana · 2019-01-09T04:41:30Z

src/ngraph_rewrite_pass.cc

@@ -102,13 +106,33 @@ class NGraphRewritePass : public GraphOptimizationPass {
  static std::string GraphFilenamePrefix(std::string kind, int idx) {
    std::stringstream ss;
    ss << kind << "_" << std::setfill('0') << std::setw(4) << idx;
+#ifdef NGRAPH_DISTRIBUTED


Consider a function that can be used instead of the code repetition below.

Used the distributed class defined in nGraph core

…d_json

…aSystems/ngraph-tf into langjian/distributed_json

avijit-nervana · 2019-01-11T05:43:13Z

CMakeLists.txt

-    include_directories(SYSTEM ${MPI_C_INCLUDE_PATH} ${MPI_CXX_INCLUDE_PATH})
-    link_directories(${MPI_C_LIBRARIES} ${MPI_CXX_LIBRARIES})
-endif()
+#if(NGRAPH_DISTRIBUTED_ENABLE)


Is this temporary? If not please remove the commented out code.

Changed back to include the MPI build

…aSystems/ngraph-tf into langjian/distributed_json

jianyinglang · 2019-01-18T00:44:02Z

TESTNOW

jianyinglang added 9 commits November 21, 2018 13:03

Fix new klocwork error reports on 11/20/2018

3878e43

Fix the FusedBatchNorm with the Bessel correction in variance

4decf57

Fix the format

cfb0a83

Merge remote-tracking branch 'origin/master' into langjian/BatchNorm_…

76ab3fa

…unittest

Add distributed macro

289a43d

Add multi-node .json file output

2163ac1

Merge remote-tracking branch 'origin/master' into langjian/distribute…

34c4ddf

…d_json

Change CMake file to be consistent with master

2e70b97

Format change

1c34c3d

jianyinglang changed the title ~~Langjian/distributed json --- WIP~~ [WIP] Langjian/distributed json Dec 6, 2018

jianyinglang added the WIP label Dec 6, 2018

jianyinglang added 8 commits December 10, 2018 12:26

Add a simple distributed mnist model

70449be

Merge remote-tracking branch 'origin/master' into langjian/distribute…

5ea9070

…d_json

Add distributed option for Makefile

73cf9d0

modify distributed example

2c594ee

Add distributed flags for multi-process graph dumps

d788247

Changes the Makefile to enable distributed build

0371935

Merge remote-tracking branch 'origin/master' into langjian/distribute…

d16f2d9

…d_json

Format fix

f205cb2

jianyinglang removed the WIP label Dec 21, 2018

jianyinglang changed the title ~~[WIP] Langjian/distributed json~~ Langjian/distributed json Dec 21, 2018

jianyinglang requested review from avijit-nervana and sayantan-nervana December 21, 2018 00:11

sayantan-nervana reviewed Dec 21, 2018

View reviewed changes

jianyinglang and others added 5 commits December 20, 2018 16:49

Fix the typo and delete debug comment and add run command

0ecd43b

Fix the python file format

100c129

Fix python file format

dda11aa

Add mnist data directory

6caf977

Merge branch 'master' into langjian/distributed_json

e2ba875

sayantan-nervana reviewed Jan 4, 2019

View reviewed changes

jianyinglang added 2 commits January 4, 2019 09:58

Fix the typo

59ffb37

Merge branch 'langjian/distributed_json' of https://github.com/Nervan…

7f6c498

…aSystems/ngraph-tf into langjian/distributed_json

sayantan-nervana approved these changes Jan 4, 2019

View reviewed changes

sayantan-nervana added the Fully Reviewed All reviewers have approved label Jan 4, 2019

jianyinglang added 5 commits January 4, 2019 22:45

Set the default distributed build as false

6507c3a

Add initialization if not initialized in MPI

dbb302e

Merge remote-tracking branch 'origin/master' into langjian/distribute…

b163c5c

…d_json

Fix the format

35af774

Fix the format using python2

97bb170

avijit-nervana suggested changes Jan 9, 2019

View reviewed changes

avijit-nervana and others added 4 commits January 9, 2019 10:12

Merge branch 'master' into langjian/distributed_json

7515794

Merge remote-tracking branch 'origin/master' into langjian/distribute…

2f515af

…d_json

Merge branch 'langjian/distributed_json' of https://github.com/Nervan…

e97fd82

…aSystems/ngraph-tf into langjian/distributed_json

Fix the build with no specified mpi library

1f9f611

avijit-nervana suggested changes Jan 11, 2019

View reviewed changes

jianyinglang and others added 6 commits January 10, 2019 21:55

Comment out the unused lines in CMakeLists.txt

7922d0b

Fix some errors

e8553cf

Merge branch 'master' into langjian/distributed_json

0499c73

Change to if define

361c725

Merge branch 'langjian/distributed_json' of https://github.com/Nervan…

4533c46

…aSystems/ngraph-tf into langjian/distributed_json

Added the source reference for mnist_softmax_distributed.py

e2c3009

avijit-nervana approved these changes Jan 16, 2019

View reviewed changes

avijit-nervana added 2 commits January 17, 2019 00:36

Merge branch 'master' into langjian/distributed_json

ae96c77

Merge branch 'master' into langjian/distributed_json

8a941ee

avijit-nervana merged commit adcd84f into master Jan 18, 2019

avijit-nervana deleted the langjian/distributed_json branch January 18, 2019 16:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Langjian/distributed json #366

Langjian/distributed json #366

jianyinglang commented Dec 6, 2018

sayantan-nervana Dec 21, 2018

jianyinglang Dec 21, 2018

sayantan-nervana Dec 21, 2018

jianyinglang Dec 21, 2018

sayantan-nervana Dec 21, 2018

jianyinglang Dec 21, 2018

sayantan-nervana Dec 21, 2018

jianyinglang Dec 21, 2018

jianyinglang Dec 21, 2018

sayantan-nervana Dec 21, 2018

jianyinglang Jan 4, 2019

avijit-nervana Jan 9, 2019

jianyinglang Jan 14, 2019

avijit-nervana Jan 9, 2019

jianyinglang Jan 14, 2019

avijit-nervana Jan 9, 2019

jianyinglang Jan 14, 2019

avijit-nervana Jan 9, 2019

jianyinglang Jan 14, 2019

avijit-nervana Jan 11, 2019

jianyinglang Jan 14, 2019

jianyinglang commented Jan 18, 2019

	# run comand for this distributed script
	# run command for this distributed script

		@@ -36,6 +36,10 @@

		#include "ngraph/runtime/interpreter/int_backend.hpp"

		#ifdef NGRAPH_DISTRIBUTED

Langjian/distributed json #366

Langjian/distributed json #366

Conversation

jianyinglang commented Dec 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianyinglang commented Jan 18, 2019