Home Explore Help
Register Sign In
apache
/
hadoop
mirror of https://github.com/apache/hadoop.git
2
0
Fork 1
Files Issues 0 Wiki
Tree: 8024818d33
Branches Tags
hadoop
/
hadoop-submarine
/
hadoop-submarine-core
Szilard Nemeth e5f4cd0fda SUBMARINE-57. Add more elaborate message if submarine command is not recognized. Contributed by Adam Antal 5 years ago
..
src e5f4cd0fda SUBMARINE-57. Add more elaborate message if submarine command is not recognized. Contributed by Adam Antal 5 years ago
README.md 03aa70fe19 SUBMARINE-83. Refine the documents of submarine targeting 0.2.0 release. Contributed by Zhankun Tang. 6 years ago
pom.xml 2185249481 Preparing for submarine-0.3.0 development 6 years ago

README.md

Overview

              _                              _
             | |                            (_)
  ___  _   _ | |__   _ __ ___    __ _  _ __  _  _ __    ___
 / __|| | | || '_ \ | '_ ` _ \  / _` || '__|| || '_ \  / _ \
 \__ \| |_| || |_) || | | | | || (_| || |   | || | | ||  __/
 |___/ \__,_||_.__/ |_| |_| |_| \__,_||_|   |_||_| |_| \___|

                             ?
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~|^"~~~~~~~~~~~~~~~~~~~~~~~~~o~~~~~~~~~~~
        o                   |                  o      __o
         o                  |                 o     |X__>
       ___o                 |                __o
     (X___>--             __|__            |X__>     o
                         |     \                   __o
                         |      \                |X__>
  _______________________|_______\________________
 <                                                \____________   _
  \                                                            \ (_)
   \    O       O       O                                       >=)
    \__________________________________________________________/ (_)

Submarine is a project which allows infra engineer / data scientist to run unmodified Tensorflow or PyTorch programs on YARN or Kubernetes.

Goals of Submarine:

  • It allows jobs easy access data/models in HDFS and other storages.
  • Can launch services to serve Tensorflow/PyTorch models.
  • Support run distributed Tensorflow jobs with simple configs.
  • Support run user-specified Docker images.
  • Support specify GPU and other resources.
  • Support launch tensorboard for training jobs if user specified.
  • Support customized DNS name for roles (like tensorboard.$user.$domain:6006)

Please jump to QuickStart guide to quickly understand how to use this framework.

Please jump to Examples to try other examples like running Distributed Tensorflow Training for CIFAR 10.