xgboost - xboost dmlc-submit eats command quotes (needed to run python job via scl enable) -
i use redhat 6.8 yarn cluster nodes, , uses python 2.6.6. in order python 2.7 in redhat 1 uses software collections. activate software collection 1 must scl enable. print python version example 1 scl enable python27 'python -v'
problem when try submit python job yarn follows
dmlc-submit --cluster=yarn scl enable python27 'python -v'
it seems eat quotes , produces error (instead of expected python 2.7.8):
unable open /etc/scl/prefixes/python!
that same output 1 gets when doing following @ bash prompt on machine
scl enable python27 python -v
i'm trying figure out how fool argparse letting through.
looked argparser (which dmlc-core , xgboost use) source code trying figure out possible escaping mechanism. couldn't find did find out dmlc-submit picks job submission node environment , recreates on each of execution nodes. looked @ scl enable
, /opt/rh/python27/enable
script , realized ld_library_path important one. path reason doesn't carry on specify full path scl provided python27
following happily runs on yarn 2.7.1 cluster
cd xgboost dmlc-core/tracker/dmlc-submit --cluster=yarn --num-workers=9 --worker-cores=4 /opt/rh/python27/root/usr/bin/python tests/distributed/test_basic.py
Comments
Post a Comment