http://deepdepth.djduff.net/wiki/api.php?action=feedcontributions&user=Imad&feedformat=atomDeep Depth 116E167 Project Documentation - User contributions [en]2024-03-29T05:02:06ZUser contributionsMediaWiki 1.27.3http://deepdepth.djduff.net/wiki/index.php?title=NautilusServer&diff=174NautilusServer2018-01-31T18:10:05Z<p>Imad: /* Port Tunnelling */</p>
<hr />
<div>= Help = <br />
<br />
== Accessing ==<br />
<br />
'''Access from''': ITU or [http://bidb.itu.edu.tr/hizmetler/vpn ITU VPN].<br />
<br />
=== VPN Help ===<br />
<br />
* [http://bidb.itu.edu.tr/hizmetler/vpn BIDB has instructions for accessing the VPN.]<br />
* I have also provided here [http://files.djduff.net/scripts/ a script for accessing the ITU VPN from Ubuntu (ituvpn.sh)] (tested on Kubuntu 16.04).<br />
<br />
=== SSH help ===<br />
<br />
The SSH command to connect from a Unix environment:<br />
<br />
ssh -X -p 1542 hossein@<SERVER_IP_ADDRESS><br />
<br />
Switch meanings:<br />
* '''-p 1542''': connect on port 1542<br />
* '''-X:''' This allows you to run X applications. Omit it if you will be pure command line. You can also run X applications from Windows but you will need to install an X server on your Windows machine.<br />
<br />
'''Note''': to check that X forwarding is working, once you have connected, try running on the server the command:<br />
xeyes<br />
Or:<br />
dolphin<br />
<br />
You could for example run ''spyder'' like this. But there can be some latency across the network.<br />
<br />
=== Port Tunnelling ===<br />
You can use SSH command to tunnel your port into server port in order to access several applications (e.g. Tensorboard) from your machine. <br />
The option that is used for tunneling is:<br />
-L <SERVER_PORT>:localhost:<YOUR_PORT><br />
<br />
For example:<br />
<br />
ssh -p 1542 -L 6006:localhost:6006 alican@<SERVER_IP_ADDRESS><br />
<br />
== Setting up a deep learning environment ==<br />
<br />
<br />
=== Install anaconda ===<br />
<br />
export ANACONDA_PATH_PARENT=$HOME/software<br />
export ANACONDA_PATH=$ANACONDA_PATH_PARENT/anaconda3<br />
export ANACONDA_INSTALLER=Anaconda3-4.3.1-Linux-x86_64.sh<br />
<br />
mkdir -p ~/tmp<br />
cd ~/tmp<br />
mkdir -p $ANACONDA_PATH_PARENT<br />
wget https://repo.continuum.io/archive/$ANACONDA_INSTALLER<br />
bash $ANACONDA_INSTALLER -b -p $ANACONDA_PATH<br />
<br />
export PATH=$ANACONDA_PATH/bin:$PATH<br />
echo PATH: $PATH<br />
echo >> ~/.bashrc<br />
echo export PATH=$ANACONDA_PATH/bin:\$PATH >> ~/.bashrc<br />
<br />
=== Install tensorflow and keras ===<br />
<br />
These will be installed in a [https://conda.io/docs/intro.html conda] environment called '''deep''':<br />
<br />
export ENVNAME=deep<br />
conda create --name $ENVNAME<br />
source activate $ENVNAME<br />
conda install theano keras tensorflow tensorflow-gpu opencv pillow spyder matplotlib<br />
<br />
To check Keras is working:<br />
<br />
python -c "from keras.models import Sequential;Sequential()"<br />
<br />
To check the GPU is working with tensorflow, check this first (it should list two GPUs):<br />
<br />
nvidia-smi<br />
<br />
Then make sure the following script runs and finds one CPU and two GPUS: https://bitbucket.org/damienjadeduff/uhem_keras_tf/src/master/sariyer_python3/test_tf_gpu.py<br />
<br />
Run it like this:<br />
<br />
python test_tf_gpu.py<br />
<br />
'''Warning:''' for different versions of Tensorflow, Keras or Theano you may need to use '''[https://pip.pypa.io/en/latest/user_guide/#installing-packages pip]''' to install the version you need in an environment.<br />
<br />
== Easy file access (Linux) ==<br />
<br />
This can be useful for getting files on and off the server by accessing your remote home directory as if it was on your local computer (mounted on your file system).<br />
<br />
On YOUR Linux computer run:<br />
<br />
sudo apt-get install sshfs<br />
targ=~/remote/nautilus<br />
mkdir -p $targ<br />
fusermount -u $targ # only necessary to unmount if already tried<br />
sshfs -p 1542 -o workaround=rename YOUR_SERVER_USERNAME@SERVER_IP_ADDRESS:/home/YOUR_SERVER_USERNAME $targ<br />
<br />
Note: if parts of your system hang because the connection to the ssh server gets stale (a common problem), just do:<br />
<br />
killall ssfs<br />
<br />
It should resolve most of your problems.<br />
<br />
== Using the SSD ==<br />
<br />
There is an SSD drive installed. This drive is automatically mounted at:<br />
<br />
/media/FASTDATA1<br />
<br />
The drive belongs to user '''root''' and group '''fastdata1'''.<br />
If you cannot access it you need to get an admin (Hossein) to add you to the group with the command:<br />
<br />
sudo usermod -aG fastdata1 YOURUSERNAME<br />
<br />
And to add you a folder in there with the right permissions:<br />
<br />
sudo mkdir /media/FASTDATA1/YOURUSERNAME<br />
sudo chown YOURUSERNAME:YOURUSERNAME /media/FASTDATA1/YOURUSERNAME<br />
sudo chmod 700 /media/FASTDATA1/YOURUSERNAME<br />
<br />
== Compiling your own CUDA programs ==<br />
<br />
To do this, add the following lines to your '''.bashrc''' file in your home folder:<br />
<br />
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH<br />
export PATH=/usr/local/cuda/bin:$PATH<br />
<br />
== Running programs after you log out ==<br />
<br />
Sometimes a process may take a while and if you happen to log out the lack of interactivity may cause that process to give up on you. <br />
<br />
To get around this, people use the tool '''screen''' or later versions like '''tmux'''.<br />
<br />
You log in over ssh, then type:<br />
<br />
tmux<br />
<br />
This will put you in a little session-in-a-session. Even if you log out of ssh this tmux session will persist and you can get back to it by again typing ''tmux''.<br />
<br />
Once you've finished writing your commands (e.g. '''python my_python_program_that_takes_3_days.py'''), and started it running, you can get back to an ordinary shell session by typing: <br />
<br />
Ctrl-b d<br />
<br />
(that is, a '''Ctrl-b''' followed by a '''d''').<br />
<br />
Now if you want to get back to your previous session, just run ''tmux'' again. Actually, to be sure you connect to the right session, run:<br />
<br />
tmux list-sessions<br />
<br />
See the session number of the session to which you want to connect (e.g. '''0''') then to connect to it, run:<br />
<br />
tmux attach -t 0<br />
<br />
There is a lot more to ''tmux'' than that but that's the important part. <br />
<br />
For more, see:<br />
<br />
https://robots.thoughtbot.com/a-tmux-crash-course<br />
<br />
== Other gotchas ==<br />
<br />
=== Who else is using the computer? ===<br />
<br />
Try running<br />
<br />
who<br />
<br />
Or<br />
<br />
nvidia-smi<br />
<br />
Or<br />
<br />
top<br />
<br />
=== Temperature ===<br />
<br />
The GPUs are set to slowdown at 93C and shutdown at 96C. Idle temperature should be about 50C.<br />
To see the current temperature information in full run:<br />
nvidia-smi -q -d temperature<br />
<br />
It should never reach that temperature. If it does that's a big whoops.<br />
<br />
I have noticed that the first GPU (GPU 0) gets hotter quicker, presumably due to its physical location. With this information, it might be preferable to use GPU 1 more of the time.<br />
<br />
=== GPU and Memory Allocation ===<br />
<br />
Multiple kernels and users can also run on one GPU. This may mean you don't have enough memory at some point. It is possible to set it so that a GPU can only be accessed by one user at a time. This may be necessary in the future to ensure there is enough memory for those big jobs. <br />
<br />
Tensorflow actually seems to claim all the memory on all the GPUs so it might be a nice gentle thing for other users for you to make tensorflow be a bit nicer.<br />
<br />
==== Hiding unused GPUs from your program ====<br />
<br />
One suggestion is to use<br />
<br />
nvidia-smi<br />
<br />
to check which GPUs are available first then hide the one that you don't need from your program, using <br />
<br />
CUDA_VISIBLE_DEVICES=0 yourprogram<br />
<br />
if you want to use the 1st GPU or<br />
<br />
CUDA_VISIBLE_DEVICES=1 yourprogram <br />
<br />
if you want to use the 2nd GPU.<br />
<br />
==== Restricting GPU memory used by your tensorflow program ====<br />
<br />
Another approach when using tensorlow is to only use a certain amount of memory as described in the following answer: https://stackoverflow.com/a/34200194/1616231<br />
<br />
==== Automatic memory claiming ====<br />
<br />
Alternatively you may make tensorflow take memory as it is needed by taking the steps described in the following answer: https://stackoverflow.com/a/37454574/1616231 (though this will ultimately use more memory).<br />
<br />
Non-tensorflow apparoaches may have different characteristics, so the first option (hiding GPUs from your program) is the most general.<br />
<br />
= More Information =<br />
<br />
== Server Construction ==<br />
<br />
* 1 x [http://ark.intel.com/products/88195/Intel-Core-i7-6700K-Processor-8M-Cache-up-to-4_20-GHz INTEL i7-6700K]<br />
* 2 x [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ GTX 1080ti GPU]<br />
* 1 x [https://www.asus.com/us/Motherboards/ROG-MAXIMUS-IX-HERO/specifications/ ASUS ROG MAXIMUS IX HERO]<br />
* 2 x CORSAIR 32GB (2x16) D4 3000Mhz CMU32GX4M2C3000C15 (total 64GB)<br />
* 1 x CORSAIR CP-9020094-EU 1000W PSU<br />
* 1 x [https://en.sharkoon.com/product/1678/M25-W#desc Sharkoon M25-W - Mini tower ATX 5.25"]<br />
* 1 x 3TB HDD<br />
* 1 x 256GB SSD<br />
<br />
Built by [http://www.uzmanlarpc.com/ Uzmanlar PC]<br />
<br />
== Software ==<br />
<br />
=== OS ===<br />
<br />
[http://kubuntu.org/getkubuntu/ Kubuntu 16.04.3 LTS]<br />
<br />
=== Graphics Drivers ===<br />
<br />
[http://www.nvidia.com/Download/driverResults.aspx/120917/en-us Nvidia 384.59 drivers installed using runfile NVIDIA-Linux-x86_64-384.59.run]<br />
<br />
Installed using (to keep using the integrated graphics as main display graphics):<br />
<br />
sudo ./NVIDIA-Linux-x86_64-370.28.run --no-opengl-files --no-x-check --disable-nouveau<br />
<br />
=== CUDA Drivers ===<br />
<br />
Installed using <br />
<br />
cuda_8.0.61.2_linux.run</div>Imadhttp://deepdepth.djduff.net/wiki/index.php?title=NautilusServer&diff=173NautilusServer2018-01-31T18:09:37Z<p>Imad: /* SSH help */</p>
<hr />
<div>= Help = <br />
<br />
== Accessing ==<br />
<br />
'''Access from''': ITU or [http://bidb.itu.edu.tr/hizmetler/vpn ITU VPN].<br />
<br />
=== VPN Help ===<br />
<br />
* [http://bidb.itu.edu.tr/hizmetler/vpn BIDB has instructions for accessing the VPN.]<br />
* I have also provided here [http://files.djduff.net/scripts/ a script for accessing the ITU VPN from Ubuntu (ituvpn.sh)] (tested on Kubuntu 16.04).<br />
<br />
=== SSH help ===<br />
<br />
The SSH command to connect from a Unix environment:<br />
<br />
ssh -X -p 1542 hossein@<SERVER_IP_ADDRESS><br />
<br />
Switch meanings:<br />
* '''-p 1542''': connect on port 1542<br />
* '''-X:''' This allows you to run X applications. Omit it if you will be pure command line. You can also run X applications from Windows but you will need to install an X server on your Windows machine.<br />
<br />
'''Note''': to check that X forwarding is working, once you have connected, try running on the server the command:<br />
xeyes<br />
Or:<br />
dolphin<br />
<br />
You could for example run ''spyder'' like this. But there can be some latency across the network.<br />
<br />
=== Port Tunnelling ===<br />
You can use SSH command to tunnel your port into server port in order to access several applications (e.g. Tensorboard) from your machine. <br />
The option that is used for tunneling is:<br />
-L <SERVER PORT>:localhost:<YOUR PORT><br />
<br />
For example:<br />
<br />
ssh -p 1542 -L 6006:localhost:6006 alican@<SERVER-IP-ADDRESS><br />
<br />
== Setting up a deep learning environment ==<br />
<br />
<br />
=== Install anaconda ===<br />
<br />
export ANACONDA_PATH_PARENT=$HOME/software<br />
export ANACONDA_PATH=$ANACONDA_PATH_PARENT/anaconda3<br />
export ANACONDA_INSTALLER=Anaconda3-4.3.1-Linux-x86_64.sh<br />
<br />
mkdir -p ~/tmp<br />
cd ~/tmp<br />
mkdir -p $ANACONDA_PATH_PARENT<br />
wget https://repo.continuum.io/archive/$ANACONDA_INSTALLER<br />
bash $ANACONDA_INSTALLER -b -p $ANACONDA_PATH<br />
<br />
export PATH=$ANACONDA_PATH/bin:$PATH<br />
echo PATH: $PATH<br />
echo >> ~/.bashrc<br />
echo export PATH=$ANACONDA_PATH/bin:\$PATH >> ~/.bashrc<br />
<br />
=== Install tensorflow and keras ===<br />
<br />
These will be installed in a [https://conda.io/docs/intro.html conda] environment called '''deep''':<br />
<br />
export ENVNAME=deep<br />
conda create --name $ENVNAME<br />
source activate $ENVNAME<br />
conda install theano keras tensorflow tensorflow-gpu opencv pillow spyder matplotlib<br />
<br />
To check Keras is working:<br />
<br />
python -c "from keras.models import Sequential;Sequential()"<br />
<br />
To check the GPU is working with tensorflow, check this first (it should list two GPUs):<br />
<br />
nvidia-smi<br />
<br />
Then make sure the following script runs and finds one CPU and two GPUS: https://bitbucket.org/damienjadeduff/uhem_keras_tf/src/master/sariyer_python3/test_tf_gpu.py<br />
<br />
Run it like this:<br />
<br />
python test_tf_gpu.py<br />
<br />
'''Warning:''' for different versions of Tensorflow, Keras or Theano you may need to use '''[https://pip.pypa.io/en/latest/user_guide/#installing-packages pip]''' to install the version you need in an environment.<br />
<br />
== Easy file access (Linux) ==<br />
<br />
This can be useful for getting files on and off the server by accessing your remote home directory as if it was on your local computer (mounted on your file system).<br />
<br />
On YOUR Linux computer run:<br />
<br />
sudo apt-get install sshfs<br />
targ=~/remote/nautilus<br />
mkdir -p $targ<br />
fusermount -u $targ # only necessary to unmount if already tried<br />
sshfs -p 1542 -o workaround=rename YOUR_SERVER_USERNAME@SERVER_IP_ADDRESS:/home/YOUR_SERVER_USERNAME $targ<br />
<br />
Note: if parts of your system hang because the connection to the ssh server gets stale (a common problem), just do:<br />
<br />
killall ssfs<br />
<br />
It should resolve most of your problems.<br />
<br />
== Using the SSD ==<br />
<br />
There is an SSD drive installed. This drive is automatically mounted at:<br />
<br />
/media/FASTDATA1<br />
<br />
The drive belongs to user '''root''' and group '''fastdata1'''.<br />
If you cannot access it you need to get an admin (Hossein) to add you to the group with the command:<br />
<br />
sudo usermod -aG fastdata1 YOURUSERNAME<br />
<br />
And to add you a folder in there with the right permissions:<br />
<br />
sudo mkdir /media/FASTDATA1/YOURUSERNAME<br />
sudo chown YOURUSERNAME:YOURUSERNAME /media/FASTDATA1/YOURUSERNAME<br />
sudo chmod 700 /media/FASTDATA1/YOURUSERNAME<br />
<br />
== Compiling your own CUDA programs ==<br />
<br />
To do this, add the following lines to your '''.bashrc''' file in your home folder:<br />
<br />
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH<br />
export PATH=/usr/local/cuda/bin:$PATH<br />
<br />
== Running programs after you log out ==<br />
<br />
Sometimes a process may take a while and if you happen to log out the lack of interactivity may cause that process to give up on you. <br />
<br />
To get around this, people use the tool '''screen''' or later versions like '''tmux'''.<br />
<br />
You log in over ssh, then type:<br />
<br />
tmux<br />
<br />
This will put you in a little session-in-a-session. Even if you log out of ssh this tmux session will persist and you can get back to it by again typing ''tmux''.<br />
<br />
Once you've finished writing your commands (e.g. '''python my_python_program_that_takes_3_days.py'''), and started it running, you can get back to an ordinary shell session by typing: <br />
<br />
Ctrl-b d<br />
<br />
(that is, a '''Ctrl-b''' followed by a '''d''').<br />
<br />
Now if you want to get back to your previous session, just run ''tmux'' again. Actually, to be sure you connect to the right session, run:<br />
<br />
tmux list-sessions<br />
<br />
See the session number of the session to which you want to connect (e.g. '''0''') then to connect to it, run:<br />
<br />
tmux attach -t 0<br />
<br />
There is a lot more to ''tmux'' than that but that's the important part. <br />
<br />
For more, see:<br />
<br />
https://robots.thoughtbot.com/a-tmux-crash-course<br />
<br />
== Other gotchas ==<br />
<br />
=== Who else is using the computer? ===<br />
<br />
Try running<br />
<br />
who<br />
<br />
Or<br />
<br />
nvidia-smi<br />
<br />
Or<br />
<br />
top<br />
<br />
=== Temperature ===<br />
<br />
The GPUs are set to slowdown at 93C and shutdown at 96C. Idle temperature should be about 50C.<br />
To see the current temperature information in full run:<br />
nvidia-smi -q -d temperature<br />
<br />
It should never reach that temperature. If it does that's a big whoops.<br />
<br />
I have noticed that the first GPU (GPU 0) gets hotter quicker, presumably due to its physical location. With this information, it might be preferable to use GPU 1 more of the time.<br />
<br />
=== GPU and Memory Allocation ===<br />
<br />
Multiple kernels and users can also run on one GPU. This may mean you don't have enough memory at some point. It is possible to set it so that a GPU can only be accessed by one user at a time. This may be necessary in the future to ensure there is enough memory for those big jobs. <br />
<br />
Tensorflow actually seems to claim all the memory on all the GPUs so it might be a nice gentle thing for other users for you to make tensorflow be a bit nicer.<br />
<br />
==== Hiding unused GPUs from your program ====<br />
<br />
One suggestion is to use<br />
<br />
nvidia-smi<br />
<br />
to check which GPUs are available first then hide the one that you don't need from your program, using <br />
<br />
CUDA_VISIBLE_DEVICES=0 yourprogram<br />
<br />
if you want to use the 1st GPU or<br />
<br />
CUDA_VISIBLE_DEVICES=1 yourprogram <br />
<br />
if you want to use the 2nd GPU.<br />
<br />
==== Restricting GPU memory used by your tensorflow program ====<br />
<br />
Another approach when using tensorlow is to only use a certain amount of memory as described in the following answer: https://stackoverflow.com/a/34200194/1616231<br />
<br />
==== Automatic memory claiming ====<br />
<br />
Alternatively you may make tensorflow take memory as it is needed by taking the steps described in the following answer: https://stackoverflow.com/a/37454574/1616231 (though this will ultimately use more memory).<br />
<br />
Non-tensorflow apparoaches may have different characteristics, so the first option (hiding GPUs from your program) is the most general.<br />
<br />
= More Information =<br />
<br />
== Server Construction ==<br />
<br />
* 1 x [http://ark.intel.com/products/88195/Intel-Core-i7-6700K-Processor-8M-Cache-up-to-4_20-GHz INTEL i7-6700K]<br />
* 2 x [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ GTX 1080ti GPU]<br />
* 1 x [https://www.asus.com/us/Motherboards/ROG-MAXIMUS-IX-HERO/specifications/ ASUS ROG MAXIMUS IX HERO]<br />
* 2 x CORSAIR 32GB (2x16) D4 3000Mhz CMU32GX4M2C3000C15 (total 64GB)<br />
* 1 x CORSAIR CP-9020094-EU 1000W PSU<br />
* 1 x [https://en.sharkoon.com/product/1678/M25-W#desc Sharkoon M25-W - Mini tower ATX 5.25"]<br />
* 1 x 3TB HDD<br />
* 1 x 256GB SSD<br />
<br />
Built by [http://www.uzmanlarpc.com/ Uzmanlar PC]<br />
<br />
== Software ==<br />
<br />
=== OS ===<br />
<br />
[http://kubuntu.org/getkubuntu/ Kubuntu 16.04.3 LTS]<br />
<br />
=== Graphics Drivers ===<br />
<br />
[http://www.nvidia.com/Download/driverResults.aspx/120917/en-us Nvidia 384.59 drivers installed using runfile NVIDIA-Linux-x86_64-384.59.run]<br />
<br />
Installed using (to keep using the integrated graphics as main display graphics):<br />
<br />
sudo ./NVIDIA-Linux-x86_64-370.28.run --no-opengl-files --no-x-check --disable-nouveau<br />
<br />
=== CUDA Drivers ===<br />
<br />
Installed using <br />
<br />
cuda_8.0.61.2_linux.run</div>Imadhttp://deepdepth.djduff.net/wiki/index.php?title=NautilusServer&diff=172NautilusServer2018-01-31T18:09:05Z<p>Imad: </p>
<hr />
<div>= Help = <br />
<br />
== Accessing ==<br />
<br />
'''Access from''': ITU or [http://bidb.itu.edu.tr/hizmetler/vpn ITU VPN].<br />
<br />
=== VPN Help ===<br />
<br />
* [http://bidb.itu.edu.tr/hizmetler/vpn BIDB has instructions for accessing the VPN.]<br />
* I have also provided here [http://files.djduff.net/scripts/ a script for accessing the ITU VPN from Ubuntu (ituvpn.sh)] (tested on Kubuntu 16.04).<br />
<br />
=== SSH help ===<br />
<br />
The SSH command to connect from a Unix environment:<br />
<br />
ssh -X -p 1542 hossein@<SERVER-IP-ADDRESS><br />
<br />
Switch meanings:<br />
* '''-p 1542''': connect on port 1542<br />
* '''-X:''' This allows you to run X applications. Omit it if you will be pure command line. You can also run X applications from Windows but you will need to install an X server on your Windows machine.<br />
<br />
'''Note''': to check that X forwarding is working, once you have connected, try running on the server the command:<br />
xeyes<br />
Or:<br />
dolphin<br />
<br />
You could for example run ''spyder'' like this. But there can be some latency across the network.<br />
<br />
=== Port Tunnelling ===<br />
You can use SSH command to tunnel your port into server port in order to access several applications (e.g. Tensorboard) from your machine. <br />
The option that is used for tunneling is:<br />
-L <SERVER PORT>:localhost:<YOUR PORT><br />
<br />
For example:<br />
<br />
ssh -p 1542 -L 6006:localhost:6006 alican@<SERVER-IP-ADDRESS><br />
<br />
== Setting up a deep learning environment ==<br />
<br />
<br />
=== Install anaconda ===<br />
<br />
export ANACONDA_PATH_PARENT=$HOME/software<br />
export ANACONDA_PATH=$ANACONDA_PATH_PARENT/anaconda3<br />
export ANACONDA_INSTALLER=Anaconda3-4.3.1-Linux-x86_64.sh<br />
<br />
mkdir -p ~/tmp<br />
cd ~/tmp<br />
mkdir -p $ANACONDA_PATH_PARENT<br />
wget https://repo.continuum.io/archive/$ANACONDA_INSTALLER<br />
bash $ANACONDA_INSTALLER -b -p $ANACONDA_PATH<br />
<br />
export PATH=$ANACONDA_PATH/bin:$PATH<br />
echo PATH: $PATH<br />
echo >> ~/.bashrc<br />
echo export PATH=$ANACONDA_PATH/bin:\$PATH >> ~/.bashrc<br />
<br />
=== Install tensorflow and keras ===<br />
<br />
These will be installed in a [https://conda.io/docs/intro.html conda] environment called '''deep''':<br />
<br />
export ENVNAME=deep<br />
conda create --name $ENVNAME<br />
source activate $ENVNAME<br />
conda install theano keras tensorflow tensorflow-gpu opencv pillow spyder matplotlib<br />
<br />
To check Keras is working:<br />
<br />
python -c "from keras.models import Sequential;Sequential()"<br />
<br />
To check the GPU is working with tensorflow, check this first (it should list two GPUs):<br />
<br />
nvidia-smi<br />
<br />
Then make sure the following script runs and finds one CPU and two GPUS: https://bitbucket.org/damienjadeduff/uhem_keras_tf/src/master/sariyer_python3/test_tf_gpu.py<br />
<br />
Run it like this:<br />
<br />
python test_tf_gpu.py<br />
<br />
'''Warning:''' for different versions of Tensorflow, Keras or Theano you may need to use '''[https://pip.pypa.io/en/latest/user_guide/#installing-packages pip]''' to install the version you need in an environment.<br />
<br />
== Easy file access (Linux) ==<br />
<br />
This can be useful for getting files on and off the server by accessing your remote home directory as if it was on your local computer (mounted on your file system).<br />
<br />
On YOUR Linux computer run:<br />
<br />
sudo apt-get install sshfs<br />
targ=~/remote/nautilus<br />
mkdir -p $targ<br />
fusermount -u $targ # only necessary to unmount if already tried<br />
sshfs -p 1542 -o workaround=rename YOUR_SERVER_USERNAME@SERVER_IP_ADDRESS:/home/YOUR_SERVER_USERNAME $targ<br />
<br />
Note: if parts of your system hang because the connection to the ssh server gets stale (a common problem), just do:<br />
<br />
killall ssfs<br />
<br />
It should resolve most of your problems.<br />
<br />
== Using the SSD ==<br />
<br />
There is an SSD drive installed. This drive is automatically mounted at:<br />
<br />
/media/FASTDATA1<br />
<br />
The drive belongs to user '''root''' and group '''fastdata1'''.<br />
If you cannot access it you need to get an admin (Hossein) to add you to the group with the command:<br />
<br />
sudo usermod -aG fastdata1 YOURUSERNAME<br />
<br />
And to add you a folder in there with the right permissions:<br />
<br />
sudo mkdir /media/FASTDATA1/YOURUSERNAME<br />
sudo chown YOURUSERNAME:YOURUSERNAME /media/FASTDATA1/YOURUSERNAME<br />
sudo chmod 700 /media/FASTDATA1/YOURUSERNAME<br />
<br />
== Compiling your own CUDA programs ==<br />
<br />
To do this, add the following lines to your '''.bashrc''' file in your home folder:<br />
<br />
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH<br />
export PATH=/usr/local/cuda/bin:$PATH<br />
<br />
== Running programs after you log out ==<br />
<br />
Sometimes a process may take a while and if you happen to log out the lack of interactivity may cause that process to give up on you. <br />
<br />
To get around this, people use the tool '''screen''' or later versions like '''tmux'''.<br />
<br />
You log in over ssh, then type:<br />
<br />
tmux<br />
<br />
This will put you in a little session-in-a-session. Even if you log out of ssh this tmux session will persist and you can get back to it by again typing ''tmux''.<br />
<br />
Once you've finished writing your commands (e.g. '''python my_python_program_that_takes_3_days.py'''), and started it running, you can get back to an ordinary shell session by typing: <br />
<br />
Ctrl-b d<br />
<br />
(that is, a '''Ctrl-b''' followed by a '''d''').<br />
<br />
Now if you want to get back to your previous session, just run ''tmux'' again. Actually, to be sure you connect to the right session, run:<br />
<br />
tmux list-sessions<br />
<br />
See the session number of the session to which you want to connect (e.g. '''0''') then to connect to it, run:<br />
<br />
tmux attach -t 0<br />
<br />
There is a lot more to ''tmux'' than that but that's the important part. <br />
<br />
For more, see:<br />
<br />
https://robots.thoughtbot.com/a-tmux-crash-course<br />
<br />
== Other gotchas ==<br />
<br />
=== Who else is using the computer? ===<br />
<br />
Try running<br />
<br />
who<br />
<br />
Or<br />
<br />
nvidia-smi<br />
<br />
Or<br />
<br />
top<br />
<br />
=== Temperature ===<br />
<br />
The GPUs are set to slowdown at 93C and shutdown at 96C. Idle temperature should be about 50C.<br />
To see the current temperature information in full run:<br />
nvidia-smi -q -d temperature<br />
<br />
It should never reach that temperature. If it does that's a big whoops.<br />
<br />
I have noticed that the first GPU (GPU 0) gets hotter quicker, presumably due to its physical location. With this information, it might be preferable to use GPU 1 more of the time.<br />
<br />
=== GPU and Memory Allocation ===<br />
<br />
Multiple kernels and users can also run on one GPU. This may mean you don't have enough memory at some point. It is possible to set it so that a GPU can only be accessed by one user at a time. This may be necessary in the future to ensure there is enough memory for those big jobs. <br />
<br />
Tensorflow actually seems to claim all the memory on all the GPUs so it might be a nice gentle thing for other users for you to make tensorflow be a bit nicer.<br />
<br />
==== Hiding unused GPUs from your program ====<br />
<br />
One suggestion is to use<br />
<br />
nvidia-smi<br />
<br />
to check which GPUs are available first then hide the one that you don't need from your program, using <br />
<br />
CUDA_VISIBLE_DEVICES=0 yourprogram<br />
<br />
if you want to use the 1st GPU or<br />
<br />
CUDA_VISIBLE_DEVICES=1 yourprogram <br />
<br />
if you want to use the 2nd GPU.<br />
<br />
==== Restricting GPU memory used by your tensorflow program ====<br />
<br />
Another approach when using tensorlow is to only use a certain amount of memory as described in the following answer: https://stackoverflow.com/a/34200194/1616231<br />
<br />
==== Automatic memory claiming ====<br />
<br />
Alternatively you may make tensorflow take memory as it is needed by taking the steps described in the following answer: https://stackoverflow.com/a/37454574/1616231 (though this will ultimately use more memory).<br />
<br />
Non-tensorflow apparoaches may have different characteristics, so the first option (hiding GPUs from your program) is the most general.<br />
<br />
= More Information =<br />
<br />
== Server Construction ==<br />
<br />
* 1 x [http://ark.intel.com/products/88195/Intel-Core-i7-6700K-Processor-8M-Cache-up-to-4_20-GHz INTEL i7-6700K]<br />
* 2 x [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ GTX 1080ti GPU]<br />
* 1 x [https://www.asus.com/us/Motherboards/ROG-MAXIMUS-IX-HERO/specifications/ ASUS ROG MAXIMUS IX HERO]<br />
* 2 x CORSAIR 32GB (2x16) D4 3000Mhz CMU32GX4M2C3000C15 (total 64GB)<br />
* 1 x CORSAIR CP-9020094-EU 1000W PSU<br />
* 1 x [https://en.sharkoon.com/product/1678/M25-W#desc Sharkoon M25-W - Mini tower ATX 5.25"]<br />
* 1 x 3TB HDD<br />
* 1 x 256GB SSD<br />
<br />
Built by [http://www.uzmanlarpc.com/ Uzmanlar PC]<br />
<br />
== Software ==<br />
<br />
=== OS ===<br />
<br />
[http://kubuntu.org/getkubuntu/ Kubuntu 16.04.3 LTS]<br />
<br />
=== Graphics Drivers ===<br />
<br />
[http://www.nvidia.com/Download/driverResults.aspx/120917/en-us Nvidia 384.59 drivers installed using runfile NVIDIA-Linux-x86_64-384.59.run]<br />
<br />
Installed using (to keep using the integrated graphics as main display graphics):<br />
<br />
sudo ./NVIDIA-Linux-x86_64-370.28.run --no-opengl-files --no-x-check --disable-nouveau<br />
<br />
=== CUDA Drivers ===<br />
<br />
Installed using <br />
<br />
cuda_8.0.61.2_linux.run</div>Imadhttp://deepdepth.djduff.net/wiki/index.php?title=NautilusServer&diff=171NautilusServer2018-01-31T18:08:23Z<p>Imad: /* Port Tunnelling */</p>
<hr />
<div>= Help = <br />
<br />
== Accessing ==<br />
<br />
'''Access from''': ITU or [http://bidb.itu.edu.tr/hizmetler/vpn ITU VPN].<br />
<br />
=== VPN Help ===<br />
<br />
* [http://bidb.itu.edu.tr/hizmetler/vpn BIDB has instructions for accessing the VPN.]<br />
* I have also provided here [http://files.djduff.net/scripts/ a script for accessing the ITU VPN from Ubuntu (ituvpn.sh)] (tested on Kubuntu 16.04).<br />
<br />
=== SSH help ===<br />
<br />
The SSH command to connect from a Unix environment:<br />
<br />
ssh -X -p 1542 hossein@<SERVER-IP-ADDRESS><br />
<br />
Switch meanings:<br />
* '''-p 1542''': connect on port 1542<br />
* '''-X:''' This allows you to run X applications. Omit it if you will be pure command line. You can also run X applications from Windows but you will need to install an X server on your Windows machine.<br />
<br />
'''Note''': to check that X forwarding is working, once you have connected, try running on the server the command:<br />
xeyes<br />
Or:<br />
dolphin<br />
<br />
You could for example run ''spyder'' like this. But there can be some latency across the network.<br />
<br />
=== Port Tunnelling ===<br />
You can use SSH command to tunnel your port into server port in order to access several applications (e.g. Tensorboard) from your machine. <br />
The option that is used for tunneling is:<br />
-L <SERVER PORT>:localhost:<YOUR PORT><br />
<br />
For example:<br />
<br />
ssh -p 1542 -L 6006:localhost:6006 alican@<SERVER-IP-ADDRESS><br />
<br />
== Setting up a deep learning environment ==<br />
<br />
<br />
=== Install anaconda ===<br />
<br />
export ANACONDA_PATH_PARENT=$HOME/software<br />
export ANACONDA_PATH=$ANACONDA_PATH_PARENT/anaconda3<br />
export ANACONDA_INSTALLER=Anaconda3-4.3.1-Linux-x86_64.sh<br />
<br />
mkdir -p ~/tmp<br />
cd ~/tmp<br />
mkdir -p $ANACONDA_PATH_PARENT<br />
wget https://repo.continuum.io/archive/$ANACONDA_INSTALLER<br />
bash $ANACONDA_INSTALLER -b -p $ANACONDA_PATH<br />
<br />
export PATH=$ANACONDA_PATH/bin:$PATH<br />
echo PATH: $PATH<br />
echo >> ~/.bashrc<br />
echo export PATH=$ANACONDA_PATH/bin:\$PATH >> ~/.bashrc<br />
<br />
=== Install tensorflow and keras ===<br />
<br />
These will be installed in a [https://conda.io/docs/intro.html conda] environment called '''deep''':<br />
<br />
export ENVNAME=deep<br />
conda create --name $ENVNAME<br />
source activate $ENVNAME<br />
conda install theano keras tensorflow tensorflow-gpu opencv pillow spyder matplotlib<br />
<br />
To check Keras is working:<br />
<br />
python -c "from keras.models import Sequential;Sequential()"<br />
<br />
To check the GPU is working with tensorflow, check this first (it should list two GPUs):<br />
<br />
nvidia-smi<br />
<br />
Then make sure the following script runs and finds one CPU and two GPUS: https://bitbucket.org/damienjadeduff/uhem_keras_tf/src/master/sariyer_python3/test_tf_gpu.py<br />
<br />
Run it like this:<br />
<br />
python test_tf_gpu.py<br />
<br />
'''Warning:''' for different versions of Tensorflow, Keras or Theano you may need to use '''[https://pip.pypa.io/en/latest/user_guide/#installing-packages pip]''' to install the version you need in an environment.<br />
<br />
== Easy file access (Linux) ==<br />
<br />
This can be useful for getting files on and off the server by accessing your remote home directory as if it was on your local computer (mounted on your file system).<br />
<br />
On YOUR Linux computer run:<br />
<br />
sudo apt-get install sshfs<br />
targ=~/remote/nautilus<br />
mkdir -p $targ<br />
fusermount -u $targ # only necessary to unmount if already tried<br />
sshfs -p 1542 -o workaround=rename YOUR_SERVER_USERNAME@160.75.27.83:/home/YOUR_SERVER_USERNAME $targ<br />
<br />
Note: if parts of your system hang because the connection to the ssh server gets stale (a common problem), just do:<br />
<br />
killall ssfs<br />
<br />
It should resolve most of your problems.<br />
<br />
== Using the SSD ==<br />
<br />
There is an SSD drive installed. This drive is automatically mounted at:<br />
<br />
/media/FASTDATA1<br />
<br />
The drive belongs to user '''root''' and group '''fastdata1'''.<br />
If you cannot access it you need to get an admin (Hossein) to add you to the group with the command:<br />
<br />
sudo usermod -aG fastdata1 YOURUSERNAME<br />
<br />
And to add you a folder in there with the right permissions:<br />
<br />
sudo mkdir /media/FASTDATA1/YOURUSERNAME<br />
sudo chown YOURUSERNAME:YOURUSERNAME /media/FASTDATA1/YOURUSERNAME<br />
sudo chmod 700 /media/FASTDATA1/YOURUSERNAME<br />
<br />
== Compiling your own CUDA programs ==<br />
<br />
To do this, add the following lines to your '''.bashrc''' file in your home folder:<br />
<br />
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH<br />
export PATH=/usr/local/cuda/bin:$PATH<br />
<br />
== Running programs after you log out ==<br />
<br />
Sometimes a process may take a while and if you happen to log out the lack of interactivity may cause that process to give up on you. <br />
<br />
To get around this, people use the tool '''screen''' or later versions like '''tmux'''.<br />
<br />
You log in over ssh, then type:<br />
<br />
tmux<br />
<br />
This will put you in a little session-in-a-session. Even if you log out of ssh this tmux session will persist and you can get back to it by again typing ''tmux''.<br />
<br />
Once you've finished writing your commands (e.g. '''python my_python_program_that_takes_3_days.py'''), and started it running, you can get back to an ordinary shell session by typing: <br />
<br />
Ctrl-b d<br />
<br />
(that is, a '''Ctrl-b''' followed by a '''d''').<br />
<br />
Now if you want to get back to your previous session, just run ''tmux'' again. Actually, to be sure you connect to the right session, run:<br />
<br />
tmux list-sessions<br />
<br />
See the session number of the session to which you want to connect (e.g. '''0''') then to connect to it, run:<br />
<br />
tmux attach -t 0<br />
<br />
There is a lot more to ''tmux'' than that but that's the important part. <br />
<br />
For more, see:<br />
<br />
https://robots.thoughtbot.com/a-tmux-crash-course<br />
<br />
== Other gotchas ==<br />
<br />
=== Who else is using the computer? ===<br />
<br />
Try running<br />
<br />
who<br />
<br />
Or<br />
<br />
nvidia-smi<br />
<br />
Or<br />
<br />
top<br />
<br />
=== Temperature ===<br />
<br />
The GPUs are set to slowdown at 93C and shutdown at 96C. Idle temperature should be about 50C.<br />
To see the current temperature information in full run:<br />
nvidia-smi -q -d temperature<br />
<br />
It should never reach that temperature. If it does that's a big whoops.<br />
<br />
I have noticed that the first GPU (GPU 0) gets hotter quicker, presumably due to its physical location. With this information, it might be preferable to use GPU 1 more of the time.<br />
<br />
=== GPU and Memory Allocation ===<br />
<br />
Multiple kernels and users can also run on one GPU. This may mean you don't have enough memory at some point. It is possible to set it so that a GPU can only be accessed by one user at a time. This may be necessary in the future to ensure there is enough memory for those big jobs. <br />
<br />
Tensorflow actually seems to claim all the memory on all the GPUs so it might be a nice gentle thing for other users for you to make tensorflow be a bit nicer.<br />
<br />
==== Hiding unused GPUs from your program ====<br />
<br />
One suggestion is to use<br />
<br />
nvidia-smi<br />
<br />
to check which GPUs are available first then hide the one that you don't need from your program, using <br />
<br />
CUDA_VISIBLE_DEVICES=0 yourprogram<br />
<br />
if you want to use the 1st GPU or<br />
<br />
CUDA_VISIBLE_DEVICES=1 yourprogram <br />
<br />
if you want to use the 2nd GPU.<br />
<br />
==== Restricting GPU memory used by your tensorflow program ====<br />
<br />
Another approach when using tensorlow is to only use a certain amount of memory as described in the following answer: https://stackoverflow.com/a/34200194/1616231<br />
<br />
==== Automatic memory claiming ====<br />
<br />
Alternatively you may make tensorflow take memory as it is needed by taking the steps described in the following answer: https://stackoverflow.com/a/37454574/1616231 (though this will ultimately use more memory).<br />
<br />
Non-tensorflow apparoaches may have different characteristics, so the first option (hiding GPUs from your program) is the most general.<br />
<br />
= More Information =<br />
<br />
== Server Construction ==<br />
<br />
* 1 x [http://ark.intel.com/products/88195/Intel-Core-i7-6700K-Processor-8M-Cache-up-to-4_20-GHz INTEL i7-6700K]<br />
* 2 x [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ GTX 1080ti GPU]<br />
* 1 x [https://www.asus.com/us/Motherboards/ROG-MAXIMUS-IX-HERO/specifications/ ASUS ROG MAXIMUS IX HERO]<br />
* 2 x CORSAIR 32GB (2x16) D4 3000Mhz CMU32GX4M2C3000C15 (total 64GB)<br />
* 1 x CORSAIR CP-9020094-EU 1000W PSU<br />
* 1 x [https://en.sharkoon.com/product/1678/M25-W#desc Sharkoon M25-W - Mini tower ATX 5.25"]<br />
* 1 x 3TB HDD<br />
* 1 x 256GB SSD<br />
<br />
Built by [http://www.uzmanlarpc.com/ Uzmanlar PC]<br />
<br />
== Software ==<br />
<br />
=== OS ===<br />
<br />
[http://kubuntu.org/getkubuntu/ Kubuntu 16.04.3 LTS]<br />
<br />
=== Graphics Drivers ===<br />
<br />
[http://www.nvidia.com/Download/driverResults.aspx/120917/en-us Nvidia 384.59 drivers installed using runfile NVIDIA-Linux-x86_64-384.59.run]<br />
<br />
Installed using (to keep using the integrated graphics as main display graphics):<br />
<br />
sudo ./NVIDIA-Linux-x86_64-370.28.run --no-opengl-files --no-x-check --disable-nouveau<br />
<br />
=== CUDA Drivers ===<br />
<br />
Installed using <br />
<br />
cuda_8.0.61.2_linux.run</div>Imadhttp://deepdepth.djduff.net/wiki/index.php?title=NautilusServer&diff=168NautilusServer2018-01-31T13:36:18Z<p>Imad: /* Accessing */</p>
<hr />
<div>= Help = <br />
<br />
== Accessing ==<br />
<br />
'''Access from''': ITU or [http://bidb.itu.edu.tr/hizmetler/vpn ITU VPN].<br />
<br />
=== VPN Help ===<br />
<br />
* [http://bidb.itu.edu.tr/hizmetler/vpn BIDB has instructions for accessing the VPN.]<br />
* I have also provided here [http://files.djduff.net/scripts/ a script for accessing the ITU VPN from Ubuntu (ituvpn.sh)] (tested on Kubuntu 16.04).<br />
<br />
=== SSH help ===<br />
<br />
The SSH command to connect from a Unix environment:<br />
<br />
ssh -X -p 1542 hossein@<SERVER-IP-ADDRESS><br />
<br />
Switch meanings:<br />
* '''-p 1542''': connect on port 1542<br />
* '''-X:''' This allows you to run X applications. Omit it if you will be pure command line. You can also run X applications from Windows but you will need to install an X server on your Windows machine.<br />
<br />
'''Note''': to check that X forwarding is working, once you have connected, try running on the server the command:<br />
xeyes<br />
Or:<br />
dolphin<br />
<br />
You could for example run ''spyder'' like this. But there can be some latency across the network.<br />
<br />
=== Port Tunnelling ===<br />
You can use SSH command to tunnel your port into server port in order to access several application from your machine. (e.g. Tensorboard)<br />
The option that is used for tunneling is:<br />
-L <SERVER PORT>:localhost:<YOUR PORT><br />
<br />
For example:<br />
<br />
ssh -p 1542 -L 6006:localhost:6006 alican@<SERVER-IP-ADDRESS><br />
<br />
== Setting up a deep learning environment ==<br />
<br />
<br />
=== Install anaconda ===<br />
<br />
export ANACONDA_PATH_PARENT=$HOME/software<br />
export ANACONDA_PATH=$ANACONDA_PATH_PARENT/anaconda3<br />
export ANACONDA_INSTALLER=Anaconda3-4.3.1-Linux-x86_64.sh<br />
<br />
mkdir -p ~/tmp<br />
cd ~/tmp<br />
mkdir -p $ANACONDA_PATH_PARENT<br />
wget https://repo.continuum.io/archive/$ANACONDA_INSTALLER<br />
bash $ANACONDA_INSTALLER -b -p $ANACONDA_PATH<br />
<br />
export PATH=$ANACONDA_PATH/bin:$PATH<br />
echo PATH: $PATH<br />
echo >> ~/.bashrc<br />
echo export PATH=$ANACONDA_PATH/bin:\$PATH >> ~/.bashrc<br />
<br />
=== Install tensorflow and keras ===<br />
<br />
These will be installed in a [https://conda.io/docs/intro.html conda] environment called '''deep''':<br />
<br />
export ENVNAME=deep<br />
conda create --name $ENVNAME<br />
source activate $ENVNAME<br />
conda install theano keras tensorflow tensorflow-gpu opencv pillow spyder matplotlib<br />
<br />
To check Keras is working:<br />
<br />
python -c "from keras.models import Sequential;Sequential()"<br />
<br />
To check the GPU is working with tensorflow, check this first (it should list two GPUs):<br />
<br />
nvidia-smi<br />
<br />
Then make sure the following script runs and finds one CPU and two GPUS: https://bitbucket.org/damienjadeduff/uhem_keras_tf/src/master/sariyer_python3/test_tf_gpu.py<br />
<br />
Run it like this:<br />
<br />
python test_tf_gpu.py<br />
<br />
'''Warning:''' for different versions of Tensorflow, Keras or Theano you may need to use '''[https://pip.pypa.io/en/latest/user_guide/#installing-packages pip]''' to install the version you need in an environment.<br />
<br />
== Easy file access (Linux) ==<br />
<br />
This can be useful for getting files on and off the server by accessing your remote home directory as if it was on your local computer (mounted on your file system).<br />
<br />
On YOUR Linux computer run:<br />
<br />
sudo apt-get install sshfs<br />
targ=~/remote/nautilus<br />
mkdir -p $targ<br />
fusermount -u $targ # only necessary to unmount if already tried<br />
sshfs -p 1542 -o workaround=rename YOUR_SERVER_USERNAME@160.75.27.83:/home/YOUR_SERVER_USERNAME $targ<br />
<br />
Note: if parts of your system hang because the connection to the ssh server gets stale (a common problem), just do:<br />
<br />
killall ssfs<br />
<br />
It should resolve most of your problems.<br />
<br />
== Using the SSD ==<br />
<br />
There is an SSD drive installed. This drive is automatically mounted at:<br />
<br />
/media/FASTDATA1<br />
<br />
The drive belongs to user '''root''' and group '''fastdata1'''.<br />
If you cannot access it you need to get an admin (Hossein) to add you to the group with the command:<br />
<br />
sudo usermod -aG fastdata1 YOURUSERNAME<br />
<br />
And to add you a folder in there with the right permissions:<br />
<br />
sudo mkdir /media/FASTDATA1/YOURUSERNAME<br />
sudo chown YOURUSERNAME:YOURUSERNAME /media/FASTDATA1/YOURUSERNAME<br />
sudo chmod 700 /media/FASTDATA1/YOURUSERNAME<br />
<br />
== Compiling your own CUDA programs ==<br />
<br />
To do this, add the following lines to your '''.bashrc''' file in your home folder:<br />
<br />
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH<br />
export PATH=/usr/local/cuda/bin:$PATH<br />
<br />
== Running programs after you log out ==<br />
<br />
Sometimes a process may take a while and if you happen to log out the lack of interactivity may cause that process to give up on you. <br />
<br />
To get around this, people use the tool '''screen''' or later versions like '''tmux'''.<br />
<br />
You log in over ssh, then type:<br />
<br />
tmux<br />
<br />
This will put you in a little session-in-a-session. Even if you log out of ssh this tmux session will persist and you can get back to it by again typing ''tmux''.<br />
<br />
Once you've finished writing your commands (e.g. '''python my_python_program_that_takes_3_days.py'''), and started it running, you can get back to an ordinary shell session by typing: <br />
<br />
Ctrl-b d<br />
<br />
(that is, a '''Ctrl-b''' followed by a '''d''').<br />
<br />
Now if you want to get back to your previous session, just run ''tmux'' again. Actually, to be sure you connect to the right session, run:<br />
<br />
tmux list-sessions<br />
<br />
See the session number of the session to which you want to connect (e.g. '''0''') then to connect to it, run:<br />
<br />
tmux attach -t 0<br />
<br />
There is a lot more to ''tmux'' than that but that's the important part. <br />
<br />
For more, see:<br />
<br />
https://robots.thoughtbot.com/a-tmux-crash-course<br />
<br />
== Other gotchas ==<br />
<br />
=== Who else is using the computer? ===<br />
<br />
Try running<br />
<br />
who<br />
<br />
Or<br />
<br />
nvidia-smi<br />
<br />
Or<br />
<br />
top<br />
<br />
=== Temperature ===<br />
<br />
The GPUs are set to slowdown at 93C and shutdown at 96C. Idle temperature should be about 50C.<br />
To see the current temperature information in full run:<br />
nvidia-smi -q -d temperature<br />
<br />
It should never reach that temperature. If it does that's a big whoops.<br />
<br />
I have noticed that the first GPU (GPU 0) gets hotter quicker, presumably due to its physical location. With this information, it might be preferable to use GPU 1 more of the time.<br />
<br />
=== GPU and Memory Allocation ===<br />
<br />
Multiple kernels and users can also run on one GPU. This may mean you don't have enough memory at some point. It is possible to set it so that a GPU can only be accessed by one user at a time. This may be necessary in the future to ensure there is enough memory for those big jobs. <br />
<br />
Tensorflow actually seems to claim all the memory on all the GPUs so it might be a nice gentle thing for other users for you to make tensorflow be a bit nicer.<br />
<br />
==== Hiding unused GPUs from your program ====<br />
<br />
One suggestion is to use<br />
<br />
nvidia-smi<br />
<br />
to check which GPUs are available first then hide the one that you don't need from your program, using <br />
<br />
CUDA_VISIBLE_DEVICES=0 yourprogram<br />
<br />
if you want to use the 1st GPU or<br />
<br />
CUDA_VISIBLE_DEVICES=1 yourprogram <br />
<br />
if you want to use the 2nd GPU.<br />
<br />
==== Restricting GPU memory used by your tensorflow program ====<br />
<br />
Another approach when using tensorlow is to only use a certain amount of memory as described in the following answer: https://stackoverflow.com/a/34200194/1616231<br />
<br />
==== Automatic memory claiming ====<br />
<br />
Alternatively you may make tensorflow take memory as it is needed by taking the steps described in the following answer: https://stackoverflow.com/a/37454574/1616231 (though this will ultimately use more memory).<br />
<br />
Non-tensorflow apparoaches may have different characteristics, so the first option (hiding GPUs from your program) is the most general.<br />
<br />
= More Information =<br />
<br />
== Server Construction ==<br />
<br />
* 1 x [http://ark.intel.com/products/88195/Intel-Core-i7-6700K-Processor-8M-Cache-up-to-4_20-GHz INTEL i7-6700K]<br />
* 2 x [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ GTX 1080ti GPU]<br />
* 1 x [https://www.asus.com/us/Motherboards/ROG-MAXIMUS-IX-HERO/specifications/ ASUS ROG MAXIMUS IX HERO]<br />
* 2 x CORSAIR 32GB (2x16) D4 3000Mhz CMU32GX4M2C3000C15 (total 64GB)<br />
* 1 x CORSAIR CP-9020094-EU 1000W PSU<br />
* 1 x [https://en.sharkoon.com/product/1678/M25-W#desc Sharkoon M25-W - Mini tower ATX 5.25"]<br />
* 1 x 3TB HDD<br />
* 1 x 256GB SSD<br />
<br />
Built by [http://www.uzmanlarpc.com/ Uzmanlar PC]<br />
<br />
== Software ==<br />
<br />
=== OS ===<br />
<br />
[http://kubuntu.org/getkubuntu/ Kubuntu 16.04.3 LTS]<br />
<br />
=== Graphics Drivers ===<br />
<br />
[http://www.nvidia.com/Download/driverResults.aspx/120917/en-us Nvidia 384.59 drivers installed using runfile NVIDIA-Linux-x86_64-384.59.run]<br />
<br />
Installed using (to keep using the integrated graphics as main display graphics):<br />
<br />
sudo ./NVIDIA-Linux-x86_64-370.28.run --no-opengl-files --no-x-check --disable-nouveau<br />
<br />
=== CUDA Drivers ===<br />
<br />
Installed using <br />
<br />
cuda_8.0.61.2_linux.run</div>Imadhttp://deepdepth.djduff.net/wiki/index.php?title=NautilusServer&diff=167NautilusServer2018-01-31T13:35:13Z<p>Imad: port tunelling</p>
<hr />
<div>= Help = <br />
<br />
== Accessing ==<br />
<br />
'''Access from''': ITU or [http://bidb.itu.edu.tr/hizmetler/vpn ITU VPN].<br />
<br />
=== VPN Help ===<br />
<br />
* [http://bidb.itu.edu.tr/hizmetler/vpn BIDB has instructions for accessing the VPN.]<br />
* I have also provided here [http://files.djduff.net/scripts/ a script for accessing the ITU VPN from Ubuntu (ituvpn.sh)] (tested on Kubuntu 16.04).<br />
<br />
=== SSH help ===<br />
<br />
The SSH command to connect from a Unix environment:<br />
<br />
ssh -X -p 1542 hossein@<SERVER-IP-ADDRESS><br />
<br />
Switch meanings:<br />
* '''-p 1542''': connect on port 1542<br />
* '''-X:''' This allows you to run X applications. Omit it if you will be pure command line. You can also run X applications from Windows but you will need to install an X server on your Windows machine.<br />
<br />
'''Note''': to check that X forwarding is working, once you have connected, try running on the server the command:<br />
xeyes<br />
Or:<br />
dolphin<br />
<br />
You could for example run ''spyder'' like this. But there can be some latency across the network.<br />
<br />
=== Port Tunnelling ===<br />
You can use SSH command to tunnel your port into server port in order to access several application from your machine. (e.g. Tensorboard)<br />
The option that is used for tunneling is:<br />
-L <SERVER PORT>:localhost:<YOUR PORT><br />
<br />
For example:<br />
<br />
ssh -p 1542 -L 6006:localhost:6006 alican@<SERVER-IP-ADDRESS><br />
<br />
== Setting up a deep learning environment ==<br />
<br />
<br />
=== Install anaconda ===<br />
<br />
export ANACONDA_PATH_PARENT=$HOME/software<br />
export ANACONDA_PATH=$ANACONDA_PATH_PARENT/anaconda3<br />
export ANACONDA_INSTALLER=Anaconda3-4.3.1-Linux-x86_64.sh<br />
<br />
mkdir -p ~/tmp<br />
cd ~/tmp<br />
mkdir -p $ANACONDA_PATH_PARENT<br />
wget https://repo.continuum.io/archive/$ANACONDA_INSTALLER<br />
bash $ANACONDA_INSTALLER -b -p $ANACONDA_PATH<br />
<br />
export PATH=$ANACONDA_PATH/bin:$PATH<br />
echo PATH: $PATH<br />
echo >> ~/.bashrc<br />
echo export PATH=$ANACONDA_PATH/bin:\$PATH >> ~/.bashrc<br />
<br />
=== Install tensorflow and keras ===<br />
<br />
These will be installed in a [https://conda.io/docs/intro.html conda] environment called '''deep''':<br />
<br />
export ENVNAME=deep<br />
conda create --name $ENVNAME<br />
source activate $ENVNAME<br />
conda install theano keras tensorflow tensorflow-gpu opencv pillow spyder matplotlib<br />
<br />
To check Keras is working:<br />
<br />
python -c "from keras.models import Sequential;Sequential()"<br />
<br />
To check the GPU is working with tensorflow, check this first (it should list two GPUs):<br />
<br />
nvidia-smi<br />
<br />
Then make sure the following script runs and finds one CPU and two GPUS: https://bitbucket.org/damienjadeduff/uhem_keras_tf/src/master/sariyer_python3/test_tf_gpu.py<br />
<br />
Run it like this:<br />
<br />
python test_tf_gpu.py<br />
<br />
'''Warning:''' for different versions of Tensorflow, Keras or Theano you may need to use '''[https://pip.pypa.io/en/latest/user_guide/#installing-packages pip]''' to install the version you need in an environment.<br />
<br />
== Easy file access (Linux) ==<br />
<br />
This can be useful for getting files on and off the server by accessing your remote home directory as if it was on your local computer (mounted on your file system).<br />
<br />
On YOUR Linux computer run:<br />
<br />
sudo apt-get install sshfs<br />
targ=~/remote/nautilus<br />
mkdir -p $targ<br />
fusermount -u $targ # only necessary to unmount if already tried<br />
sshfs -p 1542 -o workaround=rename YOUR_SERVER_USERNAME@160.75.27.83:/home/YOUR_SERVER_USERNAME $targ<br />
<br />
Note: if parts of your system hang because the connection to the ssh server gets stale (a common problem), just do:<br />
<br />
killall ssfs<br />
<br />
It should resolve most of your problems.<br />
<br />
== Using the SSD ==<br />
<br />
There is an SSD drive installed. This drive is automatically mounted at:<br />
<br />
/media/FASTDATA1<br />
<br />
The drive belongs to user '''root''' and group '''fastdata1'''.<br />
If you cannot access it you need to get an admin (Hossein) to add you to the group with the command:<br />
<br />
sudo usermod -aG fastdata1 YOURUSERNAME<br />
<br />
And to add you a folder in there with the right permissions:<br />
<br />
sudo mkdir /media/FASTDATA1/YOURUSERNAME<br />
sudo chown YOURUSERNAME:YOURUSERNAME /media/FASTDATA1/YOURUSERNAME<br />
sudo chmod 700 /media/FASTDATA1/YOURUSERNAME<br />
<br />
== Compiling your own CUDA programs ==<br />
<br />
To do this, add the following lines to your '''.bashrc''' file in your home folder:<br />
<br />
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH<br />
export PATH=/usr/local/cuda/bin:$PATH<br />
<br />
== Running programs after you log out ==<br />
<br />
Sometimes a process may take a while and if you happen to log out the lack of interactivity may cause that process to give up on you. <br />
<br />
To get around this, people use the tool '''screen''' or later versions like '''tmux'''.<br />
<br />
You log in over ssh, then type:<br />
<br />
tmux<br />
<br />
This will put you in a little session-in-a-session. Even if you log out of ssh this tmux session will persist and you can get back to it by again typing ''tmux''.<br />
<br />
Once you've finished writing your commands (e.g. '''python my_python_program_that_takes_3_days.py'''), and started it running, you can get back to an ordinary shell session by typing: <br />
<br />
Ctrl-b d<br />
<br />
(that is, a '''Ctrl-b''' followed by a '''d''').<br />
<br />
Now if you want to get back to your previous session, just run ''tmux'' again. Actually, to be sure you connect to the right session, run:<br />
<br />
tmux list-sessions<br />
<br />
See the session number of the session to which you want to connect (e.g. '''0''') then to connect to it, run:<br />
<br />
tmux attach -t 0<br />
<br />
There is a lot more to ''tmux'' than that but that's the important part. <br />
<br />
For more, see:<br />
<br />
https://robots.thoughtbot.com/a-tmux-crash-course<br />
<br />
== Other gotchas ==<br />
<br />
=== Who else is using the computer? ===<br />
<br />
Try running<br />
<br />
who<br />
<br />
Or<br />
<br />
nvidia-smi<br />
<br />
Or<br />
<br />
top<br />
<br />
=== Temperature ===<br />
<br />
The GPUs are set to slowdown at 93C and shutdown at 96C. Idle temperature should be about 50C.<br />
To see the current temperature information in full run:<br />
nvidia-smi -q -d temperature<br />
<br />
It should never reach that temperature. If it does that's a big whoops.<br />
<br />
I have noticed that the first GPU (GPU 0) gets hotter quicker, presumably due to its physical location. With this information, it might be preferable to use GPU 1 more of the time.<br />
<br />
=== GPU and Memory Allocation ===<br />
<br />
Multiple kernels and users can also run on one GPU. This may mean you don't have enough memory at some point. It is possible to set it so that a GPU can only be accessed by one user at a time. This may be necessary in the future to ensure there is enough memory for those big jobs. <br />
<br />
Tensorflow actually seems to claim all the memory on all the GPUs so it might be a nice gentle thing for other users for you to make tensorflow be a bit nicer.<br />
<br />
==== Hiding unused GPUs from your program ====<br />
<br />
One suggestion is to use<br />
<br />
nvidia-smi<br />
<br />
to check which GPUs are available first then hide the one that you don't need from your program, using <br />
<br />
CUDA_VISIBLE_DEVICES=0 yourprogram<br />
<br />
if you want to use the 1st GPU or<br />
<br />
CUDA_VISIBLE_DEVICES=1 yourprogram <br />
<br />
if you want to use the 2nd GPU.<br />
<br />
==== Restricting GPU memory used by your tensorflow program ====<br />
<br />
Another approach when using tensorlow is to only use a certain amount of memory as described in the following answer: https://stackoverflow.com/a/34200194/1616231<br />
<br />
==== Automatic memory claiming ====<br />
<br />
Alternatively you may make tensorflow take memory as it is needed by taking the steps described in the following answer: https://stackoverflow.com/a/37454574/1616231 (though this will ultimately use more memory).<br />
<br />
Non-tensorflow apparoaches may have different characteristics, so the first option (hiding GPUs from your program) is the most general.<br />
<br />
= More Information =<br />
<br />
== Server Construction ==<br />
<br />
* 1 x [http://ark.intel.com/products/88195/Intel-Core-i7-6700K-Processor-8M-Cache-up-to-4_20-GHz INTEL i7-6700K]<br />
* 2 x [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ GTX 1080ti GPU]<br />
* 1 x [https://www.asus.com/us/Motherboards/ROG-MAXIMUS-IX-HERO/specifications/ ASUS ROG MAXIMUS IX HERO]<br />
* 2 x CORSAIR 32GB (2x16) D4 3000Mhz CMU32GX4M2C3000C15 (total 64GB)<br />
* 1 x CORSAIR CP-9020094-EU 1000W PSU<br />
* 1 x [https://en.sharkoon.com/product/1678/M25-W#desc Sharkoon M25-W - Mini tower ATX 5.25"]<br />
* 1 x 3TB HDD<br />
* 1 x 256GB SSD<br />
<br />
Built by [http://www.uzmanlarpc.com/ Uzmanlar PC]<br />
<br />
== Software ==<br />
<br />
=== OS ===<br />
<br />
[http://kubuntu.org/getkubuntu/ Kubuntu 16.04.3 LTS]<br />
<br />
=== Graphics Drivers ===<br />
<br />
[http://www.nvidia.com/Download/driverResults.aspx/120917/en-us Nvidia 384.59 drivers installed using runfile NVIDIA-Linux-x86_64-384.59.run]<br />
<br />
Installed using (to keep using the integrated graphics as main display graphics):<br />
<br />
sudo ./NVIDIA-Linux-x86_64-370.28.run --no-opengl-files --no-x-check --disable-nouveau<br />
<br />
=== CUDA Drivers ===<br />
<br />
Installed using <br />
<br />
cuda_8.0.61.2_linux.run</div>Imadhttp://deepdepth.djduff.net/wiki/index.php?title=NautilusServer&diff=166NautilusServer2018-01-31T13:30:51Z<p>Imad: Remove IP Address</p>
<hr />
<div>= Help = <br />
<br />
== Accessing ==<br />
<br />
'''Access from''': ITU or [http://bidb.itu.edu.tr/hizmetler/vpn ITU VPN].<br />
<br />
=== VPN Help ===<br />
<br />
* [http://bidb.itu.edu.tr/hizmetler/vpn BIDB has instructions for accessing the VPN.]<br />
* I have also provided here [http://files.djduff.net/scripts/ a script for accessing the ITU VPN from Ubuntu (ituvpn.sh)] (tested on Kubuntu 16.04).<br />
<br />
=== SSH help ===<br />
<br />
The SSH command to connect from a Unix environment:<br />
<br />
ssh -X -p 1542 hossein@<SERVER-IP-ADDRESS><br />
<br />
Switch meanings:<br />
* '''-p 1542''': connect on port 1542<br />
* '''-X:''' This allows you to run X applications. Omit it if you will be pure command line. You can also run X applications from Windows but you will need to install an X server on your Windows machine.<br />
<br />
'''Note''': to check that X forwarding is working, once you have connected, try running on the server the command:<br />
xeyes<br />
Or:<br />
dolphin<br />
<br />
You could for example run ''spyder'' like this. But there can be some latency across the network.<br />
<br />
== Setting up a deep learning environment ==<br />
<br />
<br />
=== Install anaconda ===<br />
<br />
export ANACONDA_PATH_PARENT=$HOME/software<br />
export ANACONDA_PATH=$ANACONDA_PATH_PARENT/anaconda3<br />
export ANACONDA_INSTALLER=Anaconda3-4.3.1-Linux-x86_64.sh<br />
<br />
mkdir -p ~/tmp<br />
cd ~/tmp<br />
mkdir -p $ANACONDA_PATH_PARENT<br />
wget https://repo.continuum.io/archive/$ANACONDA_INSTALLER<br />
bash $ANACONDA_INSTALLER -b -p $ANACONDA_PATH<br />
<br />
export PATH=$ANACONDA_PATH/bin:$PATH<br />
echo PATH: $PATH<br />
echo >> ~/.bashrc<br />
echo export PATH=$ANACONDA_PATH/bin:\$PATH >> ~/.bashrc<br />
<br />
=== Install tensorflow and keras ===<br />
<br />
These will be installed in a [https://conda.io/docs/intro.html conda] environment called '''deep''':<br />
<br />
export ENVNAME=deep<br />
conda create --name $ENVNAME<br />
source activate $ENVNAME<br />
conda install theano keras tensorflow tensorflow-gpu opencv pillow spyder matplotlib<br />
<br />
To check Keras is working:<br />
<br />
python -c "from keras.models import Sequential;Sequential()"<br />
<br />
To check the GPU is working with tensorflow, check this first (it should list two GPUs):<br />
<br />
nvidia-smi<br />
<br />
Then make sure the following script runs and finds one CPU and two GPUS: https://bitbucket.org/damienjadeduff/uhem_keras_tf/src/master/sariyer_python3/test_tf_gpu.py<br />
<br />
Run it like this:<br />
<br />
python test_tf_gpu.py<br />
<br />
'''Warning:''' for different versions of Tensorflow, Keras or Theano you may need to use '''[https://pip.pypa.io/en/latest/user_guide/#installing-packages pip]''' to install the version you need in an environment.<br />
<br />
== Easy file access (Linux) ==<br />
<br />
This can be useful for getting files on and off the server by accessing your remote home directory as if it was on your local computer (mounted on your file system).<br />
<br />
On YOUR Linux computer run:<br />
<br />
sudo apt-get install sshfs<br />
targ=~/remote/nautilus<br />
mkdir -p $targ<br />
fusermount -u $targ # only necessary to unmount if already tried<br />
sshfs -p 1542 -o workaround=rename YOUR_SERVER_USERNAME@160.75.27.83:/home/YOUR_SERVER_USERNAME $targ<br />
<br />
Note: if parts of your system hang because the connection to the ssh server gets stale (a common problem), just do:<br />
<br />
killall ssfs<br />
<br />
It should resolve most of your problems.<br />
<br />
== Using the SSD ==<br />
<br />
There is an SSD drive installed. This drive is automatically mounted at:<br />
<br />
/media/FASTDATA1<br />
<br />
The drive belongs to user '''root''' and group '''fastdata1'''.<br />
If you cannot access it you need to get an admin (Hossein) to add you to the group with the command:<br />
<br />
sudo usermod -aG fastdata1 YOURUSERNAME<br />
<br />
And to add you a folder in there with the right permissions:<br />
<br />
sudo mkdir /media/FASTDATA1/YOURUSERNAME<br />
sudo chown YOURUSERNAME:YOURUSERNAME /media/FASTDATA1/YOURUSERNAME<br />
sudo chmod 700 /media/FASTDATA1/YOURUSERNAME<br />
<br />
== Compiling your own CUDA programs ==<br />
<br />
To do this, add the following lines to your '''.bashrc''' file in your home folder:<br />
<br />
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH<br />
export PATH=/usr/local/cuda/bin:$PATH<br />
<br />
== Running programs after you log out ==<br />
<br />
Sometimes a process may take a while and if you happen to log out the lack of interactivity may cause that process to give up on you. <br />
<br />
To get around this, people use the tool '''screen''' or later versions like '''tmux'''.<br />
<br />
You log in over ssh, then type:<br />
<br />
tmux<br />
<br />
This will put you in a little session-in-a-session. Even if you log out of ssh this tmux session will persist and you can get back to it by again typing ''tmux''.<br />
<br />
Once you've finished writing your commands (e.g. '''python my_python_program_that_takes_3_days.py'''), and started it running, you can get back to an ordinary shell session by typing: <br />
<br />
Ctrl-b d<br />
<br />
(that is, a '''Ctrl-b''' followed by a '''d''').<br />
<br />
Now if you want to get back to your previous session, just run ''tmux'' again. Actually, to be sure you connect to the right session, run:<br />
<br />
tmux list-sessions<br />
<br />
See the session number of the session to which you want to connect (e.g. '''0''') then to connect to it, run:<br />
<br />
tmux attach -t 0<br />
<br />
There is a lot more to ''tmux'' than that but that's the important part. <br />
<br />
For more, see:<br />
<br />
https://robots.thoughtbot.com/a-tmux-crash-course<br />
<br />
== Other gotchas ==<br />
<br />
=== Who else is using the computer? ===<br />
<br />
Try running<br />
<br />
who<br />
<br />
Or<br />
<br />
nvidia-smi<br />
<br />
Or<br />
<br />
top<br />
<br />
=== Temperature ===<br />
<br />
The GPUs are set to slowdown at 93C and shutdown at 96C. Idle temperature should be about 50C.<br />
To see the current temperature information in full run:<br />
nvidia-smi -q -d temperature<br />
<br />
It should never reach that temperature. If it does that's a big whoops.<br />
<br />
I have noticed that the first GPU (GPU 0) gets hotter quicker, presumably due to its physical location. With this information, it might be preferable to use GPU 1 more of the time.<br />
<br />
=== GPU and Memory Allocation ===<br />
<br />
Multiple kernels and users can also run on one GPU. This may mean you don't have enough memory at some point. It is possible to set it so that a GPU can only be accessed by one user at a time. This may be necessary in the future to ensure there is enough memory for those big jobs. <br />
<br />
Tensorflow actually seems to claim all the memory on all the GPUs so it might be a nice gentle thing for other users for you to make tensorflow be a bit nicer.<br />
<br />
==== Hiding unused GPUs from your program ====<br />
<br />
One suggestion is to use<br />
<br />
nvidia-smi<br />
<br />
to check which GPUs are available first then hide the one that you don't need from your program, using <br />
<br />
CUDA_VISIBLE_DEVICES=0 yourprogram<br />
<br />
if you want to use the 1st GPU or<br />
<br />
CUDA_VISIBLE_DEVICES=1 yourprogram <br />
<br />
if you want to use the 2nd GPU.<br />
<br />
==== Restricting GPU memory used by your tensorflow program ====<br />
<br />
Another approach when using tensorlow is to only use a certain amount of memory as described in the following answer: https://stackoverflow.com/a/34200194/1616231<br />
<br />
==== Automatic memory claiming ====<br />
<br />
Alternatively you may make tensorflow take memory as it is needed by taking the steps described in the following answer: https://stackoverflow.com/a/37454574/1616231 (though this will ultimately use more memory).<br />
<br />
Non-tensorflow apparoaches may have different characteristics, so the first option (hiding GPUs from your program) is the most general.<br />
<br />
= More Information =<br />
<br />
== Server Construction ==<br />
<br />
* 1 x [http://ark.intel.com/products/88195/Intel-Core-i7-6700K-Processor-8M-Cache-up-to-4_20-GHz INTEL i7-6700K]<br />
* 2 x [https://www.nvidia.com/en-us/geforce/products/10series/geforce-gtx-1080-ti/ GTX 1080ti GPU]<br />
* 1 x [https://www.asus.com/us/Motherboards/ROG-MAXIMUS-IX-HERO/specifications/ ASUS ROG MAXIMUS IX HERO]<br />
* 2 x CORSAIR 32GB (2x16) D4 3000Mhz CMU32GX4M2C3000C15 (total 64GB)<br />
* 1 x CORSAIR CP-9020094-EU 1000W PSU<br />
* 1 x [https://en.sharkoon.com/product/1678/M25-W#desc Sharkoon M25-W - Mini tower ATX 5.25"]<br />
* 1 x 3TB HDD<br />
* 1 x 256GB SSD<br />
<br />
Built by [http://www.uzmanlarpc.com/ Uzmanlar PC]<br />
<br />
== Software ==<br />
<br />
=== OS ===<br />
<br />
[http://kubuntu.org/getkubuntu/ Kubuntu 16.04.3 LTS]<br />
<br />
=== Graphics Drivers ===<br />
<br />
[http://www.nvidia.com/Download/driverResults.aspx/120917/en-us Nvidia 384.59 drivers installed using runfile NVIDIA-Linux-x86_64-384.59.run]<br />
<br />
Installed using (to keep using the integrated graphics as main display graphics):<br />
<br />
sudo ./NVIDIA-Linux-x86_64-370.28.run --no-opengl-files --no-x-check --disable-nouveau<br />
<br />
=== CUDA Drivers ===<br />
<br />
Installed using <br />
<br />
cuda_8.0.61.2_linux.run</div>Imad