Building Tensorflow GPU on Fedora Linux

<update on Sept 13th 2019>
I have written another post on how to install (rather than build) Tensorflow GPU for Fedora that uses a different and much simpler method.  See Installing Tensorflow GPU on Fedora Linux.
</update>

First off, let's say that there are easy ways of configuring Tensorflow for GPU usage such as using one of the docker images.  However, I'm a bit old school for some things and having always done so I've recently got Tensorflow going on my machine using my GPU.  Tensorflow CPU support is quite easy to do and generally works quite nicely using the pip install method.  GPU support, I've always found, is quite a bit more difficult as there are a whole bunch of things that need to be at just the right level for everything to work i.e. it's quite brittle!

What follows are my notes (it's in the name of the blog) for how to build Tensorflow from scratch to enable GPU support and I do this on Fedora Linux.  If you want to know why it's worth bothering going to this effort, I've tested the Keras MNIST CNN example as a bench mark.  It takes:

• 11 minutes 7 seconds on my CPU
• 2 minutes 55 seconds on my GPU

That's just over 3.8 as fast on my GPU as per my CPU so for large jobs this will be huge!

Some info on my machine and config:

• Lenovo P50 Laptop
• Intel Core i7-6820HQ CPU @ 2.70GHz (4 core with hyper threading)
• 32GB RAM
• Nvidia Quadro M1000M (CUDA compute capability 5.0)
• Fedora 28 running kernel 4.18.18-200.fc28.x86_64

Install Required Nvidia RPMs
You need to get everything Nvidia and CUDA installed on your machine first.   I quite like the Negativo17 repository for Nvidia on Fedora Linux and so I use this but you could also go with RPM Fusion or even download everything directly from Nvidia.  For me, right now, I have this little lot installed:
cuda-9.2.148.1-2.fc28.x86_64
cuda-cli-tools-9.2.148.1-2.fc28.x86_64
cuda-cublas-9.2.148.1-2.fc28.x86_64
cuda-cublas-devel-9.2.148.1-2.fc28.x86_64
cuda-cudart-9.2.148.1-2.fc28.x86_64
cuda-cudart-devel-9.2.148.1-2.fc28.x86_64
cuda-cudnn-7.2.1.38-1.fc28.x86_64
cuda-cudnn-devel-7.2.1.38-1.fc28.x86_64
cuda-cufft-9.2.148.1-2.fc28.x86_64
cuda-cufft-devel-9.2.148.1-2.fc28.x86_64
cuda-cupti-9.2.148.1-2.fc28.x86_64
cuda-cupti-devel-9.2.148.1-2.fc28.x86_64
cuda-curand-9.2.148.1-2.fc28.x86_64
cuda-curand-devel-9.2.148.1-2.fc28.x86_64
cuda-cusolver-9.2.148.1-2.fc28.x86_64
cuda-cusolver-devel-9.2.148.1-2.fc28.x86_64
cuda-cusparse-9.2.148.1-2.fc28.x86_64
cuda-cusparse-devel-9.2.148.1-2.fc28.x86_64
cuda-devel-9.2.148.1-2.fc28.x86_64
cuda-docs-9.2.148.1-2.fc28.noarch
cuda-gcc-7.3.0-1.fc28.x86_64
cuda-gcc-c++-7.3.0-1.fc28.x86_64
cuda-gcc-gfortran-7.3.0-1.fc28.x86_64
cuda-libs-9.2.148.1-2.fc28.x86_64
cuda-npp-9.2.148.1-2.fc28.x86_64
cuda-npp-devel-9.2.148.1-2.fc28.x86_64
cuda-nvgraph-9.2.148.1-2.fc28.x86_64
cuda-nvgraph-devel-9.2.148.1-2.fc28.x86_64
cuda-nvml-devel-9.2.148.1-2.fc28.x86_64
cuda-nvrtc-9.2.148.1-2.fc28.x86_64
cuda-nvrtc-devel-9.2.148.1-2.fc28.x86_64
cuda-nvtx-9.2.148.1-2.fc28.x86_64
cuda-nvtx-devel-9.2.148.1-2.fc28.x86_64
nvidia-driver-cuda-libs-410.73-4.fc28.x86_64

You might wonder about some of the above, particularly why you might need a back level version of GCC.  When Fedora 28 has a quite capable GCC version 8 why on earth would you want version 7?  The answer lies in my comment about things being difficult or brittle, it's quite simply that CUDA doesn't yet support GCC 8 so you do need a back level compiler for this

Install NVidia NCCL
This library isn't available through an RPM installation or the Negativo17 repository and so you must:

• Go to the Nvidia NCCL home page 
• Click the link to download NCCL (requires an Nvidia developer login account)
• Agree to the Terms and Conditions
• Download the NCCL zipped tar file that matches your CUDA version (9.2 for this blog post)

At the time of writing the file required is nccl_2.3.7-1+cuda9.2_x86_64.txz

I simply untar this file into /usr/local and create a symbolic link as follows:

• cd /usr/local
• sudo tar -xf /path/to/file/nccl_2.3.7-1+cuda9.2_x86_64.txz
• sudo ln -s nccl_2.3.7-1+cuda9.2_x86_64.txz nccl

Install the Bazel Build Tool
You're going to need a build tool called Bazel which isn't directly available in the Fedora repositories (that I know of at least) but fortunately there's a version in a copr repository you can use as documented run the following commands:

•  dnf copr enable vbatts/bazel
•  dnf install bazel

Get a Copy of Tensorflow Source
For this it's just as easy to use git as it is anything else.  You can directly clone the 1.12 release of Tensorflow into a new directory by running:

• git clone --single-branch -b r1.12 https://github.com/tensorflow/tensorflow tensorflow-r1.12
• cd tensorflow-r1.12

Simply replace r1.12 in the above commands if you want to use a different Tensorflow release.

Run the Tensorflow Configure Script
This step is actually quite simple but you'll need the answers to some questions to hand, simply run:

• ./configure

I accept all the default options with the exception of:

• "location of python" set to /usr/bin/python3 since Fedora still uses Python 2.7 as the default version at /usr/bin/python
• "build TensorFlow with CUDA support" set to Yes
• "CUDA SDK version" set to 9.2 (this value should match the cuda version you have installed and at the time of writing 9.2 is the current version from the Negativo17 repository)
• "location where CUDA 9.2 toolkit is installed" set to /usr
• "cuDNN version" set to 7.2 (similar to the cuda version above, this value should match the cuda-cudnn package version and 7.2 is the current version from the Negativo17 repository)
• "NCCL version" set to 2.3
• "location where NCCL 2 library is installed" set to /usr/local/nccl
• "Cuda compute capabilities you want to build with" set to 5.0 (but this value should match the CUDA compute capability of the GPU in the machine you're building for)
• "which gcc" set to /usr/bin/cuda-gcc (to use the back level GCC version 7)

Fix Bazel Config

The above config command writes a file but the location isn't compatible with the latest version of Bazel.  Presumably this issue will be fixed at some point in the future, it's not an issue with Bazel 0.18 and below as far as I'm aware, but has just become an issue on 0.19.  Simply copy the config to the correct place:

• cat tools/bazel.rc >> .tf_configure.bazelrc

Build Tensorflow with GPU Support

This took around an hour to complete on my machine:

• bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package
• bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

The first step is the long one for the build, the second simply builds the python wheel file.

Install Tensorflow with GPU Support

You've got your wheel file so simply install and enjoy:

• pip3 install tensorflow-1.12.0-cp36-cp36m-linux_x86_64.whl 

Run Some Code

The first time I attempted to run some code to test I got an error:

• failed call to cuInit: CUDA_ERROR_UNKNOWN

This can be solved by making sure you have the nvidia-modprobe package installed.  Alternatively, you can run the little script below the following explanation.

This seems to be some sort of permissions issue and running the following simple script to output the GPUs available on my machine but as root seems to have fixed the above issue i.e. put the following into a script, run that script as root, then any time you want to run code as an unprivileged user the above issue is fixed and the code will work:

from keras import backend as K

K.tensorflow_backend._get_available_gpus()

If the above works then you can try out the Keras MNIST CNN example code.

VueJS Example for IBM App ID

I was recently working on a project in VueJS that needed an authorisation layer added to it.  It turns out there aren't any existing examples of how to do this anywhere, unusually not even on Stack Overflow.  So I set about writing one and thought I would share it.  My work was based upon some other useful examples and information, particularly a blog post from the IBM Cloud blog.

Before I go any further, the code samples are available and documented on GitHub as follows:

1. IBM App ID API Server
2. App ID VueJS Client

The code is deliberately split into two such that:

1. the API Server is used to demonstrate how to secure an API on the server side.  This is done with the WebAppStrategy of App ID which is simply an implementation of a strategy package for passportjs.  The code here isn't anything particularly new over existing examples you can find on the web but it's necessary in order to fully demonstrate the capabilities of the client code.
2. the VueJS Client is used to demonstrate two things:
1. how to secure a VueJS route for which I can currently find no example implementations on the web
2. how to call an API that has been secured by App ID by passing credentials through from the client application to the API server

The API Server should be relatively trivial to get up and running as it's a standard NodeJS API implementation using Express.  If you refer to the WebAppStrategy and the blog post I mention above then you'll see the sample code I've come up with is broadly the same i.e. an amalgamation of the two.

The VueJS Client code can be simple to get up and running as well but it's probably more important to understand how it was created such that you can apply the same principles in your own application(s).  For this then, the explanation is a little longer...

Start by running the VueJS command line client (cli) to create a bare project and for the sample to make sense you will need to add VueX and Router components using the tool:

vue create vue-client

Then understand the 3 modifications you need to make in order to have a working set of authenticated routes.

1. A store for state. 
It doesn't really matter how you achieve this in VueJS, you can use any form of local state storage.  The example code I have come up with uses VueX and a modification to the store.js code you get from the client above.  The idea of this is such that the client application can cache whether the user has already authenticated themselves.  If they have not then the client must request authentication via the server.  If they have, then all the credentials required for making an authenticated call to a server-side API are already available in the browser.  Essentially, this is a speed-up mechanism that stops the client from requesting client credentials on each API call since the session store for the authentication actually lives on the server side when using App ID.

2. A new VueJS Component
This is the component whose route is to be protected via authentication.  In the case of the example code below the standard vue cli "About" component has been used and modified slightly to include an authenticated call to the server API.  The thing to note here is that the credentials from the client side must be sent over to the server with each API call.  Using the fetch API as per the below to implement your GET request means you have to add the credentials: 'include' parameter.

<template>
  <div class="about">
    <h1>This is a protected page</h1>
    <h2>hello: {{ hello }}</h2>
  </div>
</template>

<script>
export default {
  data: function () {
    return {
      hello: undefined
    }
  },
  computed: {
    user () {
      return this.$store.state.user
    }
  },
  methods: {
    getProtectedAPI () {
      fetch('http://localhost:3000/protected/get-some-info',{
            credentials: 'include',
          }).then(res => res.text())
          .then(body => {
            console.dir(body)
            this.hello = JSON.parse(body).hello
          })
    },
  },
  created() {
    this.getProtectedAPI()
  }
} 

</script>

3. A VueJS Navigation Guard
You need to write a function that will be added as a VueJS middleware upon each route change.  The middleware is inserted automatically by the VueJS route code when using the beforeEnter call on a route.  This is known in VueJS as a Navigation Guard.

function requireAuth(to, from, next) {
  // Testing authentication state of the user
  if (!store.state.user.logged) {
    // Not sure if user is logged in yet, testing their login
    const isLoggedUrl = "http://localhost:3000/auth/logged"
    fetch(isLoggedUrl, {credentials: 'include'}).then(res => res.json()).then(isLogged => {
      if (isLogged.logged) {
        // User is already logged in, storing
        store.commit("setUser", isLogged)
        next()
      } else {
        // User is not logged in, redirecting to App ID
        window.location.href=`http://localhost:3000/auth/login?redirect=${to.fullPath}`
      }
    }).catch(e => {
      // TODO: do something sensible here so the user sees their login has failed
      console.log("Testing user login failed - D'oh!")
    })
  } else {
    // User already logged in
    next()
  }
}

The requireAuth function does the following in plain English:

1. Using the VueJS client side cache, test if the user is already logged in
2. If they are not. then ask the server if the user is already logged in
1. If they are not, then redirect them to the server login page
2. If they are, then cache the information and load the next piece of middleware
3. If they are, then simply load the next piece of middleware

Each route you want to protect with the above function must have a beforeEnter: requireAuth parameter specified on the route.  When this is done, VueJS will call the requireAuth function before the component specified by the route is loaded.

{
  path: '/protected',
  name: 'protected',
  beforeEnter: requireAuth,
  component: Protected
}

Note: there are methods by which you don't have do call window.location.href to redirect the user to the login page (which does seem like a bit of a nasty hack.  However, these methods require the modification of the webpack configuration and so were kept out of scope of this example for the purposes of being simple.