Installation/build issue - Any doc for installation steps?

[arinik9]

First, I would like to thank you for this github repo! Thanks to python API its use should be easier.
This is why I would like to install the program through anaconda (as you recommend). If possible, I would like to ask you to provide some installation steps for:

• installation based on python & c++ (through anaconda) ==> are there any requirements? python 3.7? anaconda 3? or some others?
• installation based on only c++ (without support of python api, through cmake) => in case I could not make it work

In the following, I explain what I tried since yesterday:
According to the readme file, the program is supposed to run with one simple line command: GUROBI_ROOT_DIR=/path/to/gurobi conda install -c cpape nifty-with-gurobi. However, it did not work out, since it does not found the package nifty-with-gurobi in the current channel of anaconda. My second attempt was to build nifty by using this command conda build conda-recipe. However, I had many issues during build process. The last one was this one:

In file included from /home/nejat/anaconda3/conda-bld/nifty_1582733307969/work/include/nifty/graph/rag/detail_rag/compute_grid_rag.hxx:22:0,
from /home/nejat/anaconda3/conda-bld/nifty_1582733307969/work/include/nifty/graph/rag/grid_rag.hxx:19,
from /home/nejat/anaconda3/conda-bld/nifty_1582733307969/work/include/nifty/graph/rag/grid_rag_features.hxx:3,
from /home/nejat/anaconda3/conda-bld/nifty_1582733307969/work/src/python/lib/graph/rag/graph_accumulator.cxx:6:
/home/nejat/anaconda3/conda-bld/nifty_1582733307969/work/include/nifty/z5/z5.hxx:4:10: fatal error: z5/dataset_factory.hxx: No such file or directory
#include "z5/dataset_factory.hxx"
^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
compilation terminated.
src/python/lib/graph/rag/CMakeFiles/_rag.dir/build.make:110: recipe for target 'src/python/lib/graph/rag/CMakeFiles/_rag.dir/grid_rag_stacked.cxx.o' failed
make[2]: *** [src/python/lib/graph/rag/CMakeFiles/_rag.dir/grid_rag_stacked.cxx.o] Error 1
src/python/lib/graph/rag/CMakeFiles/_rag.dir/build.make:134: recipe for target 'src/python/lib/graph/rag/CMakeFiles/_rag.dir/graph_accumulator.cxx.o' failed
make[2]: *** [src/python/lib/graph/rag/CMakeFiles/_rag.dir/graph_accumulator.cxx.o] Error 1
CMakeFiles/Makefile2:1372: recipe for target 'src/python/lib/graph/rag/CMakeFiles/_rag.dir/all' failed
make[1]: *** [src/python/lib/graph/rag/CMakeFiles/_rag.dir/all] Error 2
[ 10%] Linking CXX shared module _nifty
Copying target opt to temporary module directory
[ 10%] Built target _opt
Copying target nifty to temporary module directory
[ 10%] Built target _nifty
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
File "/home/nejat/anaconda3/bin/conda-build", line 11, in
sys.exit(main())
File "/home/nejat/anaconda3/lib/python3.7/site-packages/conda_build/cli/main_build.py", line 438, in main
execute(sys.argv[1:])

Thanks in advance!
Nejat

[DerThorsten]

@constantinpape can maybe help, also his fork (https://github.com/constantinpape/nifty) is way more up to date then mine, but on a first glence it looks like z5 is missing which can be found here
https://github.com/constantinpape/z5

[arinik9]

Apparently, it can not find where numpy and z5 are installed (or, at least their header files). However, the header files of numpy are installed in: /home/nejat/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/include/numpy, and those of z5 are installed in /home/nejat/anaconda3/envs/py37/include/z5.

How and where to define manually those 2 paths in order that cmake recognize them?

examples of the related errors:

src/python/lib/graph/rag/CMakeFiles/_rag.dir/build.make:62: recipe for target 'src/python/lib/graph/rag/CMakeFiles/_rag.dir/rag.cxx.o' failed
make[2]: *** [src/python/lib/graph/rag/CMakeFiles/_rag.dir/rag.cxx.o] Error 1
In file included from /home/nejat/anaconda3/conda-bld/nifty_1582736589241/_build_env/include/xtensor-python/pytensor.hpp:21:0,
from /home/nejat/anaconda3/conda-bld/nifty_1582736589241/work/src/python/lib/graph/rag/project_to_pixels.cxx:6:
/home/nejat/anaconda3/conda-bld/nifty_1582736589241/_build_env/include/xtensor-python/pycontainer.hpp:28:10: fatal error: numpy/arrayobject.h: No such file or directory
#include "numpy/arrayobject.h"
^~~~~~~~~~~~~~~~~~~~~
compilation terminated.

---

src/python/lib/graph/rag/CMakeFiles/_rag.dir/build.make:134: recipe for target 'src/python/lib/graph/rag/CMakeFiles/_rag.dir/graph_accumulator.cxx.o' failed
make[2]: *** [src/python/lib/graph/rag/CMakeFiles/_rag.dir/graph_accumulator.cxx.o] Error 1
In file included from /home/nejat/anaconda3/conda-bld/nifty_1582736589241/work/include/nifty/graph/rag/detail_rag/compute_grid_rag.hxx:22:0,
from /home/nejat/anaconda3/conda-bld/nifty_1582736589241/work/include/nifty/graph/rag/grid_rag.hxx:19,
from /home/nejat/anaconda3/conda-bld/nifty_1582736589241/work/src/python/lib/graph/rag/accumulate.cxx:5:
/home/nejat/anaconda3/conda-bld/nifty_1582736589241/work/include/nifty/z5/z5.hxx:4:10: fatal error: z5/dataset_factory.hxx: No such file or directory
#include "z5/dataset_factory.hxx"
^~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
src/python/lib/graph/rag/CMakeFiles/_rag.dir/build.make:182: recipe for target 'src/python/lib/graph/rag/CMakeFiles/_rag.dir/accumulate.cxx.o' failed
make[2]: *** [src/python/lib/graph/rag/CMakeFiles/_rag.dir/accumulate.cxx.o] Error 1
CMakeFiles/Makefile2:1372: recipe for target 'src/python/lib/graph/rag/CMakeFiles/_rag.dir/all' failed
make[1]: *** [src/python/lib/graph/rag/CMakeFiles/_rag.dir/all] Error 2
Copying target nifty to temporary module directory
[ 16%] Built target _nifty
Makefile:140: recipe for target 'all' failed

[constantinpape]

Apparently, it can not find where numpy and z5 are installed (or, at least their header files). However, the header files of numpy are installed in: /home/nejat/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/include/numpy, and those of z5 are installed in /home/nejat/anaconda3/envs/py37/include/z5.

Could you paste the full cmake command you are using please? Also, which OS are you on?

If you want to install from a conda environment, I would recommend to run it like this:

cmake -DCMAKE_PREFIX_PATH=/path/to/conda/env -DNUMPY_INCLUDE_DIR=/path/to/conda/env/lib/python3.7/site-pacakges/numpy/core/include ...

It should pick z5 automatically (if it is installed to this environment) and find numpy by specifying the include dir.

Note that I recently fixed a few minor CMake issues on my fork of nifty, so maybe it would help if you check it out. (I will try to merge this soon, but I am currently updating a few features and want to finish this first.)

[arinik9]

actually, I am trying to install from a conda environment, mine was a 'conda build' command (I saw it in the README file located at the folder conda-recipe):
conda build -c conda-forge --python=3.7 conda-recipe
Is not it correct?

What is your full cmake command please ? Thanks!

By the way, I am on linux

[constantinpape]

conda build -c conda-forge --python=3.7 conda-recipe

This is the command to build the conda package, which just wraps cmake; it's easier to use cmake directly.

I just checked the README, and the installation instructions were outdated.
Could you try to install with this command:

conda install -c conda-forge -c cpape nifty

Let me know if it does not work, then I will update the instructions on how to build from source and we can try with CMake.

[arinik9]

thanks for your reply! It did work out with your command. I could be able to import nifty and then do some graph operations. Nevertheless, apparently in this conda package nifty.Configuration.WITH_QPBO is False. Do you have another conda package with WITH_QPBO = True ?

Essentially, since I passed some time with compilation issues, I am interested in compiling from source with CMake. Could you please share with me your Cmake commands ? What I tried is the following:

mkdir bld
cd bld
cmake -DNUMPY_INCLUDE_DIR=/path/to/anaconda3/envs/py37/lib/python3.7/site-pacakges/numpy/core/include -DCMAKE_INSTALL_PREFIX=../nifty-headers -DCMAKE_PREFIX_PATH=/path/to/anaconda3/envs/py37 ..
make install

but it gave me an error related to pybind11, e.g.

In file included from /home/nejat/eclipse/tmp/nifty/src/python/lib/graph/undirected_list_graph.cxx:9:0:
/home/nejat/eclipse/tmp/nifty/src/python/lib/graph/export_undirected_graph_class_api.hxx:50:12: error: ‘PYBIND11_DESCR’ does not name a type; did you mean ‘PYBIND11_EVAL’?
static PYBIND11_DESCR name() {
^~~~~~~~~~~~~~
PYBIND11_EVAL
In file included from /home/nejat/anaconda3/envs/py37/include/pybind11/attr.h:13:0,
from /home/nejat/anaconda3/envs/py37/include/pybind11/pybind11.h:44,
from /home/nejat/eclipse/tmp/nifty/src/python/lib/graph/undirected_list_graph.cxx:1:

Thanks in advance!

[constantinpape]

Nevertheless, apparently in this conda package nifty.Configuration.WITH_QPBO is False. Do you have another conda package with WITH_QPBO = True ?

Unfortunately not.

but it gave me an error related to pybind11, e.g.

Can you check which version of pybind11 you are using? I think I have seen this before and fixed it on my branch, can you try building from there?

[arinik9]

Regarding pybind, its version is 2.4.3

Actually, I was talking about your code. The compilation issue about pybind11 was based on your branch master of https://github.com/constantinpape/nifty

When you have some time, if you can test the installation steps (on a new environment), I would be grateful! Thanks!

[constantinpape]

Regarding pybind, its version is 2.4.3

Actually, I was talking about your code. The compilation issue about pybind11 was based on your branch master of https://github.com/constantinpape/nifty

Ok, I will look into it later.

[constantinpape]

Indeed, the build is broken with pybind11 >= 2.4; see #133.
It works with versions < 2.4.

I have also added a section on how to build from source to my branch:
https://github.com/constantinpape/nifty#from-source

Hope this helps and let me know if you run into more issues.

[arinik9]

Thank you very much for you help! I was able to build it with the option qpbo, and import it as package into python.

Nevertheless, I run into some issues with other options, that is, gurobi and lpmp.

• gurobi
  • I installed gurobi on my computer
  • I assign manually the variables related to GUROBI, i.e. GUROBI_ROOT_DIR, GUROBI_LIBRARY and GUROBI_INCLUDE_DIR to their paths (also in my $PATH variable)
• lpmp
  • I manually downloaded the required external header files, e.g.meta, tclap
  • I assign manually indicated where those header files are located

In the end, I could be able to compile two versions without problem.
But, the problem comes when I import the nifty package. I get the following error in python by typing import nifty:

Traceback (most recent call last):
File "", line 1, in
File "/opt/anaconda3/envs/GASP/lib/python3.7/site-packages/nifty/init.py", line 4, in
from ._nifty import *
ImportError: /opt/anaconda3/envs/GASP/lib/python3.7/site-packages/nifty/_nifty.cpython-37m- x86_64-linux-gnu.so: undefined symbol: _ZNK12GRBException10getMessageB5cxx11Ev

I suppose there was something wrong during the compilation. Have you had the same issue before? I mean, would you have an idea what can be the possible reason(s) which allows to compile the code, but we can not use it in execution? Thanks in advance!

[constantinpape]

Thank you very much for you help! I was able to build it with the option qpbo, and import it as package into python.

Glad it works now.

Nevertheless, I run into some issues with other options, that is, gurobi and lpmp.

lpmp support has not been tested in a while and should probably be deprecated.
The error you get for gurobi is a linker error. c++filt resolves the symbol to GRBException::getMessage[abi:cxx11]() const. There might be several causes for this; I haven't used the version build with Gurobi, so I don't really know what is going on.

Do you need any Gurobi functionality?

[arinik9]

Actually, for now I do not need any Gurobi functionality. I just thought that allowing Gurobi would mean to get the exact or LP relaxation solutions, so that would be nice.

[constantinpape]

Actually, for now I do not need any Gurobi functionality. I just thought that allowing Gurobi would mean to get the exact or LP relaxation solutions, so that would be nice.

Yes, that's true.
Linker errors can be kinda tricky though, because this might be system dependent.

Anyway, thanks for reporting the issue!