How to setup a development environment?

[whilo]

I have found out when I was trying to debug numpy that building all stuff JyNI related can be hairy. Could you provide a guide (and maybe a short screencast)?

[Stewori]

Sure. However, I can explain my setup, which might actually not be optimal, so I'm open for improvements!

First of all, I am using Eclipse as IDE. Historically this wasn't actually my first choice. I used to program in Netbeans once, until they dropped Python support somewhen after version 5 or so.
What bothers me most with Eclipse is it's inhomogenity in L&F when using it with different languages; and it's stubbornness whith appearance configuration. Also cdt refuse to perform code-navigation as soon as a c-file exceeds 7k lines or so, which frequently happens in CPython (e.g. typeobject.c), ctypes or NumPy codebase. Still, eclipse so far was the most sufficient solution for cross-language and cross-platform support.
AFAIK @peanut256 uses an XCode setup on OSX, maybe he will share some instructions on this front here too.

For JyNI I use up to 6 parallel projects, because mixing languages in one project is impossible (or at least impossibly painful).
These are:

• JyNI-C (JyNI's main C-part)
• JyNI-Java (JyNI's main Java-part)
• JyNI-loader (Another C project, piping through call-ins, see head-comment in JyNILoader.c for why it's needed)
• DemoExtension (A C-extension for testing minimalistic implemetations of features/issues)
• JyNI-Demo (Python code for demo/testing, using DemoExtension et al)
• JyNI-Lib (some Python-code for supported extensions, which is bundled by JyNI for convenience to simplify setup; currently it contains ctypes and sqlite3)

Each of these projects has a direct correspondence as a folder in JyNI's repo. For language select Java, C, or Python (take the obvious fit for each ;) ) I will give more details on project configuration next time.

Then. I gave up to use Eclipse's own build-process for any of the JyNI-projects. Working on JyNI I always hava a console window open to enter "make" from time to time. It is crucial to turn off automatic building in Eclipse, especially for JyNI-Java!

Some notes on using JyNI's makefile:

• currently you need to adjust its paths for your system:
  Make sure

JYTHON = ./jython.jar
PY_INCLUDE = /usr/include/python2.7
JAVA_HOME = /usr/lib/jvm/default-java

are adjusted for your system. JyNI builds fine with Java 7 or 8.
I usually leave the Jython-line as it is and place a symlink to Jython-standalone in my JyNI-folder and rename it to jython.jar.
Be sure to use Jython-standalone.jar of Jython 2.7.1 b3 or newer. JyOpenGL requires newest repository version, because it calls posixmodule.uname, which is not included in Jython 2.7.1b3.

• because make wasn't made for Java, the maketool fails to detect changes properly and always compiles the whole Java-part (and also JyNI-Lib to class-files). This can be annoying if you work on C-code. If you know there were no changes in Java-part, you can use "make libJyNI" which only rebuilds C-part.
• if you changed something in a header-file, run make clean before building
• use make -f makefile.clang to compile with clang.
• use make -f makefile.osx to compile on OSX with clang

In that case you should of course adjust JYTHON, PY_INCLUDE and JAVA_HOME in your corresponding makefile.

I close with some notes on debugging, will maybe specify in more detail later.

• using debuggers on JyNI is tricky, because you must step through different languages. Usually you use Python or Java-debugger to halt on some breakpoint right before the relevant code. Then you must hook-in C-debugger on the process-ID. It's long time ago that I used this lastly, now I mainly debug by logging. To use C-debugger nowaday on JyNI would require to adjust makefile to include debug-symbols. I guess I should add an option for this...
• being able to print traceback for current position is handy
  C: jPrintCStackTrace(); (see JyNI-Debug.h)
  Java: new Exception().printStackTrace()
  Python: import traceback; traceback.print_stack()
  Unfortunately this cannot cross language-boundaries
• output of printf does not line-up well with Java- or Jython-driven Python output. Use jputs (for C-strings) and jputsLong (for ints) on C-site if you need it in correct sequence with J/Py-output. E.g. jputs(__FUNCTION__) or jputsLong(__LINE__) are handy quick c/p drop-ins to check if a code section is reached etc. A jprintf is still a todo; actually a more systematic logging aproach is a long standing item on my todo-list at all.
• from time to time I encountered some deadlock, usually due to a gc-related issue. Such deadlocks can be hard to reproduce, sometimes it only happens once in a thousand times, maybe even less. So it is important to run jstack whenever a deadlock occurs. Often enough I was able to fix it from the jstack-output of a single occurrance. So make yourself familiar with jstack usage if not yet done. Would be a pity to miss this opportunity in case you might run into a still unrecognized deadlock-issue!
• it is handy to have a custom C-Python build around. In cumbersome cases it's helpful to ask "what would CPython do?", often this can't be sufficiently investigated from Python-site, so being able to insert C-level logging can be helpful.
• same for the C-extension you want to get working or to improve
• Building a customized Jython should also not scare you away. Like with JyNI I gave up to get Ecplipse do this, but instead run ant from terminal.

So much for now. I will supplement this info as it occurs to me or on demand.

[Stewori]

One more thing. You might have noticed that in JyNI-code I use tabs all the way. tabs vs spaces is a historic debate https://www.youtube.com/watch?v=SsoOG6ZeyUI
In contradiction to Jython- and CPython styleguide I am actually on the tabs-front.
Note that I only use tabs for indentation, not for alignment of comments next to code (frequently used in type-definitions in CPython codebase). Let me make my point on tabs:

• people usually complain that tabs look different on each computer - actually tabs make indentation-depth easily adjustable to each person's visual preference; I hardly know any text-editor where displayed tab-width is not configurable
• convenient use of spaces for indentation relies on an editor's capability to recognize space-indentation and to auto-expand tabs. This usually requires an IDE or is at least a much stricter requirement than customizability of tab-displaystyle
• tabs don't work for alignment, which I agree on, so only use it for indentation
• I admit this still messes up comment-alignment beneath code over several indentation levels. However a use-case for such a comment never crossed my way so far
• IMHO tabs are made for this
• this currently does not apply to JyNI-Lib, because it was mostly copied 1-to-1 from CPython's Lib.

So please stick to this as JyNI-styleguide, hope it doesn't bother you too much! For people who are uncomfortable regarding tabs vs spaces, the following post might be helpful:
https://stackoverflow.com/questions/2316677/can-git-automatically-switch-between-spaces-and-tabs

[Stewori]

As of 1937112 makefile obtains JAVA_HOME from java-command if necessary and also respects if JAVA_HOME was previously given as environment variable. So this step can be skipped now in build-setup. However JYTHON must still be hand-configured (or provided e.g. as symlink).

[CalumFreeman]

Hi,

I have managed to get a C debugger(gdb) to attach to a running jython process. Here is a rough guide as to how I did it (suggestions for improvements in both content and delivery are very welcome).

Compiling/make-ing JyNI:

As was mentioned earlier, to include debug symbols in the compiled files you need to adjust the makefile. Fortunately this is a fairly simple fix: simply add -g to the CFLAGS variable (I put it between -c and -O0 but I don't think that matters). Then you must run "make clean" before running "make" again or else it may not actually recompile the files and you won't get the debug symbols.

Before starting gdb:

To attach gdb to a java application that uses the Java native interface to run C code, I have been told you should include the following commands in the gdb init file (~/gdbinit). If you are only using the command line you could just run these every time you start gdb, but that gets annoying very quickly. I think these commands basically just stop the debugger from throwing lots of extra errors which are thrown by the JVM in normal operation, but I'm not entirely sure.

handle SIGXCPU nostop noprint pass
handle SIGPWR nostop noprint pass
handle SIGSEGV nostop noprint pass

Command line only:

If you just want to debug from the command line then gdb is now setup, all you need to do is get the process id for the JyNI run and type the command gdb attach <pid>. This will open the gdb environment, you can set breakpoints in one of two ways:

break /Path/To/File.c:lineNumber
break Function_Name

Note that gdb halts the process it attaches to so you need to run the command c to continue so that the code you want to debug can actually run. Then once the breakpoint has triggered you can use gdb to debug using it's commands, full documentation is available(https://sourceware.org/gdb/download/onlinedocs/gdb/index.html#Top) but here's a shorter list of commands:

s = step (into)
n = next (step over)
f = finish function
c = continue
print var = print the variable called var
return/enter repeats the last command

Eclipse:

To setup gdb in eclipse is slightly more involved. First you must go to the menu Run→Debug configurations. This will open a window, on the left bar select C/C++ Attach to application, then click the icon on the top left of the bar to create a New launch configuration. On the first tab(Main) select the project containing the .c files(in our case JyNI-C). The second tab(debugger) shouldn't need changing for most people, where I work the default version of gdb doesn't work very well so the path needs to be changed to point to the correct version (version 8.1.0 seems to work). Then on the third tab (source) you need to add all the folders containing all the .so files you want to be able to see the source code for while debugging. Then remember to click apply!

At this point you can set some other process running, ideally in debug mode with a breakpoint before the C code you're interested in so that you have time to attach gdb before the C code executes. Then go to the debug configuration screen, select C/C++ Attach to application again and click the debug button. You will then be asked to pick a process to attach to, you can type jps in a terminal to get a list of java processes and their process id's, if you only have one jython process running then that is the one to pick. Once attached, gdb will halt the JVM so you need to tell the C/C++ debugger to continue before you can make anything happen in your java process(press F8). You can then select the java process and start it again. You may have to take care in remembering which debugger you are in before stepping but other than that you can set breakpoints and use this like any other debugger in eclipse.

[CalumFreeman]

I've started to build a wiki page to answer this question, although it isn't done yet.

There is some extra information on the wiki so I'll repost it here until the wiki page is done:

It's helpful to enable non-stop mode in the C debugger as this avoids needing to press F8 as soon as the proccess attaches, it also allows java's threads to behave properly.

Also, there is a bug that will show up in debug mode only, this appears to be a known jython issue which is difficult to fix 2517. It's something to do with timing which get's thrown off in debug mode, I don't think it's worth pursuing a fix but we should warn developers about it.

[Stewori]

That's perfect. Moving to the wiki is the right choice at this stage.
@CalumFreeman In section https://github.com/Stewori/JyNI/wiki/How-to-set-up-a-development-environment#build-the-projects I can readily fill in the build commands for numpy and CPython. However, I did not yet figure out debug mode. Given we're moving this to the wiki, I think this issue can be closed soon. Before closing I'd like to find time to hint to the wiki from the readme...

[Stewori]

We should add a build target for debug mode. Don't spend too much effort on documenting how to change the makefile to obtain a debug version manually. I sourced this out into #30.

[Stewori]

Also worth noting:
Now that we are moving this to the wiki I noticed that it is not obvious how to get notified about wiki changes in github. A helpful stack overflow entry suggests to subscribe the atom feed for this.
I figured out the atom feed for JyNI wiki is https://github.com/stewori/jyni/wiki.atom

[Stewori]

Building NumPy with debug symbols seems not to be so obvious. This page gives some hints but as far as I can see assumes that one simply installs debug extensions of numpy. This page gave some more interesting hints. The following might work:
CFLAGS="-O0 -g" python setup.py build_ext --inplace
At some places I also encountered -g or --debug flags applied to setup.py so that's eventually also worth a try, e.g. python setup.py build_ext --inplace --debug or python setup.py build_ext --inplace -g
I did not actually try these yet, just had some time so search a bit...