Release 0.55.0 "Stork"
This new release focuses on improving documentation, build system, and testing. It also introduces Travis-based CI/CD pipeline to publish artifacts to "nightly repo". In addition, it also adds many new features/improvements:
- preliminary virtio-fs support
- QEMU microvm machine support
- numerous improvements to dynamic linker including local-exec TLS
- kernel smaller by 300 K
- new HTTP monitoring API
- less memory utilization with ROFS
Highlights
- Revised existing documentation and added a new one
- Main README
- Dynamic Linker
- Memory Allocation
- Automated Testing Framework
- Building from host guide
- Improved build process
- Setup Travis-based CI/CD pipeline to automatically build and publish kernel and artifacts
- Setup process to publish OSv build docker images to docker hub
- Cleaned up build process by eliminating the "build from externals" mode
- Migrated python scripts to version 3
- Removed dependency on JDK to build the kernel
- Merged IPV6 code from Spirent into a new branch
- Removed the dependency on the boost program options library from kernel and utility apps which made kernel smaller by 300K
- Created Automated Testing Framework designed to run > 30 real apps, some of them under stress using
aborwrktools. - Improved dynamic linker
- Added local-exec TLS support
- Expose musl version of libc
- Handle RTLD_NEXT
- Added some bionic unit tests
- Resurrected aarch64 support
- Simple hello world app works now on QEMU and Raspberry PI 4 with KVM enabled
- Basic support in run.py
- Added support of QEMU microvm machine
- Added clock synchronization mechanism with host on KVM
- File system improvements
- ZFS improvements
- Can mount and modify OSv-build drive on a host and use it back on OSv
- NFS
- Changed filesystem driver to be a library
- ROFS
- Improved memory utilization by integrating with pagecache
- Virtio-FS
- Added preliminary support
- ZFS improvements
- Added new HTTP monitoring API module
New Apps
Closed Issues
- #352 - Thread-local storage doesn't work in PIE
- #743 - Stop using libraries and header files from external/
- #784 - realloc crashes with "page fault outside application"
- #820 - Add ability to build read-only version of httpserver-api module for monitoring purposes
- #854 - Allow non-contiguous malloc() when memory is fragmented
- #880 - Tomcat doesn't run on openjdk8-fedora
- #916 - scripts/loader.py is not python3 compatible
- #918 - Importing ZFS volume fails on OSv
- #975 - mysql cannot working with fs=ramfs
- #976 - Fix annoying warnings while building tests
- #980 - cpiod.so is much too big
- #1014 - Make kernel not to use boost program_options library
- #1015 - OSv guest's wall clock does not follow host's wall clock
- #1040 - Boost 1.69.0 on Fedora 30 contains hidden symbols
- #1044 - Test build with Travis CI
- #1045 - Delay processing PT_NOTE segments until all PT_LOAD segments are mmapped
- #1053 - httpserver should not use boost::program_options
- #1056 - Upgrade utility scripts to Python 3
- #1056 - Virtio-net: use large buffers when VIRTIO_NET_F_MRG_RXBUF is not negotiated
- #1065 - When relocating do not lookup STB_LOCAL symbols by name
- #1066 - Ignore ELF versioning table when object looks up symbols by name in itself (self-lookup)
- #1067 - Make symbols visible to the object itself from threads created by INIT functions
- #1070 - Running Linux apps compiled and linked against musl
- #1073 - Bugs in itimer::work()
- #1074 - Cannot run YCSB with Redis on OSV
- #1077 - malloc_usable_size of malloc_large returns incorrect value
- #1078 - Eliminate libnfs external module
Commits by Author
BassMatt (1):
Fotis Xenakis (9):
- pci: allow 64-bit BAR offsets
- pci: add batch capability discovery
- virtio-pci: discover shared memory regions
- virtio: expose shared memory regions
- virtio-fs: minor code improvements in driver
- virtio-fs: minor code improvements in filesystem
- virtio-fs: update fuse protocol header
- virtio-fs: add driver support for the DAX window
- docker: update default distros
Mason Davis (1):
Matthew Pabst (1):
Nadav Har'El (11):
- scripts/manifest_from_host.sh: write errors to stderr
- build: stop including libboost_program_options
- elf.cc: if Boost libraries aren't visible to applications, don't report them
- tests: fix warning in tst-mmap.cc
- tests: fix warning in tst-symlink
- tests: fix compilation warning in tests/tst-run.cc
- stdio: fix scanf bug
- submodules: cleanly remove unused submodules
- alarm(): fix crash when alarm is rearmed from alarm handler
- scripts/setup.py: fix Fedora package to python3
Waldemar Kozaczuk (158):
- Added patch/pull requests clarification to the main README
- options: implement light and simple alternative to boost program options
- loader: replace boost program_options with light options utility functions
- cpiod: reduce cpiod size by replacing boost program options with the custom light alternative
- httpserver: replace boost program_options with light options utility functions
- commands: remove program options
- tests: Add boost_system.so from externals to test manifest
- scripts: enhance build-capstan-mpm-packages
- scripts: make run.py behave correctly regardless where called from
- scripts: enhance tests/testing.py
- scripts: add automated testing framework
- scripts: tweaked compose_and_test_selected_apps to simplify building some golang test examples
- scripts: generate meta descriptor when creating kernel into capstan repo
- Support local-exec TLS access
- scripts: improved automated test scripts to explicitly report success
- firecracker: upgrade to the latest version 0.19.0 and enhance script to use api-less mode
- aarch64: make it build again
- firecracker: improve networking setup tools
- firecracker: enhance firecracker.py to save/restore terminal
- tests: enhance automated tests scripts to run on firecracker
- virtio-net: use large buffers when VIRTIO_NET_F_MRG_RXBUF is not negotiated
- libc: added __log2f_finite() alias
- tls: make local-exec tls aligned
- tls: parameterize static tls reservation
- procfs: implement /proc/self/exe
- virtio-mmio: do not pretend OSv implements version other than 2
- power down: cause triple fault if keyboard controller method did not work on non-acpi hypervisors
- virtio: adjust virtual queues initialization to adhere to the suggestions in the spec
- scripts: modify run.py to return easier to read output --dry-run
- scripts/run.py: allow to pass block device cache mode to qemu
- scripts/run.py: support running OSv on new QEMU microvm machine
- makefile: introduce loader_options.ld as a better way to pass APP_LOCAL_EXEC_TLS_SIZE to the linker script
- makefile: move loader_options.ld one level up
- makefile: remove redundant linker script for vmlinuz.bin which causes gcc 9.2 linker error
- scripts: enhance test_app_with_test_script.py to allow running with vhost/tap networking
- httpserver-jvm-plugin: remove dependency on externals
- unit tests: replace deprecated BOOST_MESSAGE with BOOST_TEST_MESSAGE
- vfs: make fcntl() handle F_DUPFD_CLOEXEC
- tests: clean makefile to print quiet messages consistenly
- tst-fallocate: replace mktemp with mkstemp to fix compiler warnings
- tst-truncate: replace tmpnam with mkstemp to fix compiler warnings
- elf: reverse unloading of objects in needed list
- tst-fs-link: replace mktemp with mkstemp to fix compiler warnings
- tst-zfs-mount: replace mktemp with mkstemp to fix compiler warnings
- misc-fs-stress: replace mktemp with mkstemp to fix compiler warnings
- tests: replace tmpnam() with mkdtemp() in implementation of TempDir
- libtools: make gcc compiler messages quiet
- tests: stop using boost from externals
- elf: add debug statement aiming to help diagnose problems
- zfs: do not import extra pools unless requested
- zfs: default device name to vblk0.1 if vdev_path different than /dev/vblk*
- elf: when relocating do not lookup STB_LOCAL symbols by name
- kvm: keep wall clock in sync with host
- elf: do not lookup symbol twice in resolve_pltgot
- arm: moved constructor init function in preparation for next patch
- arm: do not relocate DTB, instead read it ahead of time
- elf: make symbols visible to the object itself from threads created by INIT functions
- procfs: add meminfo pseudo-file
- glibc: add explicit_bzero() function
- libc: expose it as musl
- elf: fix thread visiblity issue
- thread: make walking parent-child thread relationship safer
- elf: ignore versioning table if symbol is looked up by object itself
- git: ignore JSON Compilation Database files used by CLion IDE
- elf: separate loading segments from processing headers
- httpserver: use libyaml header from host instead of external
- externals: eliminate unused glibcbase build parameter
- externals: removed unneeded x64 git submodules - glibc.bin and glibc-testsuite
- java: move java_api and jni_helpers to java modules
- external x64: move java balloon code out of kernel into the module java-base
- external x64: replace openjdk7 module with with new openjdk8-from-host one
- external x64: remove misc.bin
- external x64: remove gcc.bin
- modules: add simple sysinfo utility
- setup: support Fedora 30 and 31
- virtio-fs: add fuse protocol header
- virtio-fs: initial implementation of read-only subset
- aarch64: fixed linker error introduced with virtio-fs changes
- tests: add bionic tests to the source tree
- tests: improve regex rules to catch failures in tst-dlfcn
- scripts: upgrade to python 3 - part 1.1
- scripts: migrate setup.py to python 3
- httpserver: properly handle chunked POST requests with body
- scripts: upgrade to python 3 - part 2
- scripts: upgrade to python 3 - part 1.2
- scripts: change firecracker.py to default to '--no-api' mode
- scripts: re-enable tcp_close_without_reading to run on firecracker
- scripts: upgrade to python 3 - part 3
- scripts: enhance run.py to allow running OSv on arm
- scripts/loader.py: make osv info callouts work with older C++ ABI
- aarch64: support JUMP_SLOT relocation in the kernel
- scripts: use strip command as set in makefile so it works properly with aarch64
- scripts: new script to download and extract arbitrary Fedora rpm
- externals: enhance setup.py to download aarch64 gcc and boost code and libraries
- scripts/setup.py: add support of Ubuntu 19.10
- memmove: fix a bug in memmove_backwards
- scripts/setup.py: refactored by moving common packages to distribution level
- scripts/setup.py: move code to download aarch64 packages to its own script
- docker: add new docker files to build OSv 'runner' images
- docker: added build hook file to allow specifying particular fedora or ubuntu version
- docker: add docker files defining base images
- scripts: remove redundant parsing of arguments in run.py
- scripts: enhance test.py to allow blacklisting tests and specifying test manifest
- scripts: fix build-capstan-mpm-packages to remove obsolete fonts module
- docker: changed builder file to git clone full
- travis CI: setup nightly publishing of OSv artifacts
- travis: clean the build tree just in case
- externals: eliminate aarch64 gcc and boost libraries from externals
- dl_tests: fix the makefile to not duplicate building some artifacts
- cloud-init: replace boost program_options with light options utility functions
- libc: make timerfd_* functions handle wall clock jumps
- libc: refine how timerfd_* functions handle wall clock jumps
- virtio: optimize number of exits when parsing capabilities
- virtio: extract common logic to map capability bar
- httpserver: use openjdk8-from-host module in tests
- fs: move nfs support out of kernel in lieu of a separate pluggable module
- tests: move nfs testing out to a separate module
- httpserver: allow compiling out features not needed for monitoring api
- httpserver: add read-only monitoring module
- httpserver: add monitoring API unit tests
- mempool: fix a bug in page_range_allocator() when handling worst case O(n) scenario
- mempool: use map_anon() for large allocations or when memory is fragmented
- httpserver: fix monitoring module Makefile to properly generate JSON cc files
- module.py: do not resolve java-base for non-java apps
- bootfs: fix harmless message about complaining libgcc_s.so.1 when uploading manifest
- build: reference source bootfs.manifest.skel to prevent rebuiding kernel everytime for ZFS build
- build cleanup: export files regardless of filesystem chosen
- make: add new uniform kernel.elf artifact intended for direct-kernel boot mode
- build: allow building kernel-less ZFS and ROFS disk images
- certs: make certificates work with apps running on firecracker
- httpserver-api: add --kernel_path parameter to allow running tests with specific kernel
- tests: allow building ZFS and ROFS versions of test mps
- scripts: do not resolve symlink targets when exporting -
- capstan: add kernel.elf and httpserver monitoring module to the list of generated artifacts
- tests: improve automated testing framework
- travis: fix build to take into account renamed unit tests artifacts
- zfs: avoid f_fsid sign extension when translating f_fsid to va_fsid
- zfs: modify fsid_guid to store extra bits to allow detecting ZFS in pagecache
- rofs: report fsid (filesystem ID) properly
- sysfs: add new pseudo-files free_page_ranges and pools to help monitor memory utilization
- pagecache: refactor to allow integration with non-ZFS filesystems
- rofs: optimize memory utilization by integrating with page cache
- travis: simplify CIRP publishing and add new artifacts
- docker: refine build docker files
- travis: fix the build
- loader: add option to mount extra filesystem
- run.py: add support of mounting directory via virtiofs device
- socket: warn about missing ancillary information handling