Doc updates (#385)

* Documentation updates * Documentation updates * Update README.md * Fix tuple indexing error messages * Update roadmap, differences * Update README, FAQ * Trim newline * Update README.md * Update README.md * Update README.md * Update roadmap.md * Update cpp.md * Update README.md * Update roadmap.md * Update README.md * Fix test * clang-format * Fix exporting function named "main" * Update export test * Fix paths * Rename extra/python -> jit * Update license change date * Minor docs updates * Re-add __init__.py * Update header * Update gitignore * Update README.md --------- Co-authored-by: Ibrahim Numanagić <ibrahimpasa@gmail.com> Co-authored-by: Ibrahim Numanagić <inumanag@users.noreply.github.com>
exaloop · May 23, 2023 · 38e08b4 · 38e08b4
1 parent 5085dae
commit 38e08b4
Show file tree

Hide file tree

Showing 25 changed files with 253 additions and 57 deletions.
diff --git a/.gitignore b/.gitignore
@@ -58,14 +58,16 @@ Thumbs.db
 .mypy_cache
 .vscode
 .cache
-
-extra/jupyter/share/jupyter/kernels/codon/kernel.json
-extra/python/codon/version.py
-scratch*.*
-_*
 .ipynb_checkpoints
 
+# CMake generated files #
+#########################
+jupyter/share/jupyter/kernels/codon/kernel.json
+jit/codon/version.py
+
 # Testing files #
 #################
 temp/
 playground/
+scratch*.*
+_*
diff --git a/CMakeLists.txt b/CMakeLists.txt
@@ -8,7 +8,7 @@ set(CODON_JIT_PYTHON_VERSION "0.1.5")
 configure_file("${PROJECT_SOURCE_DIR}/cmake/config.h.in"
                "${PROJECT_SOURCE_DIR}/codon/config/config.h")
 configure_file("${PROJECT_SOURCE_DIR}/cmake/config.py.in"
-               "${PROJECT_SOURCE_DIR}/extra/python/codon/version.py")
+               "${PROJECT_SOURCE_DIR}/jit/codon/version.py")
 
 option(CODON_GPU "build Codon GPU backend" OFF)
 
@@ -465,5 +465,5 @@ install(FILES ${CMAKE_BINARY_DIR}/libomp${CMAKE_SHARED_LIBRARY_SUFFIX} DESTINATI
 install(TARGETS codon DESTINATION bin)
 install(DIRECTORY ${CMAKE_BINARY_DIR}/include/codon DESTINATION include)
 install(DIRECTORY ${CMAKE_SOURCE_DIR}/stdlib DESTINATION lib/codon)
-install(DIRECTORY ${CMAKE_SOURCE_DIR}/extra/python/ DESTINATION python)
+install(DIRECTORY ${CMAKE_SOURCE_DIR}/jit/ DESTINATION python)
 install(DIRECTORY DESTINATION lib/codon/plugins)
diff --git a/LICENSE b/LICENSE
@@ -14,7 +14,7 @@ Licensed Work:        Codon compiler, runtime, and standard library
 
 Additional Use Grant: None
 
-Change Date:          2025-11-01
+Change Date:          2026-05-01
 
 Change License:       Apache License, Version 2.0
 

diff --git a/README.md b/README.md
@@ -4,15 +4,15 @@
 
 <h3 align="center">
   <a href="https://docs.exaloop.io/codon" target="_blank"><b>Docs</b></a>
-  &nbsp;&#65372;&nbsp;
+  &nbsp;&#183;&nbsp;
   <a href="https://docs.exaloop.io/codon/general/faq" target="_blank"><b>FAQ</b></a>
-  &nbsp;&#65372;&nbsp;
+  &nbsp;&#183;&nbsp;
   <a href="https://blog.exaloop.io" target="_blank"><b>Blog</b></a>
-  &nbsp;&#65372;&nbsp;
-  <a href="https://github.com/exaloop/codon/discussions" target="_blank"><b>Forum</b></a>
-  &nbsp;&#65372;&nbsp;
+  &nbsp;&#183;&nbsp;
   <a href="https://join.slack.com/t/exaloop/shared_invite/zt-1jusa4kc0-T3rRWrrHDk_iZ1dMS8s0JQ" target="_blank">Chat</a>
-  &nbsp;&#65372;&nbsp;
+  &nbsp;&#183;&nbsp;
+  <a href="https://docs.exaloop.io/codon/general/roadmap" target="_blank">Roadmap</a>
+  &nbsp;&#183;&nbsp;
   <a href="https://exaloop.io/benchmarks" target="_blank">Benchmarks</a>
 </h3>
 
@@ -23,10 +23,36 @@
 
 ## What is Codon?
 
-Codon is a high-performance Python compiler that compiles Python code to native machine code without any runtime overhead.
-Typical speedups over Python are on the order of 10-100x or more, on a single thread. Codon's performance is typically on par with
-(and sometimes better than) that of C/C++. Unlike Python, Codon supports native multithreading, which can lead to speedups many
-times higher still. Codon grew out of the [Seq project](https://github.com/seq-lang/seq).
+Codon is a high-performance Python implementation that compiles to native machine code without
+any runtime overhead. Typical speedups over vanilla Python are on the order of 10-100x or more, on
+a single thread. Codon's performance is typically on par with (and sometimes better than) that of
+C/C++. Unlike Python, Codon supports native multithreading, which can lead to speedups many times
+higher still.
+
+*Think of Codon as Python reimagined for static, ahead-of-time compilation, built from the ground
+up with best possible performance in mind.*
+
+### Goals
+
+- :bulb: **No learning curve:** Be as close to CPython as possible in terms of syntax, semantics and libraries
+- :rocket: **Top-notch performance:** At *least* on par with low-level languages like C, C++ or Rust
+- :computer: **Hardware support:** Full, seamless support for multicore programming, multithreading (no GIL!), GPU and more
+- :chart_with_upwards_trend: **Optimizations:** Comprehensive optimization framework that can target high-level Python constructs
+  and libraries
+- :battery: **Interoperability:** Full interoperability with Python's ecosystem of packages and libraries
+
+### Non-goals
+
+- :x: *Drop-in replacement for CPython:* Codon is not a drop-in replacement for CPython. There are some
+  aspects of Python that are not suitable for static compilation — we don't support these in Codon.
+  There are ways to use Codon in larger Python codebases via its [JIT decorator](https://docs.exaloop.io/codon/interoperability/decorator)
+  or [Python extension backend](https://docs.exaloop.io/codon/interoperability/pyext). Codon also supports
+  calling any Python module via its [Python interoperability](https://docs.exaloop.io/codon/interoperability/python).
+  See also [*"Differences with Python"*](https://docs.exaloop.io/codon/general/differences) in the docs.
+
+- :x: *New syntax and language constructs:* We try to avoid adding new syntax, keywords or other language
+  features as much as possible. While Codon does add some new syntax in a couple places (e.g. to express
+  parallelism), we try to make it as familiar and intuitive as possible.
 
 ## Install
 
@@ -76,9 +102,21 @@ codon build -release -llvm fib.py
 
 See [the docs](https://docs.exaloop.io/codon/general/intro) for more options and examples.
 
-This prime counting example showcases Codon's [OpenMP](https://www.openmp.org/) support, enabled with the addition of one line.
-The `@par` annotation tells the compiler to parallelize the following `for`-loop, in this case using a dynamic schedule, chunk size
-of 100, and 16 threads.
+You can import and use any Python package from Codon. For example:
+
+```python
+from python import matplotlib.pyplot as plt
+data = [x**2 for x in range(10)]
+plt.plot(data)
+plt.show()
+```
+
+(Just remember to set the `CODON_PYTHON` environment variable to the CPython shared library,
+as explained in the [the docs](https://docs.exaloop.io/codon/interoperability/python).)
+
+This prime counting example showcases Codon's [OpenMP](https://www.openmp.org/) support, enabled
+with the addition of one line. The `@par` annotation tells the compiler to parallelize the
+following `for`-loop, in this case using a dynamic schedule, chunk size of 100, and 16 threads.
 
 ```python
 from sys import argv
@@ -133,16 +171,6 @@ mandelbrot(pixels, grid=(N*N)//1024, block=1024)
 
 GPU programming can also be done using the `@par` syntax with `@par(gpu=True)`.
 
-## What isn't Codon?
-
-While Codon supports nearly all of Python's syntax, it is not a drop-in replacement, and large codebases might require modifications
-to be run through the Codon compiler. For example, some of Python's modules are not yet implemented within Codon, and a few of Python's
-dynamic features are disallowed. The Codon compiler produces detailed error messages to help identify and resolve any incompatibilities.
-
-Codon can be used within larger Python codebases via the [`@codon.jit` decorator](https://docs.exaloop.io/codon/interoperability/decorator).
-Plain Python functions and libraries can also be called from within Codon via
-[Python interoperability](https://docs.exaloop.io/codon/interoperability/python).
-
 ## Documentation
 
 Please see [docs.exaloop.io](https://docs.exaloop.io/codon) for in-depth documentation.
diff --git a/codon/cir/llvm/llvisitor.cpp b/codon/cir/llvm/llvisitor.cpp
@@ -26,6 +26,9 @@ const std::string EXPORT_ATTR = "std.internal.attributes.export";
 const std::string INLINE_ATTR = "std.internal.attributes.inline";
 const std::string NOINLINE_ATTR = "std.internal.attributes.noinline";
 const std::string GPU_KERNEL_ATTR = "std.gpu.kernel";
+
+const std::string MAIN_UNCLASH = ".main.unclash";
+const std::string MAIN_CTOR = ".main.ctor";
 } // namespace
 
 llvm::DIFile *LLVMVisitor::DebugInfo::getFile(const std::string &path) {
@@ -426,18 +429,24 @@ void executeCommand(const std::vector<std::string> &args) {
 
 void LLVMVisitor::setupGlobalCtorForSharedLibrary() {
   const std::string llvmCtor = "llvm.global_ctors";
-  auto *main = M->getFunction("main");
-  if (M->getNamedValue(llvmCtor) || !main)
+  if (M->getNamedValue(llvmCtor))
     return;
-  main->setName(".main"); // avoid clash with other main
+
+  auto *main = M->getFunction(MAIN_UNCLASH);
+  if (!main) {
+    main = M->getFunction("main");
+    if (!main)
+      return;
+    main->setName(MAIN_UNCLASH); // avoid clash with other main
+  }
 
   auto *ctorFuncTy = llvm::FunctionType::get(B->getVoidTy(), {}, /*isVarArg=*/false);
   auto *ctorEntryTy = llvm::StructType::get(B->getInt32Ty(), ctorFuncTy->getPointerTo(),
                                             B->getInt8PtrTy());
   auto *ctorArrayTy = llvm::ArrayType::get(ctorEntryTy, 1);
 
-  auto *ctor = cast<llvm::Function>(
-      M->getOrInsertFunction(".main.ctor", ctorFuncTy).getCallee());
+  auto *ctor =
+      cast<llvm::Function>(M->getOrInsertFunction(MAIN_CTOR, ctorFuncTy).getCallee());
   ctor->setLinkage(llvm::GlobalValue::InternalLinkage);
   auto *entry = llvm::BasicBlock::Create(*context, "entry", ctor);
   B->SetInsertPoint(entry);
@@ -1122,7 +1131,7 @@ void LLVMVisitor::writeToPythonExtension(const PyModule &pymod,
   B->SetInsertPoint(block);
 
   if (auto *main = M->getFunction("main")) {
-    main->setName(".main");
+    main->setName(MAIN_UNCLASH);
     B->CreateCall({main->getFunctionType(), main}, {zero32, null});
   }
 
@@ -1461,8 +1470,10 @@ void LLVMVisitor::visit(const Module *x) {
   auto *strlenFunc = llvm::cast<llvm::Function>(
       M->getOrInsertFunction("strlen", B->getInt64Ty(), B->getInt8PtrTy()).getCallee());
 
+  // check if main exists already as an exported function
+  const std::string mainName = M->getFunction("main") ? MAIN_UNCLASH : "main";
   auto *canonicalMainFunc = llvm::cast<llvm::Function>(
-      M->getOrInsertFunction("main", B->getInt32Ty(), B->getInt32Ty(),
+      M->getOrInsertFunction(mainName, B->getInt32Ty(), B->getInt32Ty(),
                              B->getInt8PtrTy()->getPointerTo())
           .getCallee());
 

diff --git a/codon/runtime/re.cpp b/codon/runtime/re.cpp
@@ -73,8 +73,7 @@ template <typename KV> struct GCMapAllocator : public std::allocator<KV> {
   GCMapAllocator() = default;
   GCMapAllocator(GCMapAllocator<KV> const &) = default;
 
-  template <typename KV1>
-  GCMapAllocator(const GCMapAllocator<KV1>&) noexcept {}
+  template <typename KV1> GCMapAllocator(const GCMapAllocator<KV1> &) noexcept {}
 
   KV *allocate(std::size_t n) { return (KV *)seq_alloc(n * sizeof(KV)); }
 

diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
@@ -8,6 +8,7 @@
 * [Frequently asked questions](intro/faq.md)
 * [Differences with Python](intro/differences.md)
 * [Release notes](intro/releases.md)
+* [Roadmap](intro/roadmap.md)
 
 ## Language
 

diff --git a/docs/advanced/ir.md b/docs/advanced/ir.md
@@ -309,8 +309,8 @@ When subclassing nodes other than types (e.g. instructions, flows, etc.), be sur
 
 # Utilities
 
-The `codon/ir/util/` directory has a number of utility and generally helpful functions, for things like
-cloning IR, inlining/outlining, matching and more. `codon/ir/util/irtools.h` in particular has many helpful
+The `codon/cir/util/` directory has a number of utility and generally helpful functions, for things like
+cloning IR, inlining/outlining, matching and more. `codon/cir/util/irtools.h` in particular has many helpful
 functions for performing various common tasks. If you're working with CIR, be sure to take a look at these
 functions to make your life easier!
 

diff --git a/docs/interop/cpp.md b/docs/interop/cpp.md
@@ -15,16 +15,19 @@ can create a shared library containing `foo` (assuming source file
 *foo.codon*):
 
 ``` bash
-codon build -o libfoo.so foo.codon
+codon build --relocation-model=pic --lib -o libfoo.so foo.codon
 ```
 
-Now we can call `foo` from a C program:
+Now we can call `foo` from a C program (if you're using C++, mark the
+Codon function as `extern "C"`):
 
 ``` c
 #include <stdint.h>
 #include <stdio.h>
 
 int64_t foo(int64_t);
+// In C++, it would be:
+// extern "C" int64_t foo(int64_t);
 
 int main() {
   printf("%llu\n", foo(10));
@@ -34,12 +37,19 @@ int main() {
 Compile:
 
 ``` bash
-gcc -o foo -L. -lfoo foo.c
+gcc -o foo -L. -lfoo foo.c  # or g++ if using C++
 ```
 
 Now running `./foo` should invoke `foo()` as defined in Codon, with an
 argument of `10`.
 
+Note that if the generated shared library is in a non-standard path, you
+can either:
+
+- Add the `rpath` to the `gcc` command: `-Wl,-rpath=/path/to/lib/dir`
+- Add the library path to `LD_LIBRARY_PATH` (or `DYLD_LIBRARY_PATH` if
+  using macOS): `export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/lib/dir`.
+
 # Converting types
 
 The following table shows the conversions between Codon and C/C++ types:
@@ -51,5 +61,5 @@ The following table shows the conversions between Codon and C/C++ types:
   | `bool`    | `bool`                               |
   | `byte`    | `char` or `int8_t`                   |
   | `str`     | `{int64_t, char*}` (length and data) |
+  | `tuple`   | Struct of fields                     |
   | `class`   | Pointer to corresponding tuple       |
-  | `@tuple`  | Struct of fields                     |
diff --git a/docs/intro/differences.md b/docs/intro/differences.md
@@ -1,9 +1,12 @@
-While Codon's syntax and semantics are virtually identical
+While Codon's syntax and semantics are nearly identical
 to Python's, there are some notable differences that are
 worth mentioning. Most of these design decisions were made
 with the trade-off between performance and Python compatibility
 in mind.
 
+Please see our [roadmap](roadmap.md) for more information about
+how we plan to close some of these gaps in the future.
+
 # Data types
 
 - **Integers:** Codon's `int` is a 64-bit signed integer,
@@ -17,6 +20,10 @@ in mind.
 - **Dictionaries:** Codon's dictionary type does not preserve
   insertion order, unlike Python's as of 3.6.
 
+- **Tuples**: Since tuples compile down to structs, tuple lengths
+  must be known at compile time, meaning you can't convert an
+  arbitrarily-sized list to a tuple, for instance.
+
 # Type checking
 
 Since Codon performs static type checking ahead of time, a

diff --git a/docs/intro/faq.md b/docs/intro/faq.md
@@ -16,7 +16,22 @@ codebases might require modifications to be run through the Codon compiler. For
 of Python's modules are not yet implemented within Codon, and a few of Python's dynamic features
 are disallowed. The Codon compiler produces detailed error messages to help identify and resolve
 any incompatibilities. Codon supports seamless [Python interoperability](../interop/python.md) to
-handle cases where specific Python libraries or dynamism are required.
+handle cases where specific Python libraries or dynamism are required, and also supports writing
+[Python extension modules](../interop/pyext.md) that can be imported and used from larger Python
+codebases.
+
+## Why Codon?
+
+Python is arguably the world's most popular programming language, and is gradually becoming the
+*lingua franca* particularly amongst non-technical or non-CS practitioners in numerous fields.
+It provides a readable, clean syntax, is easy to learn, and has an unmatched ecosystem of libraries.
+However, Python's achilles heel has always been performance: a typical codebase in pure Python is
+orders of magnitude slower than its C/C++/Rust counterpart.
+
+Codon bridges the gap between Python's simplicity and ease-of-use, and the performance of low-level
+languages like C++ or Rust, by using [novel compiler and type checking techniques](https://dl.acm.org/doi/abs/10.1145/3578360.3580275)
+to statically compile code ahead-of-time, avoiding all of vanilla Python's runtime overhead and
+performance drawbacks.
 
 ## How does Codon compare to...
 
@@ -48,6 +63,12 @@ handle cases where specific Python libraries or dynamism are required.
   Codon type checks the entire program ahead of time. Codon also tries to circumvent the learning
   curve of a new language by adopting Python's syntax and semantics.
 
+- **Mojo?** Mojo strives to add low-level programming support/features to the Python language,
+  while also supporting the rest of Python by relying on CPython. By contrast, Codon aims to
+  make Python itself more performant by using new type checking and compilation techniques,
+  without trying to be a superset or drop-in replacement. Codon tries to minimize new syntax
+  and language features with respect to Python.
+
 You can see results from [Codon's benchmark suite](https://github.com/exaloop/codon/tree/develop/bench)
 suite at [exaloop.io/benchmarks](https://exaloop.io/benchmarks).
 More benchmarks can be found in the [2019 paper](https://dl.acm.org/doi/10.1145/3360551)
@@ -61,6 +82,9 @@ which can be used within Python codebases. This will compile only the annotated
 and automatically handle data conversions to and from Codon. It also allows for
 the use of any Codon-specific modules or extensions, such as multithreading.
 
+Codon can also [compile to Python extension modules](../interop/pyext.md) that can be
+imported and used from Python.
+
 ## What about interoperability with other languages and frameworks?
 
 Interoperability is and will continue to be a priority for Codon.