Parse Kotlin's metadata + Remove suspend_fun_to_async flag in config (

#308)

* Parse Kotlin's metadata in API Summarizer
* Remove `suspend_fun_to_async` from config now that `isSuspend` reliably detect `suspend fun`s.

README.md

@@ -218,7 +218,6 @@ A `*` denotes required configuration.
 | `source_path`          | List of directory paths | Directories to search for source files. Note: source_path for dependencies downloaded using `maven_downloads` configuration is added automatically without the need to specify here. |
 | `class_path`           | List of directory / JAR paths | Classpath for API summary generation. This should include any JAR dependencies of the source files in `source_path`. |
 | `classes` *            | List of qualified class / package names | List of qualified class / package names. `source_path` will be scanned assuming the sources follow standard java-ish hierarchy. That is a.b.c either maps to a directory `a/b/c` or a class file `a/b/c.java`.  |
-| `suspend_fun_to_async` | True/False | Converting Kotlin's suspend functions to Dart's async functions. Defaults to False.  |
 | `output:`              | (Subsection) | This subsection will contain configuration related to output files. |
 | `output:` >> `bindings_type` | `c_based` (default) or `dart_only` | Binding generation strategy. [Trade-offs](#pure-dart-bindings) are explained at the end of this document. |
 | `output:` >> `c:`      | (Subsection) | This subsection specified C output configuration. Required if `bindings_type` is `c_based`. |

jnigen/CHANGELOG.md

@@ -1,3 +1,6 @@
+## 0.6.0-dev.0
+* **Breaking Change** Removed `suspend_fun_to_async` flag from the config. It's now happening by default since we read the Kotlin's metadata and reliably identify the `suspend fun`s.
+
 ## 0.5.0
 * **Breaking Change** ([#72](https://github.com/dart-lang/jnigen/issues/72)): Removed support for `importMap` in favor of the newly added interop mechanism with importing yaml files.
 * **Breaking Change** ([#72](https://github.com/dart-lang/jnigen/issues/72)): `java.util.Set`, `java.util.Map`, `java.util.List`, `java.util.Iterator` and the boxed types like `java.lang.Integer`, `java.lang.Double`, ... will be generated as their corresponding classes in `package:jni`.

jnigen/java/README.md

@@ -1,16 +1,19 @@
 ## ApiSummarizer
+
 An early version of ApiSummarizer.
 
 It analyzes java source code / jars and outputs a JSON representation of the public API.
 
 It's currently used in `jnigen` to get the information of the Java API.
 
 ## Build
+
 When using it via `jnigen`, the `jnigen:setup` script will take care of building the jar in appropriate location.
 
 To build the jar manually, run `mvn compile` in project root. To build the jar and run the tests as well, run `mvn test`. The jar will be created in `target/` directory.
 
 ## Command line
+
 ```
 usage: java -jar <JAR> [-s <SOURCE_DIR=.>] [-c <CLASSES_JAR>]
 <CLASS_OR_PACKAGE_NAMES>
@@ -27,13 +30,14 @@ or 'asm').
 -v,--verbose              Enable verbose output
 ```
 
-Here class or package names are specified as fully qualified names, for example `org.apache.pdfbox.pdmodel.PDDocument` will load `org/apache/pdfbox/pdmodel/PDDocument.java`. It assumes the package naming reflects directory structure. If such mapping results in a directory, for example `android.os` is given and a directory `android/os` is found under the source path, it is considered as a package and all Java source files under that directory are loaded recursively. 
+Here class or package names are specified as fully qualified names, for example `org.apache.pdfbox.pdmodel.PDDocument` will load `org/apache/pdfbox/pdmodel/PDDocument.java`. It assumes the package naming reflects directory structure. If such mapping results in a directory, for example `android.os` is given and a directory `android/os` is found under the source path, it is considered as a package and all Java source files under that directory are loaded recursively.
 
 Note that some options are directly forwarded to the underlying tool.
 
 ApiSummarizer's current use is in `jnigen` for obtaining public API of java packages. Only the features strictly required for that purpose are focused upon.
 
 ## Running tests
+
 Run `mvn surefire:test`
 
 There are not many tests at the moment. We plan to add some later.
@@ -43,4 +47,5 @@ There are not many tests at the moment. We plan to add some later.
 The main backend is based on javadoc API and generates summary based on java sources. A more experimental ASM backend also exists, and works somewhat okay-ish. It can summarize the compiled JARs. However, compiled jars without debug information do not include method parameter names. Some basic renaming is applied, i.e If type is `Object`, the parameter name will be output as `object` if an actual name is absent.
 
 ## TODO
-See issue #23.
+
+See issue #23.

jnigen/java/pom.xml

@@ -33,6 +33,11 @@
             <artifactId>asm-tree</artifactId>
             <version>9.3</version>
         </dependency>
+        <dependency>
+            <groupId>org.jetbrains.kotlinx</groupId>
+            <artifactId>kotlinx-metadata-jvm</artifactId>
+            <version>0.6.2</version>
+        </dependency>
         <dependency>
             <groupId>junit</groupId>
             <artifactId>junit</artifactId>

jnigen/java/src/main/java/com/github/dart_lang/jnigen/apisummarizer/disasm/AsmClassVisitor.java

@@ -91,6 +91,7 @@ public MethodVisitor visitMethod(
       return null;
     }
     method.name = name;
+    method.descriptor = descriptor;
     var type = Type.getType(descriptor);
     var params = new ArrayList<Param>();
     var paramTypes = type.getArgumentTypes();
@@ -117,6 +118,14 @@ public MethodVisitor visitMethod(
     return new AsmMethodVisitor(method);
   }
 
+  @Override
+  public AnnotationVisitor visitAnnotation(String descriptor, boolean visible) {
+    if (descriptor.equals("Lkotlin/Metadata;")) {
+      return new KotlinMetadataAnnotationVisitor(peekVisiting());
+    }
+    return super.visitAnnotation(descriptor, visible);
+  }
+
   @Override
   public void addAnnotation(JavaAnnotation annotation) {
     peekVisiting().annotations.add(annotation);

jnigen/java/src/main/java/com/github/dart_lang/jnigen/apisummarizer/disasm/KotlinMetadataAnnotationVisitor.java

@@ -0,0 +1,105 @@
+// Copyright (c) 2023, the Dart project authors. Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+package com.github.dart_lang.jnigen.apisummarizer.disasm;
+
+import com.github.dart_lang.jnigen.apisummarizer.elements.ClassDecl;
+import com.github.dart_lang.jnigen.apisummarizer.elements.KotlinClass;
+import java.util.ArrayList;
+import java.util.List;
+import kotlinx.metadata.jvm.KotlinClassHeader;
+import kotlinx.metadata.jvm.KotlinClassMetadata;
+import org.objectweb.asm.AnnotationVisitor;
+
+/**
+ * The format of Kotlin's metadata can be found here:
+ * https://kotlinlang.org/api/latest/jvm/stdlib/kotlin/-metadata/
+ */
+public class KotlinMetadataAnnotationVisitor extends AnnotationVisitor {
+  private ClassDecl decl;
+
+  private int kind;
+  private int[] metadataVersion;
+  private List<String> data1 = new ArrayList<>();
+  private List<String> data2 = new ArrayList<>();
+  private String extraString;
+  private String packageName;
+  private int extraInt;
+
+  public KotlinMetadataAnnotationVisitor(ClassDecl decl) {
+    super(AsmConstants.API);
+    this.decl = decl;
+  }
+
+  @Override
+  public void visit(String name, Object value) {
+    switch (name) {
+      case "k":
+        kind = (int) value;
+        return;
+      case "mv":
+        metadataVersion = (int[]) value;
+        return;
+      case "xs":
+        extraString = (String) value;
+        return;
+      case "pn":
+        packageName = (String) value;
+        return;
+      case "xi":
+        extraInt = (int) value;
+    }
+  }
+
+  @Override
+  public AnnotationVisitor visitArray(String name) {
+    List<String> arr;
+    switch (name) {
+      case "d1":
+        arr = data1;
+        break;
+      case "d2":
+        arr = data2;
+        break;
+      default:
+        return super.visitArray(name);
+    }
+    return new AnnotationVisitor(AsmConstants.API) {
+      @Override
+      public void visit(String name, Object value) {
+        arr.add((String) value);
+        super.visit(name, value);
+      }
+    };
+  }
+
+  @Override
+  public void visitEnd() {
+    var header =
+        new KotlinClassHeader(
+            kind,
+            metadataVersion,
+            data1.toArray(String[]::new),
+            data2.toArray(String[]::new),
+            extraString,
+            packageName,
+            extraInt);
+    var metadata = KotlinClassMetadata.read(header);
+    if (metadata instanceof KotlinClassMetadata.Class) {
+      decl.kotlinClass =
+          KotlinClass.fromKmClass(((KotlinClassMetadata.Class) metadata).toKmClass());
+    } else if (metadata instanceof KotlinClassMetadata.FileFacade) {
+      // TODO(#301): Handle file facades.
+    } else if (metadata instanceof KotlinClassMetadata.SyntheticClass) {
+      // Ignore synthetic classes such as lambdas.
+    } else if (metadata instanceof KotlinClassMetadata.MultiFileClassFacade) {
+      // Ignore multi-file classes
+    } else if (metadata instanceof KotlinClassMetadata.MultiFileClassPart) {
+      // Ignore multi-file classes
+    } else if (metadata instanceof KotlinClassMetadata.Unknown) {
+      // Unsupported
+    }
+    super.visitEnd();
+  }
+}

jnigen/java/src/main/java/com/github/dart_lang/jnigen/apisummarizer/elements/ClassDecl.java

@@ -37,6 +37,7 @@ public class ClassDecl {
   public boolean hasInstanceInit;
   public JavaDocComment javadoc;
   public List<JavaAnnotation> annotations;
+  public KotlinClass kotlinClass;
 
   /** In case of enum, names of enum constants */
   public List<String> values = new ArrayList<>();

jnigen/java/src/main/java/com/github/dart_lang/jnigen/apisummarizer/elements/KotlinClass.java

@@ -0,0 +1,62 @@
+// Copyright (c) 2023, the Dart project authors. Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+package com.github.dart_lang.jnigen.apisummarizer.elements;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import kotlinx.metadata.KmClass;
+import kotlinx.metadata.jvm.JvmExtensionsKt;
+
+public class KotlinClass {
+  public String name;
+  public String moduleName;
+  public List<KotlinFunction> functions;
+  public List<KotlinProperty> properties;
+  public List<KotlinConstructor> constructors;
+  public List<KotlinTypeParameter> typeParameters;
+  public List<KotlinType> contextReceiverTypes;
+  public List<KotlinType> superTypes;
+  public List<String> nestedClasses;
+  public List<String> enumEntries;
+  public List<String> sealedClasses;
+  public String companionObject;
+  public String inlineClassUnderlyingPropertyName;
+  public KotlinType inlineClassUnderlyingType;
+  public int flags;
+  public int jvmFlags;
+
+  public static KotlinClass fromKmClass(KmClass c) {
+    var klass = new KotlinClass();
+    klass.name = c.getName();
+    klass.moduleName = JvmExtensionsKt.getModuleName(c);
+    klass.functions =
+        c.getFunctions().stream().map(KotlinFunction::fromKmFunction).collect(Collectors.toList());
+    klass.properties =
+        c.getProperties().stream().map(KotlinProperty::fromKmProperty).collect(Collectors.toList());
+    klass.constructors =
+        c.getConstructors().stream()
+            .map(KotlinConstructor::fromKmConstructor)
+            .collect(Collectors.toList());
+    klass.typeParameters =
+        c.getTypeParameters().stream()
+            .map(KotlinTypeParameter::fromKmTypeParameter)
+            .collect(Collectors.toList());
+    klass.contextReceiverTypes =
+        c.getContextReceiverTypes().stream()
+            .map(KotlinType::fromKmType)
+            .collect(Collectors.toList());
+    klass.superTypes =
+        c.getSupertypes().stream().map(KotlinType::fromKmType).collect(Collectors.toList());
+    klass.enumEntries = c.getEnumEntries();
+    klass.flags = c.getFlags();
+    klass.jvmFlags = JvmExtensionsKt.getJvmFlags(c);
+    klass.nestedClasses = c.getNestedClasses();
+    klass.companionObject = c.getCompanionObject();
+    klass.inlineClassUnderlyingPropertyName = c.getInlineClassUnderlyingPropertyName();
+    klass.inlineClassUnderlyingType = KotlinType.fromKmType(c.getInlineClassUnderlyingType());
+    klass.sealedClasses = c.getSealedSubclasses();
+    return klass;
+  }
+}

jnigen/java/src/main/java/com/github/dart_lang/jnigen/apisummarizer/elements/KotlinConstructor.java

@@ -0,0 +1,30 @@
+// Copyright (c) 2023, the Dart project authors. Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+package com.github.dart_lang.jnigen.apisummarizer.elements;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import kotlinx.metadata.KmConstructor;
+import kotlinx.metadata.jvm.JvmExtensionsKt;
+
+public class KotlinConstructor {
+  public String name;
+  public String descriptor;
+  public List<KotlinValueParameter> valueParameters;
+  public int flags;
+
+  public static KotlinConstructor fromKmConstructor(KmConstructor c) {
+    var ctor = new KotlinConstructor();
+    ctor.flags = c.getFlags();
+    var signature = JvmExtensionsKt.getSignature(c);
+    ctor.name = signature == null ? null : signature.getName();
+    ctor.descriptor = signature == null ? null : signature.getDesc();
+    ctor.valueParameters =
+        c.getValueParameters().stream()
+            .map(KotlinValueParameter::fromKmValueParameter)
+            .collect(Collectors.toList());
+    return ctor;
+  }
+}

jnigen/java/src/main/java/com/github/dart_lang/jnigen/apisummarizer/elements/KotlinFunction.java

@@ -0,0 +1,55 @@
+// Copyright (c) 2023, the Dart project authors. Please see the AUTHORS file
+// for details. All rights reserved. Use of this source code is governed by a
+// BSD-style license that can be found in the LICENSE file.
+
+package com.github.dart_lang.jnigen.apisummarizer.elements;
+
+import java.util.List;
+import java.util.stream.Collectors;
+import kotlinx.metadata.Flag;
+import kotlinx.metadata.KmFunction;
+import kotlinx.metadata.jvm.JvmExtensionsKt;
+
+public class KotlinFunction {
+  /** Name in the byte code. */
+  public String name;
+
+  public String descriptor;
+
+  /** Name in the Kotlin's metadata. */
+  public String kotlinName;
+
+  public List<KotlinValueParameter> valueParameters;
+  public KotlinType returnType;
+  public KotlinType receiverParameterType;
+  public List<KotlinType> contextReceiverTypes;
+  public List<KotlinTypeParameter> typeParameters;
+  public int flags;
+  public boolean isSuspend;
+
+  public static KotlinFunction fromKmFunction(KmFunction f) {
+    var fun = new KotlinFunction();
+    var signature = JvmExtensionsKt.getSignature(f);
+    fun.descriptor = signature == null ? null : signature.getDesc();
+    fun.name = signature == null ? null : signature.getName();
+    fun.kotlinName = f.getName();
+    fun.flags = f.getFlags();
+    // Processing the information needed from the flags.
+    fun.isSuspend = Flag.Function.IS_SUSPEND.invoke(fun.flags);
+    fun.valueParameters =
+        f.getValueParameters().stream()
+            .map(KotlinValueParameter::fromKmValueParameter)
+            .collect(Collectors.toList());
+    fun.returnType = KotlinType.fromKmType(f.getReturnType());
+    fun.receiverParameterType = KotlinType.fromKmType(f.getReceiverParameterType());
+    fun.contextReceiverTypes =
+        f.getContextReceiverTypes().stream()
+            .map(KotlinType::fromKmType)
+            .collect(Collectors.toList());
+    fun.typeParameters =
+        f.getTypeParameters().stream()
+            .map(KotlinTypeParameter::fromKmTypeParameter)
+            .collect(Collectors.toList());
+    return fun;
+  }
+}