Distribution open connector archives appears in both samples & utilities

[planetf1]

This utility should only be in the distribution (via maven) once, not in both locations:

jonesn:egeria-omag-2.5/ (egeria-release-2.5) $ find . -name 'open-connector-archives*'                                        [9:19:04]
./samples/open-connector-archives-2.5-jar-with-dependencies.jar
./utilities/open-connector-archives-2.5-jar-with-dependencies.jar

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[mandy-chessell]

It looks like the archive is being removed from both places - we need it in one place since they should be imported into a metadata server that is cataloguing data assets.

[mandy-chessell]

Note this is an open metadata archive file of connector type definitions and is different from the connector implementation jar.

[planetf1]

The 'samples' and 'utilities' directories currently contain executable jars - ie they have a main and can be launched.

The connector archives is not and cannot be run as built: (sorry I have an old build for testing right now - presume same)

java -jar ./open-connector-archives-2.5-SNAPSHOT.jar                                                            [9:22:34]
no main manifest attribute, in ./open-connector-archives-2.5-SNAPSHOT.jar

If we look at an older version we see

jonesn:target/ (gradle9) $ jar -tf  ./open-connector-archives-2.5-SNAPSHOT.jar                                                             [9:22:40]
META-INF/MANIFEST.MF
META-INF/
org/
org/odpi/
org/odpi/openmetadata/
org/odpi/openmetadata/archiveutilities/
org/odpi/openmetadata/archiveutilities/openconnectors/
org/odpi/openmetadata/archiveutilities/openconnectors/datastoreconnectors/
org/odpi/openmetadata/archiveutilities/openconnectors/base/
META-INF/maven/
META-INF/maven/org.odpi.egeria/
META-INF/maven/org.odpi.egeria/open-connector-archives/
logback.xml
org/odpi/openmetadata/archiveutilities/openconnectors/datastoreconnectors/DataStoreConnectorsArchiveWriter.class
org/odpi/openmetadata/archiveutilities/openconnectors/datastoreconnectors/DataStoreConnectorsArchiveBuilder.class
org/odpi/openmetadata/archiveutilities/openconnectors/base/OpenConnectorArchiveBuilder.class
META-INF/maven/org.odpi.egeria/open-connector-archives/pom.xml
META-INF/maven/org.odpi.egeria/open-connector-archives/pom.properties

Looking at the current dependencies I see:

[INFO] --------------< org.odpi.egeria:open-connector-archives >---------------
[INFO] Building Open Connector Archives 2.8-SNAPSHOT                  [346/383]
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-dependency-plugin:3.1.2:tree (default-cli) @ open-connector-archives ---
[INFO] org.odpi.egeria:open-connector-archives:jar:2.8-SNAPSHOT
[INFO] +- org.odpi.egeria:repository-services-apis:jar:2.8-SNAPSHOT:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.7.30:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.12.2:compile
[INFO] |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.12.2:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.12.2:compile
[INFO] |  +- org.odpi.egeria:audit-log-framework:jar:2.8-SNAPSHOT:compile
[INFO] |  \- org.apache.commons:commons-collections4:jar:4.4:compile
[INFO] +- org.odpi.egeria:repository-services-archive-utilities:jar:2.8-SNAPSHOT:compile
[INFO] |  +- org.odpi.egeria:open-metadata-archive-file-connector:jar:2.8-SNAPSHOT:compile
[INFO] |  \- commons-io:commons-io:jar:2.8.0:compile
[INFO] +- org.odpi.egeria:open-metadata-types:jar:2.8-SNAPSHOT:compile
[INFO] |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
[INFO] +- org.testng:testng:jar:7.4.0:test
[INFO] |  +- com.beust:jcommander:jar:1.78:test
[INFO] |  \- org.webjars:jquery:jar:3.5.1:test
[INFO] +- ch.qos.logback:logback-classic:jar:1.2.3:compile
[INFO] +- org.odpi.egeria:open-connector-framework:jar:2.8-SNAPSHOT:compile
[INFO] +- org.odpi.egeria:cassandra-data-store-connector:jar:2.8-SNAPSHOT:compile
[INFO] |  \- com.datastax.oss:java-driver-core:jar:4.10.0:compile
[INFO] |     +- com.datastax.oss:native-protocol:jar:1.4.12:compile
[INFO] |     +- io.netty:netty-handler:jar:4.1.59.Final:compile
[INFO] |     |  +- io.netty:netty-common:jar:4.1.59.Final:compile
[INFO] |     |  +- io.netty:netty-resolver:jar:4.1.59.Final:compile
[INFO] |     |  +- io.netty:netty-buffer:jar:4.1.59.Final:compile
[INFO] |     |  +- io.netty:netty-transport:jar:4.1.59.Final:compile
[INFO] |     |  \- io.netty:netty-codec:jar:4.1.59.Final:compile
[INFO] |     +- com.datastax.oss:java-driver-shaded-guava:jar:25.1-jre-graal-sub-1:compile
[INFO] |     +- com.typesafe:config:jar:1.4.1:compile
[INFO] |     +- com.github.jnr:jnr-posix:jar:3.1.2:compile
[INFO] |     |  +- com.github.jnr:jnr-ffi:jar:2.2.0:compile
[INFO] |     |  |  +- com.github.jnr:jffi:jar:1.3.0:compile
[INFO] |     |  |  +- com.github.jnr:jffi:jar:native:1.3.0:runtime
[INFO] |     |  |  +- org.ow2.asm:asm:jar:7.1:compile
[INFO] |     |  |  +- org.ow2.asm:asm-commons:jar:7.1:compile
[INFO] |     |  |  +- org.ow2.asm:asm-analysis:jar:7.1:compile
[INFO] |     |  |  +- org.ow2.asm:asm-tree:jar:7.1:compile
[INFO] |     |  |  +- org.ow2.asm:asm-util:jar:7.1:compile
[INFO] |     |  |  +- com.github.jnr:jnr-a64asm:jar:1.0.0:compile
[INFO] |     |  |  \- com.github.jnr:jnr-x86asm:jar:1.0.2:compile
[INFO] |     |  \- com.github.jnr:jnr-constants:jar:0.10.0:compile
[INFO] |     +- io.dropwizard.metrics:metrics-core:jar:4.1.16:compile
[INFO] |     +- org.hdrhistogram:HdrHistogram:jar:2.1.12:compile
[INFO] |     +- com.esri.geometry:esri-geometry-api:jar:1.2.1:compile
[INFO] |     |  +- org.json:json:jar:20090211:compile
[INFO] |     |  \- org.codehaus.jackson:jackson-core-asl:jar:1.9.13-atlassian-2:runtime
[INFO] |     +- org.reactivestreams:reactive-streams:jar:1.0.3:compile
[INFO] |     +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] |     \- com.github.spotbugs:spotbugs-annotations:jar:4.2.2:compile
[INFO] |        \- com.google.code.findbugs:jsr305:jar:3.0.2:compile
[INFO] +- org.odpi.egeria:avro-file-connector:jar:2.8-SNAPSHOT:compile
[INFO] +- org.odpi.egeria:basic-file-connector:jar:2.8-SNAPSHOT:compile
[INFO] +- org.odpi.egeria:csv-file-connector:jar:2.8-SNAPSHOT:compile
[INFO] +- org.odpi.egeria:data-folder-connector:jar:2.8-SNAPSHOT:compile
[INFO] \- org.slf4j:slf4j-simple:jar:1.7.30:test

So there are classes relating to connectors included, but this is implementation, not types.. and those connectors (maybe not cassandra as it's moving) already generally feature in server/lib in the distribution

It does contain the archive reader and writer but without any guidance on execution, or a main, or an archive of connector type definitions it wasn't clear of the intent/where should be put.

In the 2.8 code we now have an 'uber' archive which includes these dependencies - but already plan to remove this big archives (in general) shortly. This archive includes the dependencies above, which again is a lot of implementation code for connectors

What do you think makes sense here? If we need specifically a type archive, this isn't it?

[mandy-chessell]

This should be the open metadata archive builder for connector type definitions. It definiately belongs in utilities not samples. I am not sure what has happened to the main because there is a main coded in DataStoreConnectorsArchiveWriter.

This is in the pom file ...

<plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <executions>
                    <execution>
                        <id>assemble-all</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                        <configuration>
                            <descriptorRefs>
                                <descriptorRef>jar-with-dependencies</descriptorRef>
                            </descriptorRefs>
                            <archive>
                                <manifest>
                                    <mainClass>org.odpi.openmetadata.archiveutilities.openconnectors.datastoreconnectors.DataStoreConnectorsArchiveWriter</mainClass>
                                </manifest>
                            </archive>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

The connector provider implementations are linked because the archive writer extracts the connector type information from the connector provider. It is possible to build it with the provided setting to create a skinny jar.

[mandy-chessell]

I have just run DataStoreConnectorsArchiveWriter and it correctly produced the archive file.
I also ran the jar file and it also worked successfully

[planetf1]

Thanks for clarifying. I was able to run after a clean build. I think my testing was incorrect after I'd be experimenting with build changes in my tree....

I think from this discussion we need to

• Add open-connector-archives back in (checking maven and gradle) as I previously removed it in #4280 remove open-connector-archives archive from samples/utilities i… #4897 (soon)

Since this is more than a build tool, but required by others, It would also be worth considering:

• whether we want to rename the file (makes it more obvious what it's purpose is)

• Documenting the purpose of the tool:

  • help within the tool (--help or similar?) & clearer output when run (rather than reporting about the GUID map)
  • Admin/install guide
  • reference for assembly (this could be specific READMEs added into the assembly directory structure)

  The change from uber to skinny is already covered more generally in Repackaging of egeria distribution & dependency streamlining #4667

[mandy-chessell]

I am not sure why this one is singled out. We have similar utilities for building the open metadata archives for the design model content and open metadata types. They should all be treated the same.

This is where the documentation is located:
https://egeria.odpi.org/open-metadata-resources/open-metadata-archives/

[planetf1]

The initial observation related to this specifically, stemming from the jar's duplication. through the explanation it's purpose then became clear. The documentation suggestions are an increment that I think could apply to all of our tools, and assembly content

I think the docs should be in two forms -- a guide, which comes from the design angle of how to configure/setup/build the system, as well as a reference which leads back from the directory structure. We also could do with an index of utilities. Currently the guide doesn't specifically call out this utility beyond the oneline readme, though it does have detail on the design. We could consider adding examples/use case

Only the first step is important now, since I've removed it, but the other suggestions should make egeria easier to consume I think - but they're not urgent in any way. I'm happy to leave it assigned to myself to try and draft some content.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 20 days if no further activity occurs. Thank you for your contributions.

[planetf1]

this was fixed in #4897