[Issue 4175] [pulsar-function-go] Create integration tests for Go Functions for production-readiness #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ng (apache#6101) *Motivation* Related to apache#6084 apache#5400 introduces `customRuntimeOptions` in function details. But the description was wrong. The mistake was probably introduced by bad merges. *Modification* Fix the argument and description for `deadletterTopic` and `customRuntimeOptions`.
### Motivation Pulsar dashboard is deprecated. We need to remove it from sidebar. ### Modifications 1. Remove Pulsar dashboard from the sidebar (for both master and 2.5.0 release); 2. Add related note and link to Pulsar dashboard in Pulsar manager.
…pache#6063) ### Motivation Some of the classes in the pulsar-functions module had a mixture of the following lombok annotations: ``` @DaTa @Setter @getter @EqualsAndHashCode @tostring ``` The [@DaTa](https://projectlombok.org/features/Data) annotation includes all other annotations: > All together now: A shortcut for @tostring, @EqualsAndHashCode, @getter on all fields, @Setter on all non-final fields, and @requiredargsconstructor! ### Modifications Removed `@Setter`, `@Getter`, `@EqualsAndHashCode`, and '@tostring' if the `@Data` annotation was also present
…ation on how to work with schemas (apache#6089) ### Motivation When I read the documentation on how to work with schemas I saw that some code blocks are not highlighted, because of incorrect language identifiers. ### Modifications Correct language identifiers were applied.
…pache#5915) * Allow to enable/disable delyed delivery for messages on namespace Signed-off-by: xiaolong.ran <rxl@apache.org> * add isDelayedDeliveryEnabled function Signed-off-by: xiaolong.ran <rxl@apache.org> * add delayed_delivery_time process logic Signed-off-by: xiaolong.ran <rxl@apache.org> * add test case Signed-off-by: xiaolong.ran <rxl@apache.org> * update admin cli docs Signed-off-by: xiaolong.ran <rxl@apache.org> * fix comments Signed-off-by: xiaolong.ran <rxl@apache.org> * fix comments Signed-off-by: xiaolong.ran <rxl@apache.org> * fix comments Signed-off-by: xiaolong.ran <rxl@apache.org> * update import lib Signed-off-by: xiaolong.ran <rxl@apache.org> * avoid import * Signed-off-by: xiaolong.ran <rxl@apache.org> * fix comments Signed-off-by: xiaolong.ran <rxl@apache.org> * fix comments Signed-off-by: xiaolong.ran <rxl@apache.org> * remove unuse code Signed-off-by: xiaolong.ran <rxl@apache.org> * fix comments Signed-off-by: xiaolong.ran <rxl@apache.org> * add test case for delayed delivery messages Signed-off-by: xiaolong.ran <rxl@apache.org> * fix comments Signed-off-by: xiaolong.ran <rxl@apache.org> * fix comments Signed-off-by: xiaolong.ran <rxl@apache.org>
Fixes apache#6114 Modifications The swagger files were generated for the version 2.5.0 according to the instruction provided by @tuteng
…pache#6095) Fixes apache#6094 ### Motivation Incorrect picture in pulsar function-overview ### Modifications Fix the last word "in" to "pudding".
…ache#6050) Fixes apache#3681 ### Motivation The client library doc content is out of date. ### Modifications 1. Pulsar supports more clients, add links to the new clients. 2. Remove redundant information, and will update each client respectively. 3. Update Pulsar client go info. 4. Proofread parts of the Python/Go client content while reading. 5. Fixes apache#3681 by adding examples in Python client. 6. Update Python API.
Fixes apache#5620 Added support for Python native logging within the Pulsar Client Python wrapper. * added a new Logger implementation that forwards the logging to python * updated the python module to allow setting the logger on a `pulsar.Client` via the `ClientConfiguration` object
…a key (apache#6065) Fixes apache#6056 ### Motivation There is no doc telling users how to choose partition when you use a key. ### Modifications Add example on how to choose partition when using a key.
I noticed the documentation around batching didn't clarify some important details about how batching works in Pulsar, and particularly how it interacts with acknowledgements and redeliveries. This is my attempt to provide some clarity.
…Pulsar standalone (apache#5998)
…r configurable. (apache#6028) Fixes apache#5690 ### Motivation The default type name cannot be compatible with Elasticsearch versions both before 6.2 and after 7.0, so this change makes it configurable. ### Modifications Make type name configurable, and set the default value to "_doc" to make it compatible with Elasticsearch versions after 6.2 by default. ### Verifying this change Added unit tests and local integration tests.
* [CPP] Compile fixes for ARM cpus * Fixed formatting
Fixed: apache#6115 apache#6110 apache#6108 apache#6100 ### Motivation There are some errors in the content of the current document due to the adjustment of the architecture, so we will fix it. ### Modifications * The display is inconsistent with the downloaded content. * Document display error * Add io-use.md for version 2.5.0
### Motivation Available permits of ZeroQueueConsuemer must be 1 or less, however ZeroQueueConsuemer using listener may be greater than 1. ### Modifications If listener is processing message, ZeroQueueConsumer doesn't send permit when it reconnect to broker. ### Reproduction 1. ZeroQueueConsuemer using listener consume a topic. 2. Unload that topic( or restart a broker) when listener is processing message. 3. ZeroQueueConsumer sends permit when it reconnect to broker. https://github.com/apache/pulsar/blob/v2.5.0/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ZeroQueueConsumerImpl.java#L133 4. ZeroQueueConsumer also sends permit when finished processing message. https://github.com/apache/pulsar/blob/v2.5.0/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ZeroQueueConsumerImpl.java#L163 5. Available permits become 2.
* Fix typo in Reader documentation. Proker -> Broker. * update copy to be consistent with rest of docs
### Motivation Current help description of `bin/pulsar tokens` is redundancy when only specified one subcommand. ### Modifications Print corresponding usage according to the parsed command.
Fix some comment typos in the pulsar-broker module, no real code changes.
… used (apache#5883) Co-authored-by: Sijie Guo <guosijie@gmail.com>
) *Motivation* Fixes apache#5997 Fixes apache#6079 A regression was introduced in apache#5486. If websocket service as running as part of pulsar standalone, the cluster data is set with null service urls. This causes service url is not set correctly in the pulsar client and an illegal argument exception ("Param serviceUrl must not be blank.") will be thrown. *Modifications* 1. Pass `null` when constructing the websocket service. So the local cluster data can be refreshed when creating pulsar client. 2. Set the cluster data after both broker service and web service started and ports are allocated.
### Motivation *A change that had to be moved to 2.5.0 (because it [changed the .proto file](apache#5230 (comment))), is still listed as having been done in 2.4.2 in the release notes.* ### Modifications *Move the change from 2.4.2 to 2.5.0.*
### Motivation *Parts of the docs for Pulsar Functions are redundant.* ### Modifications *Remove parts of the docs for Pulsar Functions that are redundant.*
### Motivation After completing this pull request apache#5738, it seems that the document was not copied to the correct place to fix the problem. ### Modifications * Copy the document to the correct place
…onsumer are thread-safe in 5 language clients (apache#6214) Fix apache#6186
### Motivation There are a large number of 500 errors in current integration tests cli and function state. Fix this problem. ### Modifications * Split integration tests and delete useless images
Update url for Apache Heron
*Motivation* ASF Infra provides `.asf.yaml` to control features for a given github repo https://cwiki.apache.org/confluence/display/INFRA/.asf.yaml+features+for+git+repositories
*Motivation* A `pulsarbot` action was added to pulsar-test-infra for processing comments and trigger running github actions. This pull request is to enable this action to process comments.
Motivation Fix the license check errors. * Fix license header * Always run license check
…#6113)" (apache#6174) As stated in apache#6171, change apache#6113 broke the function instance runner. This change attempts to revert apache#6113 first to make sure CI back to normal. Additionally, it fixes and improves a bunch of integration tests. (unfortunately, we have to couple these changes to make sure it pass CI).
Fix: apache#5037 apache#6234 ### Motivation After all versions of API documents are added apache#5660, we should use the correct version in the document content. The auto-generated pulsar-admin tool should also point to the correct version. ### Modifications * Replace client API URL using the correct version * Add header to the navigation bar to point to the correct pulsar-admin page ### Verifying this change local test pass
### Motivation Supplied Kubernetes yaml's for AWS are outdated and just don't work. ### Modifications Update yaml files and so that appying them on AWS EKS will actually set up a working Pulsar environment. ### Verifying this change This change is a trivial rework / code cleanup without any test coverage.
### Motivation Fix get schema version in HttpLookupService. The com.yahoo.sketches.Util.bytesToLong method need to flip the byte[]. Otherwise, will get a wrong long value. So use ByteBuffer to convert byte[] version to long. This issue will happens when users use http protocol client and multiple version schemas. ### Verifying this change New tests added for HttpLookupService and BinaryLookupService.
* Fixed static linking on C++ lib on MacOS * Use `-undefined dynamic_lookup` when linking on Mac to not include python's own runtime * Fixed searching for protobuf
…d due to TTL (apache#6211) Fixes apache#5579 ### Motivation In Pulsar 2.4.1 and later versions, if message TTL is enabled, `PersistentMessageExpiryMonitor` always deletes one non-expired message every 5 minutes. The cause of this bug is apache#4744. `PersistentMessageExpiryMonitor` expects `ManagedCursor#asyncFindNewestMatching()` to pass null as its found position to itself as a callback if no expired messages exist. https://github.com/apache/pulsar/blob/c5ba52983fee994de61984aae7d1757e9b738caf/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentMessageExpiryMonitor.java#L119-L130 However, due to the change in apache#4744, if no entry is found that matches the search condition, the callback will be passed `startPosition` instead of null now. For this reason, the earliest backlog message is always deleted by `PersistentMessageExpiryMonitor`. This means that unexpected message loss can occur. ### Modifications Revert the apache#4744 changes. The motivation of apache#4744 is to avoid NPE caused in pulse-sql, but that seems to be fixed in apache#4757. https://github.com/apache/pulsar/blob/2069f761753940ed6a1faca8999af70036f20fd6/pulsar-sql/presto-pulsar/src/main/java/org/apache/pulsar/sql/presto/PulsarSplitManager.java#L363-L382
…ct is alive (apache#6226) * [C++] Ensure ExecutorServicePtr is not destroyed while the timer object is alive * Fixed formatting Co-authored-by: Sijie Guo <guosijie@gmail.com>
add code base code.
Thank you for allowing me to write Korean codebase.
{
enabled: true,
name: '한국어',
tag: 'ko-KR'},
…t/pulsar into go-features-integration
devinbost
pushed a commit
that referenced
this pull request
Mar 20, 2021
…artup (apache#9499) ### Motivation Standalone pulsar broker and function service can have deadlock which also sometime causes failures in function unit-test case. Pulsar function tries to create a subscription which blocks the zk thread and can be cause of possible deadlock. Below is the thread-dump when the deadlock happens and function service start up fails. So, remove blocking call while creating subscription using admin-api. ``` "pulsar-load-manager-139-1" apache#717 prio=5 os_prio=31 tid=0x00007f8a68698000 nid=0x3610f waiting on condition [0x00007000156e9000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000792a07020> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1709) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1788) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928) at org.apache.pulsar.zookeeper.ZooKeeperDataCache.get(ZooKeeperDataCache.java:97) at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.getDynamicConfigurationFromZK(SimpleLoadManagerImpl.java:391) at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.getDynamicConfigurationDouble(SimpleLoadManagerImpl.java:401) at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.getLoadBalancerBrokerOverloadedThresholdPercentage(SimpleLoadManagerImpl.java:450) at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.generateLoadReportForcefully(SimpleLoadManagerImpl.java:1139) - locked <0x000000078e53f2d0> (a java.util.HashSet) at org.apache.pulsar.broker.loadbalance.impl.SimpleLoadManagerImpl.writeLoadReportOnZookeeper(SimpleLoadManagerImpl.java:1295) at org.apache.pulsar.broker.loadbalance.LoadReportUpdaterTask.run(LoadReportUpdaterTask.java:39) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:748) "pulsar-ordered-OrderedExecutor-1-0-EventThread" apache#621 daemon prio=5 os_prio=31 tid=0x00007f89fd8e1800 nid=0x2fe07 waiting on condition [0x000070000f7ce000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000792ecb8b0> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalCreateSubscriptionForNonPartitionedTopic(PersistentTopicsBase.java:2060) at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.lambda$69(PersistentTopicsBase.java:2034) at org.apache.pulsar.broker.admin.impl.PersistentTopicsBase$$Lambda$676/1709042625.accept(Unknown Source) at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646) at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) at org.apache.pulsar.zookeeper.ZooKeeperCache.lambda$5(ZooKeeperCache.java:249) at org.apache.pulsar.zookeeper.ZooKeeperCache$$Lambda$154/5998675.processResult(Unknown Source) at org.apache.bookkeeper.zookeeper.ZooKeeperClient$15$1.processResult(ZooKeeperClient.java:879) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:583) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) main" #1 prio=5 os_prio=31 tid=0x00007f8aa0013000 nid=0x1603 waiting on condition [0x0000700004160000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000007930efe68> (a java.util.concurrent.CompletableFuture$Signaller) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908) at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.apply(AsyncHttpConnector.java:178) at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:297) at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$1(JerseyInvocation.java:632) at org.glassfish.jersey.client.JerseyInvocation$$Lambda$672/237755480.call(Unknown Source) at org.glassfish.jersey.client.JerseyInvocation.call(JerseyInvocation.java:654) at org.glassfish.jersey.client.JerseyInvocation.lambda$runInScope$3(JerseyInvocation.java:648) at org.glassfish.jersey.client.JerseyInvocation$$Lambda$673/197134325.call(Unknown Source) at org.glassfish.jersey.internal.Errors.process(Errors.java:292) at org.glassfish.jersey.internal.Errors.process(Errors.java:274) at org.glassfish.jersey.internal.Errors.process(Errors.java:205) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:390) at org.glassfish.jersey.client.JerseyInvocation.runInScope(JerseyInvocation.java:648) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:631) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:434) at org.glassfish.jersey.client.JerseyInvocation$Builder.put(JerseyInvocation.java:318) at org.apache.pulsar.client.admin.internal.TopicsImpl.createSubscription(TopicsImpl.java:1143) at org.apache.pulsar.functions.worker.PulsarWorkerService.start(PulsarWorkerService.java:454) at org.apache.pulsar.broker.PulsarService.startWorkerService(PulsarService.java:1343) at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:671) at org.apache.pulsar.io.PulsarFunctionE2ETest.setup(PulsarFunctionE2ETest.java:209) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ```
|
The pr had no activity for 30 days, mark with Stale label. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR depends upon the changes here: apache#6031
This PR is to provide integration tests that test execution of Go functions that are managed by the Java FunctionManager. This will allow us to test things like behavior during function timeouts, heartbeat failures, and other situations that can only be effectively tested in an integration test.
Master issue: apache#4175
This PR also depends on the code in apache#6031 .
Modifications
We must add Go to the integration testing logic. We must also build the Go dependencies into the test Dockerfile to ensure the Go binaries are available at runtime for the integration tests.
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Documentation is part of Go functions. We will need to enhance the documentation of Go functions as we get closer to being production-ready with them.