ejbcaClientToolBox.sh for stress testing

[aleloco09]

Hello,
I want to use the ejbcaClientToolBox.sh EjbcaWsRACli stress tool to test my infrastructure creating a batch of certificates but i am having some problems.
The README file located in the script folder (/opt/keyfactor/ejbca/dist/clientToolBox) says that I need to edit ejbcawsracli.properties file to make this tool work.

# for client authentication I am using the SuperAdmin certificate
ejbcawsracli.keystore.path = ./SuperAdmin.p12 
# as truststore I am using my external truststore, which is imported at every start up of EJBCA 
ejbcawsracli.truststore.path= /mnt/external/secrets/tls/ts/truststore.jks

Then, inside the container as root, I execute a sample test like:

./ejbcaClientToolBox.sh EjbcaWsRACli stress PoliTOSubCA 20 1000 TLSClientProfile "TLS Client Profile"

The response is:

2024-04-07 09:18:46,452+0000 INFO  [org.apache.commons.beanutils.FluentPropertyBeanIntrospector] (main) Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property.
Enter keystore password: 

After entering the password the error comes:

java.lang.Exception: File './SuperAdmin.p12 ' does not exist
org.ejbca.ui.cli.ErrorAdminCommandException: java.lang.Exception: File './SuperAdmin.p12 ' does not exist
	at org.ejbca.core.protocol.ws.client.StressTestCommand.execute(StressTestCommand.java:582)
	at org.ejbca.core.protocol.ws.client.ejbcawsracli.main(ejbcawsracli.java:38)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.base/java.lang.reflect.Method.invoke(Unknown Source)
	at org.ejbca.ui.cli.EjbcaWsRaCli.execute(EjbcaWsRaCli.java:37)
	at org.ejbca.ui.cli.ClientToolBox.executeIfSelected(ClientToolBox.java:40)
	at org.ejbca.ui.cli.ClientToolBox.main(ClientToolBox.java:72)
Caused by: java.lang.Exception: File './SuperAdmin.p12 ' does not exist
	at org.ejbca.core.protocol.ws.client.EJBCAWSRABaseCommand.checkIfFileExists(EJBCAWSRABaseCommand.java:191)
	at org.ejbca.core.protocol.ws.client.EJBCAWSRABaseCommand.<init>(EJBCAWSRABaseCommand.java:133)
	at org.ejbca.core.protocol.ws.client.StressTestCommand.<init>(StressTestCommand.java:499)
	at org.ejbca.core.protocol.ws.client.EJBCAWSRACommandFactory.getCommand(EJBCAWSRACommandFactory.java:77)
	at org.ejbca.core.protocol.ws.client.ejbcawsracli.main(ejbcawsracli.java:35)
	... 7 more

I thought it was a permission error but i am executing the script as a root and the i have tried to change the ownership of the p12 file into 10001:0 first and then into 1002:0 (apparently the EjbcaClientToolBox.sh is owned by this user), but without success.
Do you have any hint on how I can figure this out?

[primetomas]

The message "File './SuperAdmin.p12 ' does not exist" means that it can't find the file. It's because "./" is probably not relative to the right place. Replace "./" with the full path for the p12 file and it should work.

[aleloco09]

Thanks for the answer, actually the problem was that there was an unwanted space character right after the path.
Now the tests run but I am not sure how they work exactly. Do you know if there is a documentation about the stress tests?
The doubts are:

1. what are the ? BASIC BASICSINGLETRANS BASICSINGLETRANS_SAMEUSER REVOKE REVOKE_BACKDATED REVOKEALOT
2. if i start a test of this type:

ejbcaClientToolBox.sh EjbcaWsRaCli stress PoliTOSubCA 2 1000 EMPTY ENDUSER BASIC 3

and more in general, if I do not specify the number of tests, the first n-threads tests will succeed, while the other will fail with the error:

ERROR [org.ejbca.core.protocol.ws.EjbcaWS] (default task-5) EJBCA WebService error: org.cesecore.certificates.certificate.CertificateCreateException: User 'WSTESTUSER-5872271850571089359' is not allowed to use same key as another user is using.

Is this expected?

[primetomas]

For the failure, it's expected due to your (default) CA policy setting. You need to uncheck check "Enforce unique public keys" for the multi-threaded test.

[primetomas]

The different types of tests to slightly different tings. For example you could run BASICSINGLETRANS_SAMEUSER, which should also get rid of the above error, which is due to multiple "usernames" making requests with a CSR containing the same public key.
The two common tests though are BASIC, which will make multiple calls, first an editUser followed by a pkcs10Request. BASICSINGLETRANS instead uses the single call certRequest which combines the above two into a single call (less roundtrips hence expected to be faster)(.

[aleloco09]

I have stress tested the RA Web Service successfully, but now I have problems with the OCSP stress...
From my VA, I tried the EJBCA ToolBox OCSP with a single test certificate:
ejbcaClientToolBox.sh OCSP http://ejbca-va-node:8080/ejbca/publicweb/status/ocsp WSTESTUSER3846118223201627364.pem PoliTOSubCA.pem
and the response was good.

Then I tried the OCSP stress first using the result.ser that came from the EjbcaWsRaCli stress (as suggested by the toolbox), then I created a .txt file containing only two certificate serial number (respecting the syntax 0x...). But all the tests fail.
Here's the command I run:
./ejbcaClientToolBox.sh OCSP stress http://ejbca-va-node:8080/publicweb/status/ocsp ./serials.txt ./PoliTOSubCA.pem 2 500

When I check the error.log all the errors give me the same message:

Sun May 05 17:37:54 UTC 2024 : Error querying OCSP server for 381618586268061878599488438538389105189530862106 . Error code is: 1
Sun May 05 17:37:54 UTC 2024 : Test in thread 1 completed but failed when the command 'org.ejbca.ui.cli.Ocsp.StressTest.Lookup' was executed. The time it took was 451 ms.

I don't get where I am wrong...

[primetomas]

Does it work with the result.ser file? I assume yes from the above?

In that case, it's likely something with the format of serials.txt. Actually error code 1 means unknown error, you should look in the server.log it should say something about this error.

[aleloco09]

no, if I pass result.ser i get that error as well

[aleloco09]

The OCSP stress does not work neither with the .ser file. If I cat on /opt/keyfactor/wildfly-26.1.3.Final/standalone/log/server.log the file is empty. I tried to see the docker logs while the stress testing was running but nothing appears. It looks like that it does not reach the server at all... I really dont understand what is going on...

[primetomas]

You didn't mention you were using the container. In container logs are not stored on disk but output to the container console, as best practice is that your container runtime handles logging best.
You can enable DEBUG logging of your container, see the docs. https://hub.docker.com/r/keyfactor/ejbca-ce

[aleloco09]

OCSP stress works just fine, there was an error in the oscp link.
Now, I am testing the tool PKCS11HSMKeyTool and I have a more serious problem that I have reported in issue #589.

What I was trying to do was testing the signing (and encrypting/decrypting) performance of the EJBCA crypto library (which supposedly is Bouncy Castle) and compare it with a direct interaction with the HSM (perhaps using opensc's pkcs11-tool). It may sound stupid but it is for performance assesment to check how the "filter"-EJBCA affects the performance of crypto operations.

Is there another way to test EJBCA keys from the CLI?

[primetomas]

PKCS11HSMKeyTool does not test anything through EJBCA. PKCS11HSMKeyTool also does not test signing using BouncyCastle, as it uses PKCS11, hence the HSM you have configured is used. So you will be testing PKCS#11 and the HSM...