How to use phase4 with S3 storage

[mvolkert]

Hello,

As followup to https://stackoverflow.com/questions/71798060/how-to-use-holodeck-with-s3 I would like to ask how phase4 could be used with a self-hosted S3-Storage as filesystem.

[phax]

Thanks Marco, for asking the question here :)

There are two levels of data storage that you might consider:

• The "dumps", that are optionally created when sending and receiving messages and
• The messages on the business level, that are sent and/or received

What you anyone need to do, is to add S3 as a build dependency (e.g. to your pom.xml):

    <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>s3</artifactId>
      <version>2.17.192</version>
    </dependency>

Assuming you want to replace the "outgoing dumper" to write to S3, you could do something like this:

  private static final class S3OutgoingDumper extends AbstractAS4OutgoingDumperWithHeaders
  {
    private final ICommonsMap <String, File> m_aMessageIDToFilename = new CommonsConcurrentHashMap <> ();

    @Override
    public OutputStream openOutputStream (@Nonnull final EAS4MessageMode eMsgMode,
                                          @Nullable final IAS4IncomingMessageMetadata aMessageMetadata,
                                          @Nullable final IAS4MessageState aState,
                                          @Nonnull @Nonempty final String sMessageID,
                                          @Nullable final HttpHeaderMap aCustomHeaders,
                                          @Nonnegative final int nTry) throws IOException
    {
      final String sUniqueID = aMessageMetadata.getIncomingUniqueID ();
      final File aResponseFile = new File (SystemProperties.getTmpDir (),
                                           PDTIOHelper.getLocalDateTimeForFilename (aMessageMetadata.getIncomingDT ()
                                                                                                    .toLocalDateTime ()) +
                                                                          "-" +
                                                                          FilenameHelper.getAsSecureValidASCIIFilename (sUniqueID) +
                                                                          ".as4out");
      LOGGER.info ("Logging outgoing AS4 request to '" + aResponseFile.getAbsolutePath () + "'");
      m_aMessageIDToFilename.put (sMessageID, aResponseFile);
      return FileHelper.getBufferedOutputStream (aResponseFile);
    }

    @Override
    public void onEndRequest (@Nonnull final EAS4MessageMode eMsgMode,
                              @Nullable final IAS4IncomingMessageMetadata aMessageMetadata,
                              @Nullable final IAS4MessageState aState,
                              @Nonnull @Nonempty final String sMessageID)
    {
      LOGGER.info ("Now storing outgoing AS4 request for '" + sMessageID + "'");
      final File aFile = m_aMessageIDToFilename.remove (sMessageID);
      if (aFile != null) {
        if (aFile.exists ()) {
          LOGGER.info ("File existing: '" + aFile.getAbsolutePath () + "' with " + aFile.length () + " bytes");
          final String sTargetBucket = "YOUR_BUCKET_NAME";
          final String sTargetKey = "outbound/" +
                                    PDTIOHelper.getLocalDateTimeForFilename (aMessageMetadata.getIncomingDT ().toLocalDateTime ()) +
                                    "/" + aMessageMetadata.getIncomingUniqueID () +"/msg.as4out";

          s3Client.putObject (PutObjectRequest.builder ().bucket (sTargetBucket).key (sTargetKey)
                                                   .contentType (CMimeType.APPLICATION_OCTET_STREAM.getAsString ()).build (),
                                   RequestBody.fromFile (aFile));
          aFile.delete ();
          LOGGER.info ("Finished storing outgoing AS4 request for '" + sMessageID + "'");
        }
        else
          LOGGER.error ("File for '" + sMessageID + "' is " + aFile.getAbsolutePath () + " does not exist");
      }
      else
        LOGGER.error ("No file found for '" + sMessageID + "'");
    }
  }

It first dumps the message to a temporary file, and then uploads the file to a custom S3 bucket.

If you want to use S3 to store business messages only, than you can use the s3Client.putObject with byte array of the message itself. Therefore I would need to state your requirements more clearly.

hth, Philip

[mvolkert]

Thanks for your suggestion, I have to evaluate with my Colleges wether switching to pase4 would be feasible.

[ml-deb]

@phax What's the difference between saving it using the dumper or saving it in the IAS4ServletMessageProcessorSPI or IPhase4PeppolIncomingSBDHandlerSPI ?

[phax]

@ml-deb the dumper captures the native request as it is transmitted on the network level. So it contains all the HTTP headers, AS4 headers and the encrypted payload. Inside the IPhase4PeppolIncomingSBDHandlerSPI implementation you "only" get the decrypted main payload, in case it is a valid Peppol SBDH.

[ml-deb]

@phax So if I want to ensure I wont lose any messages, I need to save to s3 with the dumper or if I only need to process the content I can use the spi and if something fail it will send an error to the sender (client) ?

Also what's the difference between IAS4ServletMessageProcessorSPI and IPhase4PeppolIncomingSBDHandlerSPI ? If I'm using peppol I should use IPhase4PeppolIncomingSBDHandlerSPI instead of the orther ?

[phax]

With the dumper, you get ALL messages, so also the failed ones (no AS4, decrypt errors, signing errors, none-Peppol, no SBDH, invalid SBDH).
With SPI you only get the "successfully parsed" messages. It depends on your business requirements what you want to achieve :)

• If you want to archive everything you get: dump
• If you are only interested in the "useful" messages: SPI