Messages specifications for Protobuf, MsgPack, ODVD, ...

[piperoc]

This project looks like a great idea.

I was trying to test, but I'm confused a bit.

It took me a minute but I eventually understood that I cannot define my protobuf messages using the actual google proto language. I must use the format that the cluon-msc converter understands.

However, that seems pretty limited to scalar values, no arrays or "repeated" structures.

An example in proto language (very simple):

syntax = "proto2";

message SimpleMessage {
  required int32 msg_id = 1;
  optional string msg_string = 2;
}

message ComplexMessage {
  required int32 complex_msg_id = 1;
  repeated SimpleMessage messages = 2;
}

does not have a corresponding sintax in "odvd":

message SimpleMessage [id = 30000] {
    uint32 msg_id [ id = 1 ];
    string msg_payload [ default = "Hello World!", id = 2 ];
}

message ComplexMessage [id = 30001] {
    uint32 complex_msg_id [ id = 1 ];
    ?
}

If I want something more complex like that, am I expected to extend the classes with definitions of things like you have in your cluon::data namespace? I'm asking because I did not see much info on that other than looking at the code.

Thanks in advance.

[piperoc]

Sorry, I just saw the issue 22 . I looked at it before but did not understand it.

Are you suggesting that I wrap complex data into a string:

message SimpleMessage [id = 30000] {
    uint32 msg_id [ id = 1 ];
    string msg_payload [ default = "Hello World!", id = 2 ];
}

message ComplexMessage [id = 30001] {
    uint32 complex_msg_id [ id = 1 ];
    string ListOfSimpleMessges [ id =2 ];
}

but that means that I will have to encode/decode that string property on my own -- which sounds like creating my own serializer.

I am not questioning, just asking.

I like the library in any case because it's so portable, but I need a more complete serialization.

As a test, I actually combined libcluon with a simple yet powerful serialization library called cereal .
Doing that I can send complex objects across and libcluon seems ok with that.

Just to share the (very crude and quick) test code for a sender and receiver over TCP (to be compiled as separate programs):

receiver.cpp

#include <iostream>
#include <chrono>
#include <sstream>
#include <vector>
#include <memory>

#define CEREAL_THREAD_SAFE 1
#include "cereal/types/vector.hpp"
#include "cereal/types/memory.hpp"
#include "cereal/archives/binary.hpp"

#include "cluon-complete.h"

bool m_running = true;

struct MyRecord
{
  uint16_t x, y;
  double z;

  template <class Archive>
  void serialize(Archive &ar)
  {
  	ar(x, y, z);
  }
};

struct SomeData
{
  uint32_t id;
  std::shared_ptr<std::vector<MyRecord>> data;

  template <class Archive>
  void save(Archive &ar) const
  {
  	ar(id, data);
  }

  template <class Archive>
  void load(Archive &ar)
  {
  	ar(id, data);
  }
};

// I need a signal handler to catch SIGINT and SIGTERM.
void signalHandler(int signal)
{
  std::cout << "Caught signal " << signal << std::endl;
  
  if (signal == SIGINT || signal == SIGTERM) m_running = false;
}

void OnNewData(std::string &&d, std::chrono::system_clock::time_point && /*timestamp*/)
{
  try
  {
  	//deserialize the data
  	
  	std::stringstream is(d, std::ios::binary | std::ios::in | std::ios::out);
  	cereal::BinaryInputArchive iarchive(is);
  	SomeData data;
  	iarchive(data);

  	std::cout << "Received data: " << data.id << std::endl;
  	for (auto &r : *data.data)
  	{
  		std::cout << "  " << (uint16_t)r.x << ", " << (uint16_t)r.y << ", " << r.z << std::endl;
  	}
  }
  catch (const std::exception &e)
  {
  	std::cerr << "Error: " << e.what() << std::endl;
  }
}

int main(int argc, char *argv[])
{
  // We collect all connections in this std::vector.
  std::vector<std::shared_ptr<cluon::TCPConnection>> connections;

  // First, we define a lambda function to handle incoming TCP connections.
  auto newConnectionHandler = [&connections](std::string && from, std::shared_ptr<cluon::TCPConnection> connection) noexcept
  {
  	std::cout << "Got connection from " << from << std::endl;
  	// This lambda is handling any incoming data on this TCP connection.
  	connection->setOnNewData(OnNewData);
  	// This lambda is handling a connection loss (ie., the client is closing the connection).
  	connection->setOnConnectionLost([]() { std::cout << "Connection closed." << std::endl; });
  	// Store the connection to keep it open.
  	connections.push_back(connection);
  };

  // Next, we create a TCP server for localhost listening on port 1234, which is using the lambda from before.
  cluon::TCPServer server(1234, newConnectionHandler);
  
  while (m_running)
  {
  	// sleep for 100ms
  	std::this_thread::sleep_for(std::chrono::milliseconds(100));
  }
  
  // Close all connections.
  connections.clear();
  
  return 0;
}

sender.cpp

#include <iostream>
#include <chrono>
#include <sstream>

#define CEREAL_THREAD_SAFE 1
#include "cereal/types/vector.hpp"
#include "cereal/types/memory.hpp"
#include "cereal/archives/binary.hpp"

#include "cluon-complete.h"

struct MyRecord
{
  uint16_t x, y;
  double z;

  template <class Archive>
  void serialize(Archive &ar)
  {
  	ar(x, y, z);
  }
};

struct SomeData
{
  uint32_t id;
  std::shared_ptr<std::vector<MyRecord>> data;

  template <class Archive>
  void save(Archive &ar) const
  {
  	ar(id, data);
  }

  template <class Archive>
  void load(Archive &ar)
  {		
  	ar(id, data);
  }
};

int main(int argc, char *argv[])
{
  cluon::TCPConnection c("127.0.0.1", 1234);
  
  
  
  
  
  if (c.isRunning())
  {
  	uint16_t count = 0;
  	
  	// loop until CTRL+C
  	while (c.isRunning())
  	{
  		// create a simple message
  		SomeData s_msg;
  		
  		// fill the message
  		s_msg.id = count++;
  		s_msg.data = std::make_shared<std::vector<MyRecord>>();
  		s_msg.data->push_back({count + 1, count + 2, count + 3.2});
  		s_msg.data->push_back({count + 4, count + 5, count + 6.7});
  		s_msg.data->push_back({count + 8, count + 9, count + 10.11});
  		
  		std::ostringstream ss(std::ios::binary);
  		
  		cereal::BinaryOutputArchive oarchive(ss);
  		oarchive(s_msg);
  		
  		// send the message
  		c.send(ss.str());
  		
  		// wait 1 second
  		std::this_thread::sleep_for(std::chrono::seconds(1));
  	}
  	

  	return 0;
  }
  else
  {
  	std::cout << "Connection failed" << std::endl;
  	return 1;
  }
  
  
}

However, something like protobuf would be more powerful and standard. I will welcome any advise.
Thanks again.