The idea is to use CPP preprocessing instructions along with standard Linux CLI tools(bash,sed,grep) to generate FIX API for a given venue.
Mindset:
- No third party dependencies. All the generated code is yours and can go straight into your lib or app.
- Repeating groups are supported.
- You can strip off all useless standard FIX tags to fit your venue specs.
- FIX tags and enums are treated as integers and not strings thus removing transformations.
Receiving (read more):
- Lightweight and low latency FIX parsing with reusable memory and objects.
- Optional message sanity checks are available.
- Flexible formatting with available predefined TTY color styles.
- Pretty printing with GDB and VSCode.
Sending (read more):
- Low latency FIX message building with reusable memory and objects.
- For a given message type, only fields changing between two sends are to be updated.
- BodyLength and MsgSeqNum fields can be zero padded to reduce memory copies.
You have to know:
fixpp
is not a FIX engine (yet)- it performs better in an optimistic mode and relies on the venue FIX conformance (this saves quite a few CPU cycles...)
- a bit slower but safer parsing is available as well (it detects bad fields, groups and enums during message scan)
Tags and enums are not converted. For instance the sequence like "|49="
will be inserted as a single integer assignment with value 0x3d'39'34'01
. On reception, the string "49"
will not be translated to decimal value 49 but rather reinterpreted as 0x39'34
.
Results obtained in a tight loop on an i7-1360P @ 5GHz
action | message type | length | time, ns | CPU cycles | HW instructions |
---|---|---|---|---|---|
trusted decode | ExecReport | 170 | 74 | 370 | 912 |
safe decode | ExecReport | 170 | 104 | 518 | 1694 |
trusted decode | MarketDataSnapshotFullRefresh* | 330 | 116 | 580 | 1797 |
safe decode | MarketDataSnapshotFullRefresh* | 330 | 122 | 610 | 2708 |
encode | NewOrderSingle | 170 | 55 | 273 | 897 |
update | NewOrderSingle | 170 | 25 | 112 | 376 |
* 6 repeating groups
Based on your venue FIX API prepare files Fields.def
, Groups.def
and Messages.def
. One can pick up ready to use files from examples/spec and adjust Groups.def
and Messages.def
. Fields.def
will be cleaned automatically during generation.
If you have already a Quickfix XML file you can use the xml2cpp converter to generate the definitions.
src/spec/Fields.def
FIX_FIELD_BEGIN_STRING( FIX.4.4 )
FIX_FIELD_DECL( Account , 1, STRING )
FIX_FIELD_DECL( AvgPx , 6, PRICE )
FIX_FIELD_DECL( BeginSeqNo , 7, SEQNUM )
FIX_FIELD_DECL( BeginString , 8, STRING )
FIX_FIELD_DECL( BodyLength , 9, LENGTH )
FIX_FIELD_DECL( CheckSum , 10, STRING )
FIX_FIELD_DECL( ClOrdID , 11, STRING )
...
FIX_ENUM_BEGIN( MsgType )
FIX_ENUM_DECL( MsgType, HEARTBEAT , SOHSTR(0) )
FIX_ENUM_DECL( MsgType, TEST_REQUEST , SOHSTR(1) )
FIX_ENUM_DECL( MsgType, RESEND_REQUEST , SOHSTR(2) )
FIX_ENUM_DECL( MsgType, REJECT , SOHSTR(3) )
FIX_ENUM_DECL( MsgType, SEQUENCE_RESET , SOHSTR(4) )
FIX_ENUM_DECL( MsgType, LOGOUT , SOHSTR(5) )
FIX_ENUM_DECL( MsgType, EXECUTION_REPORT , SOHSTR(8) )
FIX_ENUM_DECL( MsgType, LOGON , SOHSTR(A) )
FIX_ENUM_DECL( MsgType, ORDER_SINGLE , SOHSTR(D) )
FIX_ENUM_DECL( MsgType, MARKET_DATA_REQUEST , SOHSTR(V) )
FIX_ENUM_DECL( MsgType, MARKET_DATA_SNAPSHOT_FULL_REFRESH, SOHSTR(W) )
FIX_ENUM_DECL( MsgType, MARKET_DATA_INCREMENTAL_REFRESH , SOHSTR(X) )
FIX_ENUM_DECL( MsgType, MARKET_DATA_REQUEST_REJECT , SOHSTR(Y) )
FIX_ENUM_END
...
src/spec/Groups.def
FIX_MSG_GROUP_BEGIN( MDEntries, MDEntryType )
FIX_MSG_FIELD( MDEntryPx )
FIX_MSG_FIELD( Currency )
FIX_MSG_FIELD( MDEntryPositionNo )
FIX_MSG_FIELD( MDEntrySize )
FIX_MSG_FIELD( MDEntryDate )
FIX_MSG_FIELD( MDEntryTime )
FIX_MSG_GROUP_END
...
src/spec/Messages.def
FIX_MSG_BEGIN( Header )
FIX_MSG_FIELD( BeginString )
FIX_MSG_FIELD( BodyLength )
FIX_MSG_FIELD( MsgType )
FIX_MSG_FIELD( SenderCompID )
FIX_MSG_FIELD( TargetCompID )
FIX_MSG_FIELD( MsgSeqNum )
FIX_MSG_FIELD( SendingTime )
FIX_MSG_END
FIX_MSG_BEGIN( Heartbeat )
FIX_MSG_FIELD( TestReqID )
FIX_MSG_END
FIX_MSG_BEGIN( Logon )
FIX_MSG_FIELD( EncryptMethod )
FIX_MSG_FIELD( HeartBtInt )
FIX_MSG_FIELD( ResetSeqNumFlag )
FIX_MSG_FIELD( Username )
FIX_MSG_FIELD( Password )
FIX_MSG_END
FIX_MSG_BEGIN( Logout )
FIX_MSG_FIELD( Text )
FIX_MSG_END
...
See more in examples.
/path/to/fixpp/generate.sh -d MYPRJ/src/myprj/fix -s MYPRJ/src/spec -i myprj/fix -n venue::fix -p MYPRJ/src/gdb
Where the options are:
-d
destination directory-s
specification directory with .def files-i
prefix to use in include statements (ex:#include <prefix/Field.h>
)-n
namespace to use-p
pretty printers destination dir--clean-fields
remove from Fields.def fields not referred to from Messages.def and Groups.def
It will create the following tree:
MYPRJ
├── Makefile
|
└── src
|
├── spec // your input
| ├── Fields.def
| ├── Groups.def
| └── Messages.def
│
├── gdb // generated files
│ └── printers.py
|
└── myprj
│
└── fix // generated files
├── Fields.cpp
├── Fields.h
├── FixApi.h
├── FixFloat.cpp
├── FixFloat.h
├── FixTypes.h
├── GroupSanity.cpp
├── GroupScanners.cpp
├── Groups.cpp
├── Groups.h
├── MessageBuilders.h
├── MessageSanity.cpp
├── MessageScanners.cpp
├── Messages.cpp
├── Messages.h
├── PrjInfo.h
└── SenderApi.h
A fully generated and committed primitive project is available in examples/order.
class BusinessLogic
{
void onMessage( const MessageExecutionReport & msg )
{
// for mandatory fields it is safe to get the value:
Quantity qty = msg.getOrderQty();
// for optional fields:
double px = msg.isSetPrice() ? msg.getPrice() : 0;
...
}
void onMessage( const MessageMarketDataSnapshotFullRefresh & msg )
{
...
}
};
In most cases a latency sensitive implementation will tend to reuse objects and avoid allocations on the critical path. Here we suggest keeping a Header
object and messages separately. The parsing is carried out in two steps:
- scanning the header first,
- then, after identifying the message type, scanning the relevant message body.
// my class attribute:
Header _header;
MessageExecutionReport _execReport;
MessageMarketDataSnapshotFullRefresh _mdsfr;
BusinessLogic _businessLogic;
...
// after reading the message into fixString
offset_t pos = _header.scan( fixString.data(), fixString.size() );
switch( _header_.getRawMsgType() )
{
case MsgTypeRaw_EXECUTION_REPORT:
_msgExecutionReport.scan( fixString.data() + pos, fixString.size() - pos );
_businessLogic.onMessage( _execReport );
break;
case MsgTypeRaw_MARKET_DATA_SNAPSHOT_FULL_REFRESH:
_mdsfr.scan( fixString.data() + pos, fixString.size() - pos );
_businessLogic.onMessage( _mdsfr );
break;
...
}
fixpp
generates a class ParserDispatcher
implementing the above switch. You have just to override onMessage(...)
methods.
#include <tiny/Messages.h>
#include <cstring>
const char * buffer = "8=FIX.4.4" I "9=156" I "35=8" I ... I "10=075" I;
using namespace venue::fix;
class MyProcessor: public ParserDispatcher
{
protected:
virtual void unprocessedMessage( raw_enum_t msgType, MessageBase & msg ) override
{
auto it = MsgTypeEnums::itemByRaw.find( msgType );
const char * typeName = it != MsgTypeEnums::itemByRaw.end() ? it->second->name : "unknown";
std::cout << "unprocessed " << typeName << std::endl;
}
virtual void onMessage( MessageMarketDataSnapshotFullRefresh & msg ) override
{
std::cout << "processed " << getCurrentHeader().getMsgType() << " " << msg.getMessageName() << std::endl;
}
};
int main( int args, const char ** argv )
{
size_t len = strlen( buffer );
const char * cursor = buffer;
MyProcessor mp;
while( cursor = mp.parseAndDipatch( cursor, len - ( cursor - buffer ) ) )
{
}
return 0;
}
constexpr unsigned begStrAndBodyLenBytes = MESSAGE_BEGIN_MIN_BYTES_TO_READ;
std::vector<char> recvbuffer( 4096 );
while( source.read( &recvbuffer[0], begStrAndBodyLenBytes ) )
{
unsigned msgTypeOffset;
len = parseMessageLength( &recvbuffer[0], msgTypeOffset ) + CHECKSUM_FIELD_LENGTH;
if( recvbuffer.size() < len + msgTypeOffset )
{
recvbuffer.insert( recvbuffer.end(), len + msgTypeOffset - recvbuffer.size() + 100 , 0 );
}
// read the remaining bytes
if( source.read( & recvbuffer[begStrAndBodyLenBytes], len - begStrAndBodyLenBytes + msgTypeOffset ) )
{
mp.parseAndDipatch( &recvbuffer[0], len + msgTypeOffset );
}
else
{
break;
}
}
#include <myprj/fix/Messages.h>
const char * execReport = "8=FIX.4.4" I "9=332" I "35=8" I "49=foo" I "56=bar" I "52=20071123-05:30:00.000" I
"11=OID123456" I "150=E" I "39=A" I "55=XYZ" I "167=CS" I "54=1" I "38=15" I "40=2" I "44=15.001" I "58=EQUITYTESTING" I "59=0" I "32=0" I "31=0" I "151=15" I "14=0" I "6=0" I
"555=2" I "600=SYM1" I "624=0" I "687=10" I "683=1" I
"688=A" I "689=a" I
"564=1" I
"539=2" I "524=PARTY1" I "525=S" I
"524=PARTY2" I "525=S" I
"804=2" I "545=S1" I "805=1" I "545=S2" I "805=2" I
"600=SYM2" I "624=1" I "687=20" I "683=2" I
"688=A" I "689=a" I
"688=B" I "689=b" I
"10=027" I;
using namespace venue::fix;
...
MessageHeader header;
// pos = offset from message start
offset_t pos = header.scan( execReport, strlen( execReport ) );
MessageExecutionReport er;
pos = er.scan( execReport + pos, strlen( execReport ) - pos );
// print single field value
std::cout << ' ' << FixOrdStatus << " = " << er.getOrdStatus() << std::endl;
// print entire message
std::cout << "\n\n -- Pretty Printing --" << std::endl;
// use operator <<
std::cout << fixstr( execReport, ttyRgbStyle ) << std::endl;
// print flat and advance pos
pos = 0;
fixToHuman( execReport, pos, std::cout, ttyRgbStyle ) << std::endl;
// indent groups
pos = 0;
fixToHuman( execReport, pos, std::cout, ttyRgbStyle, MessageExecutionReport::getFieldDepth ) << std::endl;
// iterate over groups
if( er.isSetNoLegs() )
{
unsigned noLegs = er.getNoLegs();
for( unsigned legIdx = 0 ; legIdx < noLegs; ++legIdx )
{
const GroupLegs & leg = er.getGroupLegs( legIdx );
std::cout << legIdx << ": side:" << leg.getLegSide() << " qty: " << leg.getLegQty() << std::endl;
}
}
You will have to include SenderApi.h to build FIX messages with fixpp
. It offers both
- low level buffer construction with
FixBufferStream
- and reusable memory approach with
ReusableMessageBuilder
This class has two attributes: _begin
and _end
. Respectively pointing to the message's first and past last byte.
Each time a new field is inserted _end
will shift forward accordingly. In most cases the fields will be appended as tag-value pairs:
execReport.append<ClOrdID>("OID4567");
execReport.append<QtyType>( QtyTypeEnums::UNITS );
execReport.append<Price>( 21123.04567, 2 );
It is also possible to push tags and values separately:
execReport.pushTag<ClOrdID>().pushValue("OID4567");
The idea behind is, for a given FIX session:
- to reuse the header since most of it's fields will not change,
- to pre-compute the checksum for non-changing header's fields,
- to update only changing time within timestamps since the date does not change intra day.
This class inherits the _begin
and _end
pointers from FixBufferStream
, but _begin
refers to the first changing field like SendingTime
for instance. The very first byte of the sending buffer will be pointed to by _start
. The latter will move each time the header's width changes. For example when the sequence number or body length change their widths.
messageBegin()
|
_buffer _start _msgType _sendingTime _body
| | | | |
"..." "8=FIX.4.4" I "9=315" I "35=W" I "49=foo" I "56=bar" I "34=1234" I "52=20190101-01:01:01.000" I "..."
--- ---- | |
| body length and seqno will be updated | _begin _end
| |
bodyBegin() end()
A typical scenario will be
using namespace fix;
using namespace fix::field;
using namespace fix::message;
...
/// before we send it
// prepare
ReusableMessageBuilder order( NewOrderSingle::getMessageType(), 512, 128 );
order.header().append<SenderCompID>("ASENDER");
order.header().append<TargetCompID>("ATARGET");
order.header().finalize();
// append SendingTime to the header
order.setupSendingTime( TimestampKeeper::Precision::MILLISECONDS );
const unsigned sendingTimeLength = order.end() - order.begin();
...
/// sending it without rebuilding the header
void sendOrder( const OrderFields & of )
{
// move end past SendingTime
order.rewind( sendingTimeLength );
// update changing fields in SendingTime
order.sendingTime().update();
// append order specific fields
order.append<Account>( of.account, of.accountLen );
order.append<ClOrdID>( of.orderId, of.orderIdLen );
order.append<Symbol>( of.symbol, of.symbolLen );
order.append<Side>( of.side );
order.append<Price>( of.price, 6 );
order.append<OrderQty>( of.qty );
// copy SendingTime into TransactTime
order.append<TransactTime>( order.sendingTime.begin, tsLen );
order.append<OrdType>( of.type );
// finalize
order.setSeqnumAndUpdateHeaderAndChecksum(++seqnum);
// send it
socket.send( order.start, order.end - order.start );
}
The ReusableMessageBuilder
API allows to append any value type along with any tag.
For example one could write:
order.append<Price>( "not-a-price-value" );
order.append<OrdType>( 12345 );
To prevent from such situation and to get your IDE completion working with message sending API, consider using message specific builders. The example above will appear as follows:
#include <MessageBuilders.h>
...
using namespace fix;
using namespace fix::field;
using namespace fix::message;
...
/// before we send it
// prepare
ReusableMessageBuilder order( NewOrderSingle::getMessageType(), 512, 128 );
auto & builder = NewOrderSingleBuilder::Ref( order );
builder.getHeader().appendSenderCompID("ASENDER");
builder.getHeader().appendTargetCompID("ATARGET");
builder.getHeader().finalize();
// append SendingTime to the header
builder.setupSendingTime( TimestampKeeper::Precision::MILLISECONDS );
const unsigned sendingTimeLength = order.end - order.begin;
...
/// sending it without rebuilding the header
void sendOrder( const OrderFields & of )
{
// move end past SendingTime
order.rewind( sendingTimeLength );
// update changing fields in SendingTime
order.sendingTime().update();
// append order specific fields
builder.appendAccount( of.account );
builder.appendClOrdID( of.orderId );
builder.appendSymbol( of.symbol );
// Only proper enum values can be used
builder.appendSide( of.isBid() ? SideEnums::BUY : SideEnums::OFFER );
builder.appendPrice( of.price, 6 );
builder.appendOrderQty( of.qty );
builder.appendTransactTime( order.sendingTime.begin, order.sendingTime.length() );
builder.appendOrdType( OrdType::LIMIT );
builder.finalizeWithSeqnum(++seqnum);
// send it
socket.send( order.start, order.end - order.start );
}
For messages with repeating groups, it will look like:
ReusableMessageBuilder * mdfr = myFactory->createMessage( MessageMarketDataSnapshotFullRefresh::getMessageType() );
auto & builder = MarketDataSnapshotFullRefreshBuilder::Ref( *mdfr );
builder.appendMDReqID( "reqid" );
// groups
auto & entries = builder.appendNoMDEntries( 2 );
// groups 1
entries.appendMDEntryType( MDEntryTypeEnums::BID );
entries.appendMDEntryPositionNo( 1 );
entries.appendMDEntryPx( 1.123, 6 );
entries.appendMDEntrySize( 100 );
// group 2
entries.appendMDEntryType( MDEntryTypeEnums::OFFER );
entries.appendMDEntryPositionNo( 1 );
entries.appendMDEntryPx( 1.234, 6 );
entries.appendMDEntrySize( 200 );
builder.finalizeWithSeqnum(1);
To compile the examples you wil have to clone the makefile project next to fixpp
:
$> git clone https://github.com/sashamakarenko/makefile.git makefile
$> git clone https://github.com/sashamakarenko/fixpp.git fixpp
$> cd fixpp/examples/tiny
$> make
$> make check
- fix44 all committed complete lib with all FIX4.4 messages
- fixdump tool to decode FIX messages
- odd lib with unit tests with irregular messages
- tiny venue specific example with unit tests
- order all committed primitive project only with NewOrderSingle and ExecutionReport
- spec input files for different FIX standards
const FixFormatStyle htmlRgbStyle =
{
.messageBegin = "<pre>",
.messageEnd = "</pre>",
.indent = " ",
.groupFirstField = " •",
.fieldBegin = " ",
.fieldEnd = "\n",
.headerTagNameStart = "<font color=\"#444444\">",
.headerTagNameStop = "</font>",
.tagNameStart = "<font color=\"black\"><b>",
.tagNameStop = "</b></font>",
.tagValueStart = "<font color=\"grey\">(",
.tagValueStop = ")</font>",
.equal = " = ",
.valueStart = "<font color=\"darkblue\">",
.valueStop = "</font>",
.enumStart = " <font color=\"darkgreen\">",
.enumStop = "</font>",
.unknownStart = "<font color=\"red\">",
.unknownStop = "</font>"
};
...
std::ofstream html;
html.open( "mdfr.html" );
pos = 0;
fixToHuman( mdFullRefresh, pos, html, htmlRgbStyle, autoIndentFields );
html.close();
BeginString(8) = FIX.4.4 BodyLength(9) = 315 MsgType(35) = W MARKET_DATA_SNAPSHOT_FULL_REFRESH SenderCompID(49) = foo TargetCompID(56) = bar MsgSeqNum(34) = 1234 SendingTime(52) = 20190101-01:01:01.000 Symbol(55) = EUR/USD NoMDEntries(268) = 6 MDEntryType(269) = 1 OFFER MDEntryPositionNo(290) = 1 MDEntryPx(270) = 1.21 Currency(15) = USD MDEntrySize(271) = 1000000 MDEntryType(269) = 1 OFFER MDEntryPositionNo(290) = 2 MDEntryPx(270) = 1.211 Currency(15) = USD MDEntrySize(271) = 2000000 MDEntryType(269) = 1 OFFER MDEntryPositionNo(290) = 3 MDEntryPx(270) = 1.221 Currency(15) = USD MDEntrySize(271) = 3000000 MDEntryType(269) = 1 OFFER MDEntryPositionNo(290) = 4 MDEntryPx(270) = 1.2315 Currency(15) = USD MDEntrySize(271) = 4000000 MDEntryType(269) = 0 BID MDEntryPositionNo(290) = 5 MDEntryPx(270) = 1.201 Currency(15) = USD MDEntrySize(271) = 1000000 MDEntryType(269) = 0 BID MDEntryPositionNo(290) = 6 MDEntryPx(270) = 1.205 Currency(15) = USD MDEntrySize(271) = 2000000 CheckSum(10) = 075
Show message as HTML table:
const FixFormatStyle htmlTableRgbStyle =
{
.messageBegin = "<pre><table>",
.messageEnd = "</table></pre>",
.indent = " ",
.groupFirstField = " •",
.fieldBegin = "<tr><td>",
.fieldEnd = "</td></tr>\n",
.headerTagNameStart = "<font color=\"#444444\">",
.headerTagNameStop = "</font>",
.tagNameStart = "<font color=\"black\"><b>",
.tagNameStop = "</b></font>",
.tagValueStart = "<font color=\"grey\">(",
.tagValueStop = ")</font>",
.equal = " </td><td> ",
.valueStart = "<font color=\"darkblue\">",
.valueStop = "</font>",
.enumStart = " <font color=\"darkgreen\">" ,
.enumStop = "</font>",
.unknownStart = "<font color=\"red\">",
.unknownStop = "</font>"
};
BeginString(8) FIX.4.4 BodyLength(9) 332 MsgType(35) 8 EXECUTION_REPORT SenderCompID(49) foo TargetCompID(56) bar SendingTime(52) 20071123-05:30:00.000 ClOrdID(11) OID123456 ExecType(150) E OrdStatus(39) A PENDING_NEW Symbol(55) XYZ SecurityType(167) CS COMMON_STOCK Side(54) 1 BUY OrderQty(38) 15 OrdType(40) 2 LIMIT Price(44) 15.001 Text(58) EQUITYTESTING TimeInForce(59) 0 DAY LastQty(32) 0 LastPx(31) 0 LeavesQty(151) 15 CumQty(14) 0 AvgPx(6) 0 NoLegs(555) 2 •LegSymbol(600) SYM1 LegSide(624) 0 LegQty(687) 10 NoLegStipulations(683) 1 •LegStipulationType(688) A LegStipulationValue(689) a LegPositionEffect(564) 1 NoNestedPartyIDs(539) 2 •NestedPartyID(524) PARTY1 NestedPartyIDSource(525) S •NestedPartyID(524) PARTY2 NestedPartyIDSource(525) S NoNestedPartySubIDs(804) 2 •NestedPartySubID(545) S1 NestedPartySubIDType(805) 1 •NestedPartySubID(545) S2 NestedPartySubIDType(805) 2 •LegSymbol(600) SYM2 LegSide(624) 1 LegQty(687) 20 NoLegStipulations(683) 2 •LegStipulationType(688) A LegStipulationValue(689) a •LegStipulationType(688) B LegStipulationValue(689) b CheckSum(10) 027