forked from pmacct/pmacct
-
Notifications
You must be signed in to change notification settings - Fork 0
/
ChangeLog
5008 lines (4887 loc) · 298 KB
/
ChangeLog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
pmacct [IP traffic accounting : BGP : BMP : RPKI : IGP : Streaming Telemetry]
pmacct is Copyright (C) 2003-2022 by Paolo Lucente
The keys used are:
!: fixed/modified feature, -: deleted feature, +: new feature
1.7.8 -- 31-12-2022
+ Introduced support for eBPF for all daemons: if SO_REUSEPORT is
supported by the OS and eBPF support is compiled in, this allows
to load a custom load-balancer. To load-share, daemons have to
be part of the same cluster_name and each be configured with a
distinct cluster_id.
+ Introduced support for listening on VRF interfaces on Linux for
all daemons. The feature can be enabled via nfacctd_interface,
bgp_daemon_interface and equivalent knobs. Many thanks to
Marcel Menzel ( @WRMSRwasTaken ) for this contribution.
+ pre_tag_map: introduced limited tagging / labelling support for
BGP (pmbgpd), BMP (pmbmpd), Streaming Telemetry (pmtelemetryd)
daemons. ip, set_tag, set_label keys being currently supported.
+ pre_tag_map: defined a new pre_tag_label_encode_as_map config
knob to encode the output 'label' value as a map for JSON and
Apache Avro encodings, ie. in JSON "label": { "key1": "value1",
"key2": "value2" }. For keys and values to be correctly mapped,
the '%' delimiter is used when composing a pre_tag_map, ie.
"set_label=key1%value1,key2%value2 ip=0.0.0.0/0". Thanks to
Salvatore Cuzzilla ( @scuzzilla ) for this contribution.
+ pre_tag_map: introduced support for IP prefixes for src_net
and dst_net keys for indexed maps (maps_index set to true).
Indexing being an hash map, this feature currently tests data
against all defined IP prefix lenghts in the map for a match
(first defined matching prefix wins).
+ pre_tag_map: introduced two new 'is_nsel', 'is_nel' keys to
check for the presence of firewallEvent field (233) and
natEvent field (230) in NetFlow/IPFIX respectively in order
to infer whether data is NSEL / NEL. If set to 'true' this
does match NSEL / NEL data, if set to 'false' it does match
non NSEL / NEL data respectively.
+ Introduced a new mpls_label_stack primitive, encoded as a
string and includes a comma-separated list of integers (label
values). Thanks to Salvatore Cuzzilla ( @scuzzilla ) for this
contribution.
+ Introduced a new fw_event primitive, to support NetFlow v9/
IPFIX firewallEvent 233 Information Element.
+ Introduced a new tunnel_tcp_flags primitive for pmacctd and
sfacctd to record TCP flags for the inner layer of a tunneled
technology (ie. VXLAN). Also tunnel_dst_port decoding was
fixed for sfacctd.
+ Introduced support for in/out VLAN support for sfacctd. To be
savy, 'in_vlan' and 'vlan' were muxed onto the same primitive
depending on the daemon being used. Thanks to Jim Westfall
( @jwestfall69 ) for this contribution.
+ Introduced a new mpls_label_stack_encode_as_array config knob
to encode the MPLS label stack as an array for JSON and Apache
Avro encodings, ie. in JSON "mpls_label_stack": [ "0-label0",
"1-label1", "2-label2", "3-label3", "4-label4", "5-label5" ]
and in Avro "name": "mpls_label_stack", "type": { "type":
"array", "items": { "type": "string" } }. Thanks to Salvatore
Cuzzilla ( @scuzzilla ) for this contribution.
+ Introduced a new tcpflags_encode_as_array config knob to encode
TCP flags as an array for JSON and Apache Avro, ie. in JSON
"tcp_flags": [ "URG", "ACK", "PSH", "RST", "SYN", "FIN" ] and
in Avro "name": "tcp_flags", "type": { "type": "array",
"items": { "type": "string" } }. Thanks to Salvatore Cuzzilla
( @scuzzilla ) for this contribution.
+ Introduced a new fwd_status_encode_as_string config knob to
encode the 'fwd_status' primitive in human-readable format
like described by RFC-7270 Section 4.12 when JSON or Avro
formats are selected for output. Thanks to Salvatore Cuzzilla
( @scuzzilla ) for this contribution.
+ Introduced a new protos_file to define a list of (known/
interesting/meaningful) IP protocols. Both protocol names, ie.
"tcp", and protocol numbers, ie. 1 (for icmp), are accepted.
IANA reserved protocol value 255 is used to bucket as 'others'
those IP protocols not matching the ones defined in the list.
+ Introduced a new tos_file to define a list of (meaningful) IP
ToS values; if tos_encode_as_dscp is set to true then DSCP
values are expected as part of the file. The directive uses
value 255 to bucket as 'others' those ToS/DSCP values not
matching the ones defined in the list.
+ A new tos_encode_as_dscp config knob makes pmacct to honour
only the 6 bits used by DSCP and report only on those.
+ BGP, BMP, Streaming Telemetry daemons: introduced a new
dump_time_slots config knob to spread the load deriving by
dumps over the configured refresh time interval. The interval
is divided into time slots and nodes are assigned to such
slots. The slot for each node is determined using its IP
address. Thanks to Raphael Barazzutti ( @rbarazzutti ) for
this contribution.
+ BGP, BMP daemons: End-of-RIB messages are now being exposed
in the output feed in order to facilitate tracking their
arrival (or not!).
+ pmtelemetryd: aligned daemon to the latest Unyte UDP-Notif API
(0.6.1) and related standardization draft-ietf-netconf-udp-notif
+ RPKI daemon: added case for input "asn" value being integer (ie.
"asn" : 2914) on top of the string case (ie. "asn" : "AS2914").
+ Kafka, amqp plugins: introduced a new writer_id_string config
knob to allow to customize the the "writer_id" field value. A
few variables are supported along with static text definitions.
+ Added a new aggregate_unknown_etype config knob to account also
frames with EtherTypes for which there is no decoding support
and allow to aggregate them by the available Ethernet L2 fields
(ie. 'src_mac', 'dst_mac', 'vlan', 'cos', 'etype'). Thanks to
@singularsyntax for this contribution.
+ Added a new bgp_daemon_add_path_ignore config knob to ignore
(do not advertise back) the ADD-PATH capability advertised by
remote BGP peers.
+ nfacctd, sfacctd: extended the possibility to run daemons from
a user with non root privileges to these daemons.
+ nfacctd: if Information Element 90 (MPLS VPN RD) is present in
NetFlow v9/IPFIX, make it available for BGP/BMP correlation.
+ pmacctd, sfacctd: introduced basic support for QinQ, 802.1AD.
+ [print|kafka|amqp]_preprocess: added suppport for 'maxp',
'maxb' and 'maxf' keys when preprocessing aggregates of non-
SQL plugins. Thanks to Andrew R. Lake ( @arlake228 ) for this
contribution.
+ nDPI: newer versions of the library (ie. >= 4.0) bring changes
to the API. pmacct is now aligned to compile against these. At
the same time support for nDPI 3.x was dropped.
! fix, plugin_common.[ch]: when stitching feature was enabled,
ie. nfacctd_stitching, timestamp_min was never reset. Also both
timestamp_min and timestamp_max were clamped to sec granularity.
! fix, BGP, BMP daemons: added a tmp_bgp_daemon_origin_type_int to
print out BGP "origin" field as int (legacy behaviour) instead
of string (current behaviour). In a future major release the
legacy behaviour will be dropped.
! fix, BGP, BMP daemons: MPLS labels are now encoded in both JSON
and Apache Avro as 'mpls_label' instead of 'label'. This is to
align behaviour with pre_tag_map where 'label' has a different
semantic.
! fix, BGP, BMP daemons: resolved memory leak when encoding log
messaging (logmsg) in Avro format with Schema Registry support.
! fix, BGP daemon: improved handling of ADD-PATH capability,
making it per-AF (as it is supposed to be) and not global.
! fix, BMP daemon: now checking that ADD-PATH capability is
enabled at both ends of the monitored session (check both BGP
OPEN in a Peer Up message) in order to infer that the capability
exchange was successful. Also some heuristics were added to
conciliate BGP Open vs BGP Update 4-bytes ASN reality.
! fix, nfacctd: improved parsing of NetFlow v9 Options data
particularly when multiple IEs are packed as part of a flowset.
! fix, nfacctd: corrected parsing of Information Element 351
(layer2SegmentId).
! fix, pmacctd: improved processing of pcap_interfaces_map for
cases where the same interface is present multiple times (maybe
with different directions). Also, if the map is empty then bail
out at startup.
! fix, pmacctd: SEGV when ICMP/ICMPv6 traffic was processed and
'flows' primitive was enabled.
! fix, pmacctd: sampling_rate primitive value was not reported
correctly when 'sampling_rate' config directive was specified.
! fix, pmbgpd, pmpmbd, pmtelemtryd: changed SIGCHLD handler to
prevent zombification of last spawned data dump writer.
! fix, Kafka plugin: moved the schema registration from the dump
writer to the plugin process in order to register the schemas
only once at plugin startup and not on every start of a writer
process. Thanks to Uwe Storbeck ( @ustorbeck ) for this
contribution.
! fix, Kafka plugin: a check for kafka_partition was missing,
leading the plugin to always use the default partitioner
instead of sending data to the configured fixed partition.
Thanks to Martin Pels ( @rodecker ) for this contribution.
! fix, nfprobe plugin: BGP data enrichment was not working due to
a mistakenly moved pointer.
! fix, sfprobe plugin: AS-PATH was being populated even when null;
added a check to see if the destination AS is not zero in order
to put the destination AS into the AS-PATH for sFlow packets.
Thanks to Marcel Menzel ( @WRMSRwasTaken ) for this contribution.
! fix, networks_file: remove_dupes() was making partial commits
of valid rows hence creating data inconsistencies.
! fix, pre_tag_map: resolved a potential string overflow that was
being triggered in pretag_append_label() when data would be
assigned more than one single label. Also now allow ',' chars
in set_label.
! fix, maps_index: uninitialized var could cause SEGV in case no
results are found in the map index. Also introduced support for
catch-all rules, ie. "set_label=unknown".
! fix, maps_index: optimized the case of no 'ip' key specified
(for nfacctd and sfacctd): when indexing is enabled, prevent
recirculation from happening, ie. test v4 first then v6, since
the 'ip' key is not going to be part of the hash serializer.
! fix, pretag.c: allow to allocate maps greater than 2GB in size.
Also several optimizations were carried out yelding to a better
memory utilization for allocated maps along with improved times
to resolve JEQs.
! fix, pre_tag_label_filter: optimized and improved runtime
evaluation part of this feature, avoiding a costly strdup() and
returning immediately on certain basic mismatch conditions.
! fix, kafka_common.[ch]: a new p_kafka_produce_data_and_free()
is invoked to optimize memory allocations and releases.
! fix, plugin_cmn_avro.c: when a schema registry is being defined,
ie. kafka_avro_schema_registry, the logic to generate the schema
name has been changed: use topic plus record name as the schema
name, use underscore as separator within the record name, stop
adding a "-value" suffix. Thanks to Uwe Storbeck ( @ustorbeck )
for this contribution.
! fix, util.c: roundoff_time() to reason always with the locally
configured time, like for the rest of functional (as in non-data)
timestamps, ie. refresh time, deadline, etc.
! fix, log.c: when log messages are longer than message buffer,
the message gets cut off. As the trailing newline also gets cut
off the message will be concatenated with the following message
which makes the log hard to read. Thanks to Uwe Storbeck
( @ustorbeck ) for this contribution.
- Completed the retirement of legacy packet classification based
on home-grown code (Shared Objects) and the L7 layer project.
- Removed the mpls_stck_depth primitive due to the introduction
of the mpls_label_stack primitive.
1.7.7 -- 07-11-2021
+ BGP, BMP, Streaming Telemetry daemons: introduced parallelization
of dump events via a configurable amount of workers where the unit
of parallelization is the exporter (BGP, BMP, telemetry exporter),
ie. in a scenario where there are 4 workers and 4 exporters each
worker is assigned one exporter data to dump.
+ pmtelemetryd: added support for draft-ietf-netconf-udp-notif:
a UDP-based notification mechanism to collect data from networking
devices. A shim header is proposed to facilitate the data streaming
directly from the publishing process on network processor of line
cards to receivers. The objective is a lightweight approach to
enable higher frequency and less performance impact on publisher
and receiver process compared to already established notification
mechanisms. Many thanks to Alex Huang Feng ( @ahuangfeng ) and the
whole Unyte team.
+ BGP, BMP, Streaming Telemetry daemons: now correctly honouring the
supplied Kafka partition key for BGP, BMP and Telemetry msg logs
and dump events.
+ BGP, BMP daemons: a new "rd_origin" field is added to output log/
dump to specify the source of Route Distinguisher information (ie.
flow vs BGP vs BMP).
+ pre_tag_map: added ability to tag new NetFlow/IPFIX and sFlow
sample_type types: "flow-ipv4", "flow-ipv6", "flow-mpls-ipv4" and
"flow-mpls-ipv6". Also added a new "is_bi_flow" true/false key to
tag (or exclude) NSEL bidirectional flows. Added as well a new
"is_multicast" true/false config key to tag (or exclude) IPv4/IPv6
multicast destinations.
+ maps_index: enables indexing of maps to increase lookup speeds on
large maps and/or sustained lookup rates. The feature has been
remplemented using stream-lined structures from libcdada. This is
a major work that helps preventing the unpredictable behaviours
caused by the homegrown map indexing mechanism. Many thanks to
Marc Sune ( @msune ).
+ maps_index: support for indexing src_net and dst_net keywords has
been added.
+ Added <daemon_name>_ipv6_only config directives to optionally
enable the IPV6_V6ONLY socket option. Also changed the wrong
setsockopt() IPV6_BINDV6ONLY id to IPV6_V6ONLY.
+ Added log function to libserdes to debug transactions with the
Schema Registry when kafka_avro_schema_registry is set.
+ nDPI: newer versions of the library (ie. >= 3.5) bring changes
to the API. pmacct is now aligned to compile against these.
+ pmacctd: added pcap_arista_trailer_offset config directive since
Arista has changed the structure of the trailer format in recent
releases of EOS. Thanks to Jeremiah Millay ( @floatingstatic )
for his patch.
+ More improvements carried out on the Continuous Integration
(CI) side by migrating from Travis CI to GitHub Actions. Huge
thanks to Marc Sune ( @msune ) to make all of this possible.
+ More improvements also carried out in the space of the Docker
images being created: optimized image size and a better layered
pipeline. Thanks to Marc Sune ( @msune ) and Daniel Caballero
( @dcaba ) to make all of this possible.
+ libcdada shipped with pmacct was upgraded to version 0.3.5. Many
thanks Marc Sune ( @msune ) for his work with libcdada.
! build system: several improvements carried out in this area,
ie. improved MySQL checks, introduced pcap-config tool for
libpcap, compiling on BSD/old compilers, etc. Monumental thanks
to Marc Sune ( @msune ) for his continued help.
! fix, nfacctd: improved euristics to support the case of flows
with both IPv4 and IPv6 source / destination addresses (either
or populated). Also improved euristics to distinguish event data
vs traffic data in NetFlow v9/IPFIX from Cisco 9300/9500, ASA
firewalls and Cisco 4500X.
! fix, nfacctd: improved support for initiatorOctets (IE #231) and
responderOctets (IE #232). Thanks to Esben Laursen ( @hyberdk )
for reporting the issue.
! fix, nfacctd: in NF_mpls_vpn_id_handler() double ntohl() calls
were applied for the case of 'vrfid'-encoded mpls_vpn_rd field.
! fix, sfacctd: wrong ethertype set for VLAN-tagged, MPLS-labelled
IPv6 traffic. Impacting BGP resolution among others. Thanks to
Jeremiah Millay ( @floatingstatic ) for his help resolving the
problem.
! fix, BGP, BMP daemons: parsing improvements: added a check for
BGP Open message and BGP Open Options lengths. Strengthened
parsing of Peer Up, Route Monitoring and Peer Down v4 messages.
! fix, BGP, BMP daemon: when using Avro encoding and Avro Schema
Registry, attempt to reconnect if serdes schemas are voided.
Also now checking for serdes schema definitions before doing a
serdes_schema_serialize_avro() to avoid triggering a SEGV.
Finally improved serdes logging.
! fix, BGP, Streaming Telemetry daemons: in daemon logs, summary
counters for amount of tables / entries dumped were wrong.
! fix, BGP daemon: distinguish among null and zero value AIGP
and Prefix SID attributes. Same applies for Local Preference
and MED attributes.
! fix, BMP daemon: resolved a memory leak in bgp_peers_free().
Thanks to Peter Pothier ( @pothier-peter ) for his patch. Also
resolved a leak caused by an invalid BGP message contained in a
BMP Route Message v4.
! fix, BMP daemon: correctly setting peer_ip and peer_tcp_port
JSON fields for Term messages. Also the correct bmp_router
value when bmp_daemon_parse_proxy_header feature is enabled.
! fix, BMP daemon: several encoding issues when using Apache Avro
ie. u_int64_t now correctly encoded with avro_value_set_long(),
certain u_int32_t fields switched to avro_value_set_long() due
to lack of unsignedness in Avro encoding, improved various
aspectes of Avro-JSON format output, etc.
! fix, pmtelemetryd: wrong parsing of pm_tfind() output was
leading to mistaken data attribution of UDP-based peers (always
first peer to connect was being picked).
! fix, pmtelemetryd: when set, the pidfile config directive was
not being correctly honoured.
! fix, RPKI: the RTR PDU element for maxLength is uint8, therefore
it might have been possible to transmit incorrect RTR data.
Thanks to Job Snijders ( @job ) for his patch.
! fix, SQL plugins: amended the text composition of SQL queries
that are involving latitude and longitude keys.
! fix, MySQL plugin: check for 'unix:' prefix string only when a
sql_host configuration directive is specified.
! fix, nfprobe: modernized Application Information export. Until
the previous release pmacct was adhering to aging NBAR model
whereas now NBAR2 has been implemented. Thanks to Rob Cowart
( @robcowart ) for helping out resolving this issue.
! fix, tee plugin: restored usefulness of tee_source_ip which was
broken in 1.7.6. Thanks to Jeremiah Millay ( @floatingstatic )
for reporting the issue.
! fix, maps_index: indexing of mpls_pw_id was broken. Also now,
when the feature is enabled, actual data is being referenced in
the index structure instead of creating a copy of it; thanks to
Sander van Delden ( @SanderDelden ) for reporting the memory
leak that was resulting from the copy.
! fix, kafka_common.c: solved memory leak in p_kafka_set_topic()
when Kafka session was getting in down state. Many thanks to
Peter Pothier ( @pothier-peter ) for nailing the issue.
! fix, net_aggr.[ch]: when a networks_file is specified in the
config, gracefully handle max memory structure depth; added
also de-duplication of entries.
! fix, pmacct-defines.h: if PCAP_NETMASK_UNKNOWN is not defined,
ie. in libpcap < 1.1.0, let's define it.
! fix, SO_REUSEPORT feature was being restricted to Linux only in
previous releases: now it has been unlocked to all other OS that
do support the feature.
! fix, split SO_REUSEPORT and SO_REUSEADDR setsockopt() calls.
Thanks to @eduarrrd for reporting and resolving the issue.
! fix, several code warnings catched gcc9 and clang.
- Obsoleted sql_history_since_epoch, pre_tag_map_entries and
refresh_maps configuration directives.
1.7.6 -- 07-02-2021
+ Added dependency to libcdada in an effort to streamline basic
data structures needed for everyday coding. All new structures
will make use of libcdada, old ones will be ported over time.
Libcdada offers basic data structures in C: ie. list, set, map/
hash table, queue and is a libstdc++ wrapper. Many thanks to
Marc Sune ( @msune ) for his work with libcdada and his enormous
help facilitating the integration.
+ BGP daemon: added suppport for Accumulated IGP Metric Attribute
(AIGP) and Label-Index TLV of Prefix-SID Attribute.
+ BGP daemon: added SO_KEEPALIVE TCP socket option (ie. to keep the
sessions alive via a firewall / NAT kind of device). Thanks to
Jared Mauch ( @jaredmauch ) for his patch.
+ BGP daemon: if comparing source TCP ports among BGP peers is
being enabled (config directive tmp_bgp_lookup_compare_ports),
print also BGP Router-ID as distinguisher as part of log/dump
output.
+ BMP daemon: added support for HAProxy Proxy Protocol Header in
the first BMP message in order to determine the original sender
IP address and port. The new bmp_daemon_parse_proxy_header config
directive enables the feature. Contribution is by Peter Pothier
( @pothier-peter ).
+ BMP daemon: improved support and brought implementation on par
with the latest drafting efforts at IETF wrt draft-cppy-grow-bmp-
path-marking-tlv, draft-xu-grow-bmp-route-policy-attr-trace,
draft-ietf-grow-bmp-tlv and draft-lucente-grow-bmp-tlv-ebit.
+ BMP daemon: added 'bgp_agent_map' equivalent feature for BMP.
+ nfacctd, nfprobe plugin: added support for collection and export
of NetFlow/IPFIX data over Datagram Transport Layer Security (in
short DTLS). The feature depends on the GnuTLS library.
+ nfacctd: added support for deprecated NetFlow v9 IE #104
(layer2packetSectionData) as it is implemented for NetFlow-lite
on Cisco devices. Reused code from IPFIX IE #315.
+ nfacctd: added support for MPLS VPN RD IE #90. This comes in two
flavours both found across vendor implementations: 1) IE present
in flow data and 2) IE present in Options data as a lookup from
IE #234 (ingressVRFID) and #235 (egressVRFID).
+ nfacctd: added a new timestamp_export aggregation primitive to
record the timestamp being carried in the header of NetFlow/IPFIX
messates (that is, the time at which the export was performed).
+ nfprobe plugin: added support for ICMP/ICMPv6 information as part
of the NetFlow/IPFIX export. The piece of info is encoded in the
destination port field as per the current common understandings
across vendors. As a result of that, the 'dst_port' primitive is
to be part of the aggregation method in order to leverage this
feature.
+ MySQL plugin: introduced support to connect to a MySQL server
via UNIX sockets.
+ tee plugin: added crc32 hash algorithm as a new balancing option
for nodes in the receiving pool. It hashes original exporter IP
address against a crc32 function. Thanks to @edge-intelligence
for the contribution.
+ Massive improvements carried out on the Continuous Integration
(CI) side, ie. to ensure better quality of the code, and on the
containerization side by offering official stable / bleeding edge
Docker images. Huge thanks to Marc Sune ( @msune ) to make all of
this possible.
! fix, BGP daemon: re-worked internal structuring of 'modern' BGP
attributes: for the sake of large-scale space optimization
certain attributes are confined in a separate (less used)
bgp_info_extra structure.
! fix, BGP daemon: improved support for BGP ADD-PATH, ie. made it
per Address-Family rather than global. Also comparisons upon
doing route looup were improved and normalized.
! fix, BGP daemon: use split buffers for recv and send functions
of the BGP x-connects feature. Also improved validation when
processing a bgp_daemon_xconnect_map.
! fix, BGP daemon: when using BGP x-connects, close unused file
descriptors in bgp_peer_xconnect_init() in order to avoid
quickly reaching the maximum amount of allowed open descriptors
in case of BGP flaps.
! fix, BGP daemon: trigger a log message for a missing entry while
processing bgp_daemon_xconnect_map in bgp_peer_xconnect_init().
! fix, BGP daemon: enabled log notifications (that is, log anti-
spam measure) upon reaching limit of allowed BGP peers.
! fix, BGP daemon: ecommunity_ecom2str(), first thing make sure
that the destination size is enough! Missing this did cause some
SEGVs due to heap corruption. Thanks to Chris Danis ( @cdanis )
for his help resolving the issue.
! fix, BGP daemon: solved a memory leak in aspath_make_str_count()
by returning result from aspath_make_empty(), if any. Thanks very
much to Peter Pothier ( @pothier-peter ) for his contribution.
! fix, BMP daemon: several encoding issues when using Apache Avro
ie. missing conditional branching, wrong field names, etc. Thanks
also to Raphael Barazzutti ( @rbarazzutti ) for several of those
fixes.
! fix, BMP daemon: throw an error for any issues (error or zero
length) related to the BGP Update PDU parsing; also added marker
and length checks for BGP Open PDU in Peer Up messages.
! fix, BMP daemon: both timestamp of the BMP event and its
arrival at the collector are now recorded and printed out
separately; before they were wrongly muxed on one single field
making it uncertain for the user what was the time reference.
! fix, BMP daemon: correctly print Peer Distinguisher for Route
Monitoring messages. Also improved BMP lookup comparisons in
order to factor in Peer Distinguisher if any.
! fix, BMP daemon: print 'is_in' boolean for Adj-Rib-In data
instead of having it implicit. Also print 'is_post' for Post-
Policy Adj-Rib-In data.
! fix, BMP daemon: upon receipt of a Termination message, do
proactively close the TCP session.
! fix, nDPI: newer versions of the library (ie. >= 3.2) require
calling ndpi_finalize_initialization() somewhere after the
detection module init finished. Contribution is from Toni Uhlig
( @lnslbrty ).
! fix, pmacctd: link checks were being mistakenly skipped when
reading from a pcap_savefile. Also now if a selected aggregation
primitive is unsuitable for a given Layer2, it is simply cleared
(with an info message issued) instead of making the daemon bail
out.
| fix, print plugin: bail the plugin out if its output was set to
stdout while the daemon was started as daemonized.
! fix, PostgreSQL plugin: in PG_compose_conn_string() allow any
intersection of host, port and cafile options.
! fix, nfprobe plugin: changed default export version from NetFlow
v5 to IPFIX.
! fix, sfprobe plugin: FreeBSD was complaining of errno 22 (Invalid
argument) upon sendto().
! fix, tee plugin: replication of IPv6 packets has been now tested
working. Previously the output message size was obviously encoded
wrongly and the checksum (mandatory piece of info to fill in IPv6,
contrary to IPv4 where it is optional) was not being computed.
! fix, kafka_common.c: improved p_kafka_check_outq_len() error log
message to report the amount of elements have been successfully
processed in order to better assess impact and dynamics of the
problem when inspecting logs.
! fix, net_aggr.c: if networks_file_filter is set to true, don't
add a default route to the table.
! fix, cfg.c: throw error if config file is not a regular file.
! fix, compiling against gcc10: renamed some variables and unified
declaration of others in order to be more friendly to the new
version of gcc. Also fixed several code warnings catched gcc8.
- Removed the IP prefix label feature that was enabled via the
--enable-plabel configure script switch.
1.7.5 -- 17-06-2020
+ pmacct & Redis: pmacct daemons can now connect to a Redis cache.
The main use-case currenly covered is: registering every stable
daemon component in a table so to have, when running a cluster
comprising several daemons / components, an olistic view of what
is currently running and where; shall a component stop running
or crash it will disappear from the inventory.
+ BMP daemon: as part of the IETF 107 vHackaton, preliminar support
for draft-xu-grow-bmp-route-policy-attr-trace and draft-lucente-
grow-bmp-tlv-ebit was introduced. Also added support for Peer
Distinguisher field in the BMP Per-Peer Header.
+ BMP daemon: added support for reading from savefiles in libpcap
format (pcap_savefile, pcap_savefile_delay, pcap_savefile_replay,
pcap_filter) as an alternative to the use of bmp_play.py.
+ BMP daemon: re-worked, improved and generalized support for TLVs
at the end of BMP messages. In this context, unknown Stats data
is handled as a generic TLV.
+ BMP daemon: added SO_KEEPALIVE TCP socket option (ie. to keep the
sessions alive via a firewall / NAT kind of device). Thanks to
Jared Mauch ( @jaredmauch ) for his patch.
+ nfacctd, nfprobe plugin: added usec timestamp resolution to IPFIX
collector and export via IEs #154, #155. For export, this can be
configured via the new nfprobe_tstamp_usec knob.
+ nfacctd: new nfacctd_templates_receiver and nfacctd_templates_port
config directives allow respectively to specify a destination
where to copy NetFlow v9/IPFIX templates to and a port where to
listen for templates from. If nfacctd_templates_receiver points to
a replicator and the replicator exports to nfacctd_templates_port
of a set of collectors then, for example, it gets possible to share
templates among collectors in a cluster for the purpose of seamless
scale-out.
+ pmtelemetryd: in addition to existing TCP, UDP and ZeroMQ inputs,
the daemon can now read Streaming Telemetry data in JSON format
from a Kafka broker (telemetry_daemon_kafka_* config knobs).
+ pmgrpcd.py: Use of multiple processes for the Kafka Avro exporter
to leverage the potential of multi-core/processors architectures.
Code is from Raphael P. Barazzutti ( @rbarazzutti ).
+ pmgrpcd.py: added -F / --no-flatten command-line option to disable
object flattening (default true for backward compatibility); also
export to a Kafka broker for (flattened) JSON objects was added (in
addition to existing export to ZeroMQ).
+ nDPI: introduced support for nDPI 3.2 and dropped support for all
earlier versions of the library due to changes to the API.
+ Docker: embraced the technology for CI purposes; added a docker/
directory in the file distribution where Dockerfile and scripts to
build pmacct and dependencies are shared. Thanks to Claudio Ortega
( @claudio-ortega ) for contributing his excellent work in the area.
! fix, pmacctd: pcap_setdirection() enabled and moved to the right
place in code. Libpcap tested for function presence. Thanks to
Mikhail Sennikovsky for his patch.
! fix, pmacctd: SEGV has been detected if passing messages with an
unsupported link layer.
! fix, uacctd: handle non-ethernet packets correctly. Use mac_len = 0
for non-ethernet packets in which case a zeroed ethernet header is
used. Thanks to @aleksandrgilfanov for his patch.
! fix, BGP daemon: improved handling of withdrawals for label-unicast
and mpls-vpn NLRIs.
! fix, BGP daemon: improved decoding of MPLS labels by passing it via
a "0x%02x%02x%01x" formatter. In the past some labels may have been
printed out incorrectly.
! fix, BGP daemon: decoding origin field correctly. Thanks to Peter
Pothier ( @pothier-peter ) for his patch.
! BGP Looking Glass: LG code was moved from pmbgpd to pmacct libbgp
so to be re-used in other components (ie. BGP thread of a flow
daemon, BMP daemon, etc.).
! fix, BMP daemon: timestamps for Route Monitoring message were set
to the current time rather than timestamp in the Per-Peer Header.
Thanks to Peter Pothier ( @pothier-peter ) for reporting the issue.
! fix, BMP daemon: V flag test to allow IPv6 prefixes/peers was not
correct (details in PR #362 on GitHub). Thanks @bcavns01 for his
patch.
! fix, BGP, BMP daemons: modified approach to message segmentation
by ensuring reading exactly one message at a time (instead of the
previous unaligned approach that was aiming to fill the available
read buffer up) and waiting for all segments be available before
processing a message (instead of the previous greedy approach that
was trying to parse also partial messages).
! fix, RPKI daemon: in rpki_prefix_lookup_node_match_cmp() prevent
invalid results to over-write a previously computed valid one.
! fix, pmtelemetryd: recv() does include a MSG_WAITALL option to make
sure all data is available before processing a given message; now
n alarm() syscall is introduced so to prevent stalls due to bogus /
incomplete data.
! fix, tee plugin: Tee_init_socks() now does not overwrite previously
computed address length anymore: issues were repoted with IPv6 IPv4-
mapped addresses.
! fix, nfprobe plugin: take into account ingress/egress interfaces,
if available, when comparing flows. Thanks to Mikhail Sennikovsky
for his patch.
! fix, pretag.c: incorrect string termiantion in pretag_copy_label()
was making the daemon SEGV upon certain conditions when defining
set_label statements in a pre_tag_map.
! fix, pretag_handlers.c: pretag_copy_label() instead of memcpy() in
pretag_label_handler() to not borrow reference to label value and
consequently lead to SEGV.
! fix, zmq_common.c: missing variable init in p_zmq_zap_handler() was
giving troubles with gcc7 compiler optimizations. Thanks to Yuri
Lachin ( @yuyutime ) for his support resolving this bug.
! fix, sql_common.c: print custom primitives with hex semantics among
quotes. Hexs require a char type defined in the SQL table schema.
! fix, addr.c: passing right aligned argument to ip6_addr_cmp() in
host_addr_mask_cmp(). This was found to impact selective replication
of IPv6 flows basing on src_net and/or dst_net primitives in the
pre_tag_map.
! fix, several code warnings catched by gcc7 and gcc8. Also several
functions were renamed to avoid namespace conflicts with linked
libraries.
- Obsoleted --enable-64bit knob which was already defaulting to true
for some releases already.
- Obsoleted savefile_wait config knob (pcap_savefile_wait must be
used instead).
1.7.4p1 -- 09-02-2020
! fix, pre_tag_map: a memory leak in pretag_entry_process() has been
introduced in 1.7.4. Thanks to Fabien Vincent and Olivier Benghozi
for their support resolving the issue.
1.7.4 -- 31-12-2019
+ Released pmgrpcd.py v3: a Streaming Telemetry collector and decoder
for multi-vendor environments written in Python3. It supports gRPC
transport along with Protobuf encoding as input and can output to
Kafka with Avro encoding. Output to files and JSON encoding is
currently supported sending data via ZMQ to pmtelemetryd first. It
was tested working with data input from Cisco and Huawei routers
and v3 replaces v2. Thanks to the Streaming Telemetry core team:
Matthias Arnold ( @tbearma1 ), Camilo Cardona ( @jccardonar ),
Thomas Graf ( @graf3 , @graf3net ), Paolo Lucente ( @paololucente ).
+ Introduced support for the 'vxlan' VXLAN/VNI primitive in all traffic
daemons (NetFlow/IPFIX, sFlow and libpcap/ULOG). Existing inner tunnel
primitives (ie. tunnel_src_host, tunnel_dst_host, tunnel_proto, etc.)
have been wired to the VXLAN decoding and new ones (tunnel_src_mac,
tunnel_dst_mac, tunnel_src_port, tunnel_dst_port) were defined.
+ BMP daemon: added support for Peer Up message namespace for TLVs
(draft-ietf-grow-bmp-peer-up) and also support for Route Monitor
and Peer Down TLVs (draft-ietf-grow-bmp-tlv).
+ BGP, BMP daemons: in addition to existing JSON export, data can now
be exported in Apache Avro format. There is also support for the
Confluent Schema Registry.
+ Introduced support for JSON-encoded Apache Avro encoding. While the
binary-encoded Apache Avro is always recommended for any production
scenarios (also to optionallly leverage Confluent Schema Registry
support), JSON-encoded is powerful for testing and troubleshooting
scenarios.
+ sfprobe plugin: added support for IPv6 transport for sFlow export.
sfprobe_agentip is an IP address put in the header of the sFlow
packet. If underlying transport is IPv6, this must be configured to
an IPv6 address.
+ zmq_common.[ch]: Improved modularity of the ZMQ internal API and
decoupled bind/connect from push/pull and pub/sub; also improved
support for inproc sockets. All to increase the amount of use-cases
covered by the API.
+ bgp_peer_src_as_map: added 'filter' key to cover pmacctd/uacctd use
cases.
+ nfprobe, sfprobe plugins: introduced [sn]fprobe_index_override to
override ifindexes dynamically determined (ie. by NFLOG) with values
computed by [sn]fprobe_ifindex.
+ MySQL, PostgreSQL plugins: added support for SSL/TLS connections by
specifying a CA certificate (sql_conn_ca_file).
+ Kafka, AMQP plugins: amqp_markers and kafka_markers have now been
properly re-implemented when output encoding is Avro using an own
Avro schema (instead of squatting pieces of JSON in the data stream
for the very purpose).
+ print plugin: introduced print_write_empty_file config knob (true,
false) to create an empty output file when there are no cache entries
to purge. Such behaviour was present in versions up to 0.14 and may
be preferred by some to the new >= 1.5 versions behaviour. Thanks to
Lee Yongjae ( @setup74 ) for the contribution.
! fix, signals.c: signals handling has been restructured in order to
block certain signals during critical sections of data processing.
Thanks to Vaibhav Phatarpekar ( @vphatarp ) for the contribution.
! fix, signals.c: slimmed reload() signal handler code and moved it to
a synchronous section. The handler is to reset logging output to
files or syslog. Thanks to Jared Mauch ( @jaredmauch ) for his
support resolving this.
! fix, pmbgpd, pmbmpd and pmtelemetryd daemons: added extra signals
handling (SIGINT, SIGTERM, SIGCHLD) consistently to traffic daemons.
! fix, BGP daemon: withdrawals of label-unicast (support introduced in
1.7.3) and mpls-vpn NLRIs did fail to parse in release 1.7.3 and
were silently discarded.
! fix, nfacctd: wired (BGP, BMP, ISIS, etc) lookups to NetFlow (Secure)
Event Logging (NEL/NSEL).
! fix, pmtelemetryd: re-implemented a decoder for so-called Cisco v1
Streaming Telemetry proprietary header over UDP/TCP streams.
! fix, pmtelemetryd: improved sanitization of input JSON objects by
also checking for isspace() other than isprint() for pretty-printed
objects.
! maps_index: optimized lookups, improved debugging output upon loading
indexes.
! fix, tee plugin: overwriting computed IP address length with socket
container length was found to prevent output data on some BSDs.
! fix, kafka_common.c: if taking the p_kafka_close() route, ensure to
return and not perform any further polling in order to avoid SEGVs.
! fix, BMP daemon: incorrect decoding of type was preventing correct
logging of Init and Term messages extra info. Also in Term messages
TLV data was incorrectly consumed twice triggering length check
warnings.
! fix, BMP daemon: added checks for successful BGP PDU parsing in both
Peer Up (BGP OPEN) and Route Monitor (BGP UPDATE) messages.
! fix, BMP daemon: improved length checks and making sure that strings
potentially non null-terminated are now terminated. Also TLV-related
code has been refactored.
! fix, pmbgp.py: the example client for BGP Looking Glass was migrated
to Python3: thanks to @brusilov for the contribution.
! fix, nfacctd: if src_port or dst_port primitives are selected, enable
IP fragment handling. Needed to process L4 of IPFIX IE #351.
! fix, nfv9_template.c: correct handling of variable-length IPFIX
fields. Thanks to Nimrod Mesika ( @nimrody ) for the contribution.
! fix, PostgreSQL plugin: ABSTIME was replaced with to_timestamp() in
queries as support for ABSTIME was dropped as of PostgreSQL 12. Many
thanks to Manuel Mendez ( @mmlb ) for the contribution.
! fix, PostgreSQL plugin: SEGVs were observed when the queue of pending
queries was non-empty (ie. nfacctd_time_new set to false, default);
thanks to Guo-Wei Su ( @nansenat16 ) for the contribution.
! fix, cfg_handlers: [sn]facctd_disable_checks, nfacctd_disable_opt_
scope_check could not be properly set to false.
! fix, sql_common.c: src_host_coords and dst_host_coords primitives
have been correctly spaced in SQL queries. Also float values are now
quoted. Finally, sampling_direction primitive is encoded correctly.
! fix, kafka plugin: if kafka_avro_schema_registry is in use, subject
name is aligned to Kafka topic name (if topic is not dynamic).
! fix, pretag.c: when using 'label', store the label string in the
heap (instead of the stack). Thanks to Raphael P. Barazzutti
( @rbarazzutti ) for the contribution.
! fix, pretag.c: JEQ labels are now correctly free() during init upon
map reload.
! fix, zmq_common.c: missing variable init in p_zmq_zap_handler() was
causing plugin_pipe_zmq operations to fail on certain compilers (ie.
gcc7). Thanks to Yuri Lachin ( @yuyutime ) for his support.
! fix, cfg_handlers.c: reviewed handling of parsed 'zero' value for
several config directives.
! fix, countless code warnings when enabling -Wall (--enable-debug);
also included -Wall in Continuous Integration tests. Restructured
globals, header inclusions, function prototypes definition, etc.
Many thanks to Marc Sune ( @msune ) for all his efforts.
! fix, configure.ac: evaluation of --enable-debug pushed to the end of
the script so to not interfere with tests (ie. alignment, endianess,
etc.).
- BMP daemon: retired support for draft-hsmit-bmp-extensible-routemon-
-msgs.
- AMQP plugin: obsoleted amqp_avro_schema feature (which includes
amqp_avro_schema_routing_key and amqp_avro_schema_refresh_time keys
config keys). Avro schemas can now only be written to files.
1.7.3 -- 16-05-2019
+ Introduced the RPKI daemon to build a ROA database and check prefixes
validation status and coverages. Resource Public Key Infrastructure
(RPKI) is a specialized public key infrastructure (PKI) framework
designed to secure the Internet routing. RPKI uses certificates to
allow Local Internet Registries (LIRs) to list the Internet number
resources they hold. These attestations are called Route Origination
Authorizations (ROAs). ROA information can be acquired in one of the
two following ways: 1) importing it using the rpki_roas_file config
directive from a file in the RIPE Validator format or 2) connecting
to a RPKI RTR Cache for live ROA updates; the cache IP address/port
being defined by the rpki_rtr_cache config directive (and a few more
optional rpki_rtr_* directives are available and can be reviwed in
the CONFIG-KEYS doc). The ROA fields will be populated with one of
these five values: 'u' Unknown, 'v' Valid, 'i' Invalid no overlaps,
'V' Invalid with a covering Valid prefix, 'U' Invalid with a covering
Unknown prefix. Thanks to Job Snijders ( @job ) for his support and
vision.
+ Introducing pmgrpcd.py, written in Python, a daemon to handle gRPC-
based Streaming Telemetry sessions and unmarshall GPB data. Code
was mostly courtesy by Matthias Arnold ( @tbearma1 ). This is in
addition (or feeding into) pmtelemetryd, written in C, a daemon to
handle TCP/UDP-based Streaming Telemetry sessions with JSON-encoded
data. Thanks to Matthias Arnold ( @tbearma1 ) and Thomas Graf for
their support and contributing code.
+ pmacctd, uacctd: added support for CFP (Cisco FabricPath) and Cisco
Virtual Network Tag protocols. Both patches were courtesy by Stephen
Clark ( @sclark46 ).
+ print plugin: added 'custom' to print_output. This is to cover two
main use-cases: 1) use JSON or Avro encodings but fix the format of
the messages in a custom way and 2) use a different encoding than
JSON or Avro. See also example in examples/custom and new directives
print_output_custom_lib and print_output_custom_cfg_file. The patch
was courtesy by Edge Intelligence ( @edge-intelligence ).
+ Introducing mpls_pw_id aggregation primitive and mpls_pw_id key in
pre_tag_map to filter on signalled L2 MPLS VPN Pseudowire IDs.
+ BGP daemon: added bgp_disable_router_id knob to enable/disable BGP
Router-ID check, both at BGP OPEN time and BGP lookup. Useful, for
example, in scenarios with split BGP v4/v6 AFs over v4/v6 transports.
+ BGP, BMP daemons: translate origin attribute numeric value into IGP
(i), EGP (e) and Incomplete (u) strings.
+ plugins: added new plugin_exit_any feature to make the daemon bail
out if any (not all, which is the default behaviour) of the plugins
exits.
+ maps_index: improved selection of buckets for index hash structure
by picking the closest prime number to the double of the entries of
the map to be indexed in order to achieve better elements dispersion
and hence better performances.
+ nfacctd: added support for IPFIX templateId-scoped (IE 145) sampling
information.
+ pmacctd, uacctd, sfacctd, nfacctd: added a -M command-line option to
set *_markers (ie. print_markers) to true and fixed -A command-line
option to set print_output_file_append to align to true/false.
! fix, BGP, BMP, Streaming Telemetry daemons: improved sequencing of
dump events by assigning a single sequence number per event (ie. for
streaming pipeline scenarios in order to reduce correlation with
dump_init/dump_close messages). Also amount of record dumped was
added to the close message.
! fix, BGP, BMP, Streaming Telemetry daemons: removed hierarchical
json_decref() since json_object_get() borrows reference. This was
occasionaly leading to SEGVs.
! fix, uacctd: dynamically allocate jumbo_container buffer size as
packets larger than 10KB, previous static allocation, would lead to
crashes.
! fix, nfacctd: wired (BGP, BMP, ISIS, etc.) lookups to the NEL/NSEL
use-case.
! fix, nfacctd: search for IE 408 (dataLinkFrameType) was leading to
SEGVs. Also improved handling of variable-length IPFIX templates.
! fix, BMP daemon: solved an occasional truncation of the last message
in a packet.
! fix, BGP daemon: when processing bgp_daemon_md5_file, ipv4 addresses
were incorrectly translated to ipv4-mapped ipv6 ones as a result of
which TCP-MD5 hashes were not correctly bound to sockets.
! fix, BGP daemon: improved label-unicast and mpls-vpn SAFIs handling
(some bogus messages, multiple labels, etc.).
! fix, BGP daemon: introduced PREFIX_STRLEN to make enough room for
prefix2str() calls (before unsufficient INET6_ADDRSTRLEN was used).
! fix, BMP daemon: improved handling of ADD-PATH capability.
! fix, plugins: an incorrect evaluation in P_cache_attach_new_node did
make possible to buffer overrun in plugins cache allocation. This was
found related to a "[..]: Assertion `!cache_ptr->stitch' failed."
daemon bail-out message.
! fix, plugins: if pidfile directive was enabled, exit_gracefully() was
mistakenly deleting the plugin pidfile when called by a child process
(ie. writer, dumper, etc.).
! fix, plugins: when taking exit_gracefully(), if the process is marked
as 'is_forked', just exit and don't perform extra ops in exit_all()
or exit_plugin().
! fix, plugins: re-evaluate dynamic tables/files name if *_refresh_time
is different than *_history period.
! fix, SQL plugins: a missing 'AND' was making SQL statements related
to src_host_coords and dst_host_coords fail.
! fix, GeoIPv2: if no match is returned by libmaxminddb, return O1 code
(Other Country) instead of a null value.
! fix, flow_to_rd_map: mpls_vpn_id was not working when maps_index was
enabled. Also partly re-written mpls_vpn_id handler.
! fix, nfprobe plugin: serialize_bin() function introduced for correct
serialization of custom primitives defined with 'raw' semantics.
! fix, PostgreSQL plugin: testing for presence of PQlibVersion() in
libpq to prevent compiling issues (ie. on CentOS 6).
! fix, MySQL plugin: including mysql_version.h to compile successfully
against newer MariaDB releases.
! fix, nDPI classification: send log message if 'class' primitive is
selected but nDPI is not compiled in; also updated code to follow
API changes in versions >= 2.6 of the library. Dropped support for
versions < 2.4.
! fix, sfprobe plugin: added (and documented) conditional for optional
export of classification info.
! fix, aggregate_primitives: field_type is now also allowed for pmacctd
and uaccd daemons so that it can be used for NetFlow v9/IPFIX export
(nfprobe plugin) purposes.
! fix, pre_tag_map: if no 'ip' keyword is specified, an entry of the
map gets recirculated in order to be set for both v4 and v6 maps. If
a 'set_label' is also specified, it was causing a SEGV. Now the label
is correctly copied in case of recirculation.
! fix, zmq_common.c: added option for non-blocking p_zmq_send_bin() as
otherwise program would block in case of no consumers (main use-case:
flow replication over ZeroMQ queues); as a result, a generous hwm
value was added on both sides of these queues.
! fix, zmq_common.c: ZAP socket moved inside thread to prevent failed
assert() when compiling with gcc7/gcc8. Also a single user/password
auto-generated combination is used for all plugins.
! fix, signals.c: SIGUSR1 handler for nfacctd and nfacctd is changed to
syncronous in order to prevent race conditions. Also, in pmacctd,
upon sending SIGUSR1, stats were not printed when reading packets
from a pcap_interfaaces_map.
! fix, plugin_cmn_json.c: if leaving protocols numerical (ie. proto,
tunnel_proto primitives), convert them to string-represented numbers
for data consistency for consumers.
! fix, util.c: open_output_file(), if file exists and it's a FIFO then
set O_NONBLOCK when opening.
! fix, pretag.c: pretag_index_report() was reporting incorrect info of
the hash structure built for the maps_index feature. Its format was
has also changed to be better parseable.
! fix, compile time warnings: several warnings were addressed including
but not restricted to -Wformat ones. Also an annotation was added to
the Log function to inform the compiler it's a printf-style function,
allowing it to give warnings for argument mismatches.
- --enable-ipv6 configure script switch has been deprecated and, as a
result, IPv6 support was made mandatory.
- BGP daemon: removed unused pathlimit field from bgp_attr structure.
- pmacct client: removed deprecated SYM field from from formatted and
CSV headers.
1.7.2 -- 26-11-2018
+ nfacctd, sfacctd: added Kafka broker among the options to receive
NetFlow/IPFIX, sFlow data from. Host, port and topic should all be
specified along with an optional config file to pass to librdkafka.
+ nfacctd, sfacctd, pmtelemetryd: added ZeroMQ queue among the options
to receive NetFlow/IPFIX, sFlow or Streaming Telemetry data from. An
IP address and port should be specified.
+ nfacctd, sfacctd: added sampling_direction to the set of supported
primitives, valid values being ingress, egress and unknown.
+ nfacctd, sfacctd: stats, ie. amount of NetFlow/IPFIX or sFlow packets
received per router, are now available when in tee mode. Stats can be
retrieved via a SIGUSR1 UNIX signal.
+ pcap_savefile_replay: a feature to replay content for the specified
amounf of time when reading from a pcap_savefile.
+ pre_tag_map: added several new keys: src_net and dst_net (to tag on
source and destination IP prefixes respectively), bgp_nexthop (to
tag on BGP nexthop) and nat_event.
+ BGP daemon: added bgp_lrgcomm_pattern feature to filter large BGP
communities (in addition to existing equivalent knobs to filter on
standard and extended communities).
+ BMP, Streaming Telemetry daemons: msglog_file and dump_file config
directives now offer $bmp_router, $bmp_router_port, $telemetry_node
and $telemetry_node_port variables.
+ BGP, BMP, Streaming Telemetry daemons: added BGP, BMP and Streaming
Telemetry exporter TCP/UDP port as variable for dump/log filenames
(to better support NAT traversal scenarios).
+ BGP, BMP daemons: added message sequencing to both BGP and BMP dumps
(bgp_table_dump_*, bmp_dump_*). If dumping and logging are enabled
in parallel then sequencing the dumps allows for check pointing at
regular time intervals.
+ BMP daemon: implemented draft-hsmit-bmp-extensible-routemon-msgs for
a tlv-based encoding of route-monitoring messages with a new message
type.
+ Streaming Telemetry daemon: added sample decoders for gRPC / GPB for
Cisco and Huawei platforms, written in Python. Telemetry data is
decoded using vendor-supplied proto files and output in JSON format
in a ZeroMQ queue - suitable for ingestion in pmtelemetryd. Docs and
sample code is available in the telemetry/ directory. This is all in
addition to TCP/UDP transports and JSON encoding supported natively
in pmtelemetryd.
+ kafka plugin: introduced support for Confluent Schema Registry via
libserdes. A registry can be supplied via kafka_avro_schema_registry
config directive; the schema is generated automatically. The feature
enables validation of data passed through a Kafka broker and uses
Avro encoding.
+ kafka plugin: added $in_iface key (input interface) to the set of
variables supported by kafka_partition_key. Extremely useful when
coupled to $peer_src_ip in some scenarios.
+ print, IMT plugins: separator for CSV format can now be space (\s)
or tab (\t).
+ tee plugin: added Kafka broker among the emitters. kafka_broker and
kafka_topic knobs are now available in the tee_receivers map and a
tee_kafka_config_file directive allows to define a file with config
to pass to librdkafka.
+ tee plugin: added ZeroMQ queue among the emitters. zmq_address knob
defines the queue IP address and port to emit to.
+ tee plugin: introducing support for complex pre_tag_map when doing
replication of NetFlow/IPFIX (sFlow replication had already this).
With this feature flows are individually evaluated against supplied
filters (input interface, BGP next-hop, etc.) and (not) replicated
accordingly.
+ GeoIP v2: added support for latitude and longitude primitives via
src_host_coords and dst_host_coords knobs. This is in addition to
existing country and pocode supports.
+ files_uid, files_gid: now also user and group strings are accepted.
This is in addition to user and group IDs.
! fix, nfacctd: NF_evaluate_flow_type() improved to not detect Cisco
ASA flows (ie. those including initiator and responder octets) as
events. Also improved sanity checking of received NetFlow v9/IPFIX
data and options templates and reviwed modulo functions and improved
template hashing.
! fix, BGP, BMP, Streaming Telemetry daemons: improved log sequencing
by handling counter wrap-up more gracefully. Also a log sequencing
API was developed to improve code re-use.
! fix, BGP daemon: added check for duplicate Router-IDs at BGP OPEN
parsing time. If a duplicate is detected, the session BGP OPENing of
the new session is dropped.
! fix, BGP daemon: ADD-PATH capability was checked only in the first
AFI/SAFI and was being set in the reply for last AFI/SAFI RECEIVE(1)
if first included SEND(2) or SEND-RECEIVE(3). Thanks to Markus Weber
( @FvDxxx ) for his patch.
! fix, BGP daemon: upon route lookup, don't perform ADD-PATH logics if
no PATH-ID (even if ADD-PATH capability is announced by the peer).
Thanks to Camilo Cardona ( @jccardonar ) for his support solving the
issue.
! fix, BGP daemon: graceful handling of invalid AS-PATH segment types
(ie. AS-PATH in BGP UPDATE inconsistent with capabilities passed in
BGP OPEN) in order to avoid SEGVs.
! fix, pmtelemetryd: improved support for UDP timeouts. Also reviewed
natively supported encodings: removed zjson and GPB was moved to pre-
processors (with samples available in telemetry/decoders directory).
! fix, pmtelemetryd: no dump_init / dump_close events sequencing since
all messages are sequenced anyway (consistency with other daemons).
! fix, kafka_common.c: now destroying both config and topic config as
part of p_kafka_close() in order to avoid memory leaks. Also, port is
omitted from broker string if not passed to p_kafka_set_broker(). And
finally output queue length checks in p_kafka_check_outq_len() have
been relaxed (to counter temporary hickups that need more patience).
! fix, kafka plugin: kafka_partition default was zero (that is, a valid
partition number) instead of -1 (RD_KAFKA_PARTITION_UA or unassigned)
which allows librdkafka to attach a partitioner.
! fix, SQL plugins: sql_table_schema is honoured even if sql_table_name
is non-dynamic. This is to cover cases where the table is rotated
externally.
! fix, mysql plugin: my_bool replaced with bool. The plugin now does
compile against MySQL 8.0. Also added inclusion of stdbool.h as on
some systems bool is not defined. Improved overall probing for MySQL
headers.
! fix, pgsql plugin: sql_recovery_backup_host was not being honoured.
PG_create_backend() now composes a proper conn_string.
! fix, print plugin: increase successful queries number, QN, only if
the output file was successfully opened.
! fix, zmq_common.c: moved ZAP socket initialization inside the ZAP
handler. See: https://github.com/zeromq/libzmq/issues/3313 .
! fix, util.c: length checks in handle_dynname_internal_strings() were
reviewed. Existings were not working in absence of starting/trailing
non-variable strings.
! fix, util.c: use lockf() instead of more problematic flock(). Thanks
to Yuri Lachin ( @yuyutime ) and Miki Takata ( @mikiT ) for their
support.
! fix, util.c: in compose_timestamp() pad usecs and use "%ld" since
time fields are signed longs. Thanks to @raymondrussell for the
patch.
! fix, ndpi_util.c: a protocol bitmask is now set in order to increase
match rate. Patch is courtesy by @rsolsn.
! fix, compile time warnings: several warnings were addressed including
but not restricted to -Wreturn-time, -Wunused-variable, implicit func
declarations, -Wformat-extra-args, -Wunused-label, -Wunused-value,
-Wunused-function, sbrk calls, -Wpointer-to-int-cast, -Wparentheses
and -Wint-to-pointer-cast.
! fix, dangerous uninitialized values: net_aggr.c, pmacct.c: in merge()