Skip to content

Commit

Permalink
Fix benchmarks and citations
Browse files Browse the repository at this point in the history
  • Loading branch information
@gdalle
gdalle committed Mar 13, 2024
1 parent 29775bd commit 083c26b
Show file tree
Hide file tree
Showing 8 changed files with 2,006 additions and 1,973 deletions.
2 changes: 0 additions & 2 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,6 @@ scratchpad.jl
*.ipynb
*.ipynb_checkpoints

*.csv
*.txt
.benchmarkci/
/benchmark/*.json

Expand Down
1,851 changes: 1,851 additions & 0 deletions libs/HMMComparison/experiments/results/benchmark.svg
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
40 changes: 40 additions & 0 deletions libs/HMMComparison/experiments/results/julia_setup.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 12 × Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, skylake)
Threads: 1 default, 0 interactive, 1 GC (on 12 virtual cores)
Environment:
LD_LIBRARY_PATH = :/home/guillaume/Software/gurobi1002/linux64/lib:/home/guillaume/Software/gurobi1002/linux64/lib
JULIA_CONDAPKG_BACKEND = System
JULIA_CONDAPKG_EXE = /home/guillaume/miniforge3/bin/mamba

# Multithreading

Julia threads = 1
OpenBLAS threads = 6

# Julia packages

Project HMMComparison v0.1.0
Status `~/Work/GitHub/Julia/HiddenMarkovModels.jl/libs/HMMComparison/Project.toml`
[6e4b80f9] BenchmarkTools v1.5.0
[13f3f980] CairoMakie v0.11.9
[992eb4ea] CondaPkg v0.2.22
[a93c6f00] DataFrames v1.6.1
[31c24e10] Distributions v0.25.107
[b2b3ca75] HMMBase v1.0.7
[557005d5] HMMBenchmark v0.1.0 `../HMMBenchmark`
[84ca31d5] HiddenMarkovModels v0.5.0 `../..`
[2ab3a3ac] LogExpFunctions v0.3.27
[6099a3de] PythonCall v0.9.15
[189a3867] Reexport v1.2.2
[860ef19b] StableRNGs v1.0.1
[37e2e46d] LinearAlgebra
[9a3f8284] Random
[2f01184e] SparseArrays v1.10.0
16 changes: 16 additions & 0 deletions libs/HMMComparison/experiments/results/python_setup.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
Pytorch threads = 1

# Python packages

CondaPkg Status /home/guillaume/Work/GitHub/Julia/HiddenMarkovModels.jl/libs/HMMComparison/CondaPkg.toml
Environment
/home/guillaume/Work/GitHub/Julia/HiddenMarkovModels.jl/libs/HMMComparison/.CondaPkg/env
Packages
cpuonly v2.0 (channel=pytorch)
jax v0.4.25 (channel=conda-forge)
numpy v1.26.4 (channel=conda-forge)
pytorch v2.2.1 (channel=pytorch)
Pip packages
dynamax v0.1.3
hmmlearn v0.3.2
pomegranate v1.0.4
96 changes: 96 additions & 0 deletions libs/HMMComparison/experiments/results/results.csv
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
implem,algo,nb_states,obs_dim,seq_length,nb_seqs,bw_iter,sparse,custom_dist,samples,time_median,time_minimum,time_std,time_quantile25,time_quantile75,time_maximum,time_mean
dynamax,baum_welch,8,1,100,50,5,false,false,5,5.55372961e8,5.30873628e8,2.696824778343125e7,5.41874465e8,5.74246438e8,5.98835343e8,5.60240567e8
dynamax,forward,8,1,100,50,5,false,false,20,841908.0,667058.0,140665.28027980952,682256.0,958038.75,992104.0,826680.25
dynamax,viterbi,8,1,100,50,5,false,false,20,1.4858475e6,1.413911e6,132014.75862509504,1.44973275e6,1.56127075e6,1.907671e6,1.531387e6
dynamax,forward_backward,8,1,100,50,5,false,false,20,2.109884e6,1.743491e6,134713.7643411998,2.0641295e6,2.1278695e6,2.166556e6,2.05135035e6
dynamax,baum_welch,2,1,100,50,5,false,false,5,4.68308403e8,4.48667986e8,9.058320588222703e6,4.65697752e8,4.68381647e8,4.71323285e8,4.644758146e8
dynamax,forward,2,1,100,50,5,false,false,20,283395.5,268133.0,14774.377975952686,278040.0,285345.75,328352.0,285507.45
dynamax,viterbi,2,1,100,50,5,false,false,20,150743.5,145975.0,15132.111427010896,147711.25,159351.75,202508.0,157364.35
dynamax,forward_backward,2,1,100,50,5,false,false,20,518555.0,469043.0,55538.65262212956,483720.25,587306.75,621600.0,536118.85
dynamax,baum_welch,6,1,100,50,5,false,false,4,5.7089923e8,5.59364261e8,8.142230036274255e6,5.664230705e8,5.744339595e8,5.78668479e8,5.699578e8
dynamax,forward,6,1,100,50,5,false,false,20,661156.0,641503.0,88353.51625712114,647391.0,835947.75,848630.0,710748.05
dynamax,viterbi,6,1,100,50,5,false,false,20,800385.5,795571.0,22488.03353007995,797121.0,818049.5,882431.0,810429.05
dynamax,forward_backward,6,1,100,50,5,false,false,20,1.8382555e6,1.540022e6,109768.6197239208,1.82493125e6,1.8692085e6,1.994827e6,1.8240201e6
dynamax,baum_welch,4,1,100,50,5,false,false,5,5.15695466e8,4.70775862e8,3.200413832730334e7,5.10174039e8,5.22626463e8,5.60428709e8,5.159401078e8
dynamax,forward,4,1,100,50,5,false,false,20,389952.0,373611.0,30208.951510403247,379707.5,396536.5,506198.0,397118.1
dynamax,viterbi,4,1,100,50,5,false,false,20,650111.0,497032.0,65993.7827570861,558782.75,661220.5,680719.0,617500.8
dynamax,forward_backward,4,1,100,50,5,false,false,20,923654.0,763362.0,100581.40506370684,907356.5,934588.25,1.225786e6,936452.8
dynamax,baum_welch,10,1,100,50,5,false,false,4,6.574755055e8,6.41496077e8,8.470793623714134e7,6.41912618e8,7.09425497e8,8.1900335e8,6.938626095e8
dynamax,forward,10,1,100,50,5,false,false,20,1.247139e6,1.008284e6,147560.68007321956,1.0361915e6,1.30338125e6,1.388337e6,1.19069325e6
dynamax,viterbi,10,1,100,50,5,false,false,20,1.808002e6,1.733144e6,131247.38868215325,1.743928e6,1.9366325e6,2.209045e6,1.8567839e6
dynamax,forward_backward,10,1,100,50,5,false,false,20,3.237719e6,3.117605e6,189211.52662431175,3.20088425e6,3.34795e6,3.94738e6,3.30022115e6
pomegranate,baum_welch,8,1,100,50,5,false,false,20,7.77930635e7,7.3626116e7,6.7719912297365125e6,7.429900275e7,8.50519115e7,9.6235316e7,8.023675985e7
pomegranate,forward,8,1,100,50,5,false,false,20,4.383477e6,4.210591e6,118675.58054891095,4.25491775e6,4.417904e6,4.609048e6,4.3638719e6
pomegranate,forward_backward,8,1,100,50,5,false,false,20,1.2186022e7,1.0754096e7,1.4545618691587576e6,1.191290275e7,1.275966325e7,1.7329472e7,1.261543125e7
pomegranate,baum_welch,2,1,100,50,5,false,false,20,4.86546135e7,4.8031764e7,651598.8352084252,4.836104975e7,4.8776926e7,5.0525155e7,4.874857685e7
pomegranate,forward,2,1,100,50,5,false,false,20,3.7014845e6,3.677169e6,173975.64196878253,3.688722e6,3.77947525e6,4.368387e6,3.77755665e6
pomegranate,forward_backward,2,1,100,50,5,false,false,20,8.9486435e6,8.641816e6,259859.13766619487,8.7870705e6,9.133007e6,9.551035e6,8.99579275e6
pomegranate,baum_welch,6,1,100,50,5,false,false,20,6.5560976e7,6.4468714e7,747193.6732259616,6.51532305e7,6.592513575e7,6.7541985e7,6.56122558e7
pomegranate,forward,6,1,100,50,5,false,false,20,4.371999e6,4.03798e6,216687.24306401052,4.2096835e6,4.437135e6,4.903356e6,4.37005205e6
pomegranate,forward_backward,6,1,100,50,5,false,false,20,1.0177502e7,9.60596e6,374692.86004637746,1.012466125e7,1.028083875e7,1.1267479e7,1.020437055e7
pomegranate,baum_welch,4,1,100,50,5,false,false,20,5.76103125e7,5.4862406e7,1.3866122095763357e6,5.732704725e7,5.875926275e7,6.0618023e7,5.78863212e7
pomegranate,forward,4,1,100,50,5,false,false,20,3.8399915e6,3.784403e6,31837.649928603645,3.81227925e6,3.868094e6,3.890019e6,3.84046015e6
pomegranate,forward_backward,4,1,100,50,5,false,false,20,9.330155e6,9.081806e6,193721.4311600416,9.22255475e6,9.48621575e6,9.812605e6,9.3686478e6
pomegranate,baum_welch,10,1,100,50,5,false,false,20,1.074988565e8,8.5353267e7,8.356179238547903e6,9.88777335e7,1.096597025e8,1.11288526e8,1.0366778055e8
pomegranate,forward,10,1,100,50,5,false,false,20,4.8062375e6,4.677659e6,130395.17090630393,4.7281675e6,4.85695975e6,5.23633e6,4.825499e6
pomegranate,forward_backward,10,1,100,50,5,false,false,20,1.3849443e7,1.2299825e7,815457.72994161,1.3505304e7,1.470987075e7,1.5168059e7,1.397941805e7
HiddenMarkovModels.jl,baum_welch,8,1,100,50,5,false,false,20,1.2464112e7,1.1978602e7,1.2300987142442665e6,1.214651425e7,1.273314575e7,1.7582953e7,1.276221535e7
HiddenMarkovModels.jl,forward,8,1,100,50,5,false,false,20,1.1936095e6,1.16643e6,31707.375908080372,1.17055675e6,1.21971425e6,1.279745e6,1.19884085e6
HiddenMarkovModels.jl,viterbi,8,1,100,50,5,false,false,20,1.373442e6,1.355383e6,27927.596429451565,1.3662685e6,1.4067975e6,1.4497e6,1.3871177e6
HiddenMarkovModels.jl,forward_backward,8,1,100,50,5,false,false,20,1.95352e6,1.793047e6,244045.37852251445,1.85206175e6,2.09695675e6,2.485606e6,2.03635475e6
HiddenMarkovModels.jl,baum_welch,2,1,100,50,5,false,false,20,6.105442e6,5.868571e6,100891.45823229771,6.06819e6,6.1731355e6,6.279361e6,6.11544765e6
HiddenMarkovModels.jl,forward,2,1,100,50,5,false,false,20,586194.0,582346.0,23321.561325531497,584864.75,590243.0,650467.0,597307.85
HiddenMarkovModels.jl,viterbi,2,1,100,50,5,false,false,20,367382.0,364708.0,52849.59863739942,365806.5,386594.75,586545.0,390641.65
HiddenMarkovModels.jl,forward_backward,2,1,100,50,5,false,false,20,957350.0,912464.0,262670.4566940144,935210.25,1.09266275e6,1.703596e6,1.0895105e6
HiddenMarkovModels.jl,baum_welch,6,1,100,50,5,false,false,20,1.22526145e7,1.1379799e7,1.2854175184947702e6,1.2002506e7,1.251070625e7,1.5994728e7,1.26773131e7
HiddenMarkovModels.jl,forward,6,1,100,50,5,false,false,20,1.027658e6,1.024276e6,28295.129607454204,1.02520325e6,1.040335e6,1.116318e6,1.04137535e6
HiddenMarkovModels.jl,viterbi,6,1,100,50,5,false,false,20,960505.0,905032.0,29244.556882660367,937952.25,980802.0,1.024054e6,960904.0
HiddenMarkovModels.jl,forward_backward,6,1,100,50,5,false,false,20,1.5141645e6,1.490759e6,25490.982494073425,1.49474825e6,1.5314435e6,1.564875e6,1.51838925e6
HiddenMarkovModels.jl,baum_welch,4,1,100,50,5,false,false,20,8.2834645e6,7.813959e6,433601.14893340145,7.8609155e6,8.46813725e6,9.363313e6,8.24756395e6
HiddenMarkovModels.jl,forward,4,1,100,50,5,false,false,20,800630.5,779070.0,483823.93358102744,782787.0,877630.75,2.086082e6,1.04665085e6
HiddenMarkovModels.jl,viterbi,4,1,100,50,5,false,false,20,600377.5,597248.0,21508.9880573827,599293.75,629750.0,671383.0,615790.1
HiddenMarkovModels.jl,forward_backward,4,1,100,50,5,false,false,20,1.150694e6,1.116064e6,47375.242432264014,1.12363475e6,1.180026e6,1.314607e6,1.1602974e6
HiddenMarkovModels.jl,baum_welch,10,1,100,50,5,false,false,20,1.58476515e7,1.5440541e7,914663.3262003307,1.556346125e7,1.639449175e7,1.8746422e7,1.61643668e7
HiddenMarkovModels.jl,forward,10,1,100,50,5,false,false,20,1.4294785e6,1.362703e6,28001.579260867264,1.40582875e6,1.44326525e6,1.480862e6,1.42623395e6
HiddenMarkovModels.jl,viterbi,10,1,100,50,5,false,false,20,1.740999e6,1.660761e6,61214.9359576996,1.70594075e6,1.753856e6,1.965691e6,1.7403891e6
HiddenMarkovModels.jl,forward_backward,10,1,100,50,5,false,false,20,2.0513195e6,1.925748e6,366515.7375732783,2.0165155e6,2.71627175e6,2.85784e6,2.3169744e6
HMMBase.jl,baum_welch,8,1,100,50,5,false,false,15,3.1114279e7,3.0633095e7,1.0124903452677043e7,3.07735335e7,3.26051965e7,6.5259218e7,3.521065846666667e7
HMMBase.jl,forward,8,1,100,50,5,false,false,20,1.9331715e6,1.805652e6,241930.56419435178,1.84744925e6,2.312583e6,2.362578e6,2.06864465e6
HMMBase.jl,viterbi,8,1,100,50,5,false,false,20,2.068797e6,1.484141e6,920546.8516569462,1.786302e6,2.075038e6,5.914333e6,2.1217884e6
HMMBase.jl,forward_backward,8,1,100,50,5,false,false,20,2.6176635e6,2.503086e6,867759.6522743647,2.5730065e6,2.654923e6,6.394062e6,2.8859618e6
HMMBase.jl,baum_welch,2,1,100,50,5,false,false,20,5.706091e6,5.510154e6,796750.0631170339,5.62787925e6,6.0942905e6,8.70201e6,6.05117695e6
HMMBase.jl,forward,2,1,100,50,5,false,false,20,510893.0,495105.0,16447.79627856123,509710.75,523721.5,565226.0,515504.0
HMMBase.jl,viterbi,2,1,100,50,5,false,false,20,510256.5,356066.0,104590.16139756369,366512.75,575623.25,600550.0,479545.7
HMMBase.jl,forward_backward,2,1,100,50,5,false,false,20,997675.5,840951.0,59689.357649042126,925431.0,1.00256325e6,1.008717e6,961136.15
HMMBase.jl,baum_welch,6,1,100,50,5,false,false,20,2.03820905e7,1.9944053e7,1.3425800349466342e6,2.0264995e7,2.1763589e7,2.4016993e7,2.11247954e7
HMMBase.jl,forward,6,1,100,50,5,false,false,20,1.701162e6,1.343845e6,146820.86310836143,1.6792585e6,1.73328575e6,1.748875e6,1.6437461e6
HMMBase.jl,viterbi,6,1,100,50,5,false,false,20,1.511778e6,1.149436e6,93479.00319282232,1.49643925e6,1.51658275e6,1.627709e6,1.48950635e6
HMMBase.jl,forward_backward,6,1,100,50,5,false,false,20,2.305792e6,2.005965e6,154052.1302913144,2.28561175e6,2.31612725e6,2.6148e6,2.33080635e6
HMMBase.jl,baum_welch,4,1,100,50,5,false,false,20,1.19440895e7,1.1625888e7,1.204568421895319e6,1.172987225e7,1.3092569e7,1.5188953e7,1.26072693e7
HMMBase.jl,forward,4,1,100,50,5,false,false,20,929939.5,883059.0,17405.54537126179,914495.5,938200.25,948947.0,925960.25
HMMBase.jl,viterbi,4,1,100,50,5,false,false,20,757178.0,725061.0,68696.03869572256,750299.25,767174.75,992904.0,778724.2
HMMBase.jl,forward_backward,4,1,100,50,5,false,false,20,1.5724085e6,1.293105e6,91404.14638383695,1.56143425e6,1.58639725e6,1.649662e6,1.5546125e6
HMMBase.jl,baum_welch,10,1,100,50,5,false,false,20,4.4381982e7,4.3128737e7,1.867341248866366e6,4.40524375e7,4.56603735e7,5.0030313e7,4.51838287e7
HMMBase.jl,forward,10,1,100,50,5,false,false,20,3.07835e6,2.221054e6,734787.3010424228,2.45172475e6,3.1748105e6,5.651044e6,3.03003855e6
HMMBase.jl,viterbi,10,1,100,50,5,false,false,20,2.031973e6,1.981966e6,683068.1029934273,1.99690675e6,2.07110975e6,5.066257e6,2.23080625e6
HMMBase.jl,forward_backward,10,1,100,50,5,false,false,20,3.5872275e6,3.244124e6,624727.6571509485,3.52828575e6,3.65241125e6,5.954698e6,3.78228555e6
hmmlearn,baum_welch,8,1,100,50,5,false,false,20,2.8052146e7,2.7681564e7,243658.5879632822,2.78663365e7,2.822330625e7,2.8510526e7,2.805080045e7
hmmlearn,forward,8,1,100,50,5,false,false,20,3.149074e6,3.124566e6,19366.810126374778,3.1400015e6,3.1540755e6,3.209853e6,3.15233595e6
hmmlearn,viterbi,8,1,100,50,5,false,false,20,2.443507e6,2.388847e6,98354.39104504744,2.4297045e6,2.482396e6,2.804872e6,2.4764225e6
hmmlearn,forward_backward,8,1,100,50,5,false,false,20,4.4297155e6,4.36005e6,36264.81850455646,4.417745e6,4.447818e6,4.500574e6,4.4260197e6
hmmlearn,baum_welch,2,1,100,50,5,false,false,20,1.9118351e7,1.8845691e7,264769.8091259999,1.89658435e7,1.932057575e7,1.9873735e7,1.916442975e7
hmmlearn,forward,2,1,100,50,5,false,false,20,2.1556545e6,2.110274e6,27929.613547366942,2.12591875e6,2.1617575e6,2.199644e6,2.1508928e6
hmmlearn,viterbi,2,1,100,50,5,false,false,20,1.8057715e6,1.764651e6,48277.04336475159,1.77446775e6,1.82124225e6,1.974593e6,1.8119006e6
hmmlearn,forward_backward,2,1,100,50,5,false,false,20,2.953601e6,2.898184e6,32714.39044952864,2.93163825e6,2.962555e6,3.047143e6,2.9491002e6
hmmlearn,baum_welch,6,1,100,50,5,false,false,20,2.5102865e7,2.4620596e7,162349.03815592892,2.5042043e7,2.51641455e7,2.5335759e7,2.507671645e7
hmmlearn,forward,6,1,100,50,5,false,false,20,2.826229e6,2.768349e6,28828.18315851786,2.80826875e6,2.83331625e6,2.885852e6,2.8216357e6
hmmlearn,viterbi,6,1,100,50,5,false,false,20,2.239632e6,2.228699e6,63064.365291593786,2.2320805e6,2.24477e6,2.513923e6,2.2551495e6
hmmlearn,forward_backward,6,1,100,50,5,false,false,20,3.849211e6,3.803962e6,33542.5684706448,3.840458e6,3.89357625e6,3.920181e6,3.86387885e6
hmmlearn,baum_welch,4,1,100,50,5,false,false,20,2.13718235e7,2.1005346e7,242995.76011681225,2.11890975e7,2.14325045e7,2.2103029e7,2.13647977e7
hmmlearn,forward,4,1,100,50,5,false,false,20,2.4131835e6,2.388433e6,34556.060427376935,2.40117875e6,2.4419805e6,2.524009e6,2.42454155e6
hmmlearn,viterbi,4,1,100,50,5,false,false,20,1.943825e6,1.903686e6,20222.474465372223,1.923306e6,1.95281025e6,1.983018e6,1.93973715e6
hmmlearn,forward_backward,4,1,100,50,5,false,false,20,3.431084e6,3.338657e6,41890.713703510395,3.3705545e6,3.442232e6,3.464801e6,3.4103718e6
hmmlearn,baum_welch,10,1,100,50,5,false,false,20,3.21485175e7,3.1620734e7,508242.438273321,3.181632475e7,3.244999175e7,3.365476e7,3.22455263e7
hmmlearn,forward,10,1,100,50,5,false,false,20,3.3669075e6,3.345366e6,28935.74243274588,3.35484275e6,3.38822275e6,3.448965e6,3.37648815e6
hmmlearn,viterbi,10,1,100,50,5,false,false,20,2.695262e6,2.626188e6,178515.68270399576,2.66318e6,2.705685e6,3.467124e6,2.7278647e6
hmmlearn,forward_backward,10,1,100,50,5,false,false,20,4.848437e6,4.734252e6,107884.20597945938,4.770007e6,4.9042205e6,5.051101e6,4.85877585e6
Empty file removed paper/images/.gitkeep
View file
Empty file.
Loading

0 comments on commit 083c26b

Please sign in to comment.