Compare commits

Home / Compare c66e7e32 .. 96e014d2

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.617-54.615=0.001
σ0.351-0.351=0.001
min54.483-54.485=-0.002
max55.616-55.613=0.004
mean415.000-419.000=-4.000
σ0.000
min415.000-419.000=-4.000
max415.000-419.000=-4.000
mean0.141-0.138=0.002
σ0.005-0.006=-0.001
min0.134-0.131=0.003
max0.148-0.146=0.002
cache=y
dps=1
iters=10
policies=100
mean43.788-43.786=0.002
σ0.006-0.006=0.000
min43.781-43.778=0.002
max43.800-43.801=-0.001
mean415.000-419.000=-4.000
σ0.000
min415.000-419.000=-4.000
max415.000-419.000=-4.000
mean0.126-0.123=0.003
σ0.010-0.004=0.006
min0.116-0.119=-0.003
max0.149-0.130=0.019

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.618-36.618=0.001
σ0.002-0.002=-0.000
min36.615-36.615=0.000
max36.620-36.620=0.000
mean4276.625-4116.875=159.750
σ1866.644-1556.523=310.121
min2051.000-2059.000=-8.000
max7545.000-6051.000=1494.000
mean13.029-12.378=0.651
σ1.727-0.260=1.467
min12.071-11.888=0.183
max17.276-12.616=4.660
op=column_set() on indexed, repeat
mean36.617-36.617=-0.000
σ0.002-0.002=-0.000
min36.615
max36.619-36.620=-0.000
mean4276.625-4116.875=159.750
σ1866.644-1556.523=310.121
min2051.000-2059.000=-8.000
max7545.000-6051.000=1494.000
mean12.701-12.222=0.479
σ1.730-0.718=1.012
min11.344-11.070=0.274
max16.603-13.008=3.595
op=table_records(), no indexes
mean41.928-41.928=0.000
σ0.000-0.000=0.000
min41.928
max41.929-41.928=0.001
mean7545.000-6051.000=1494.000
σ0.000
min7545.000-6051.000=1494.000
max7545.000-6051.000=1494.000
mean13.423-13.044=0.380
σ1.045-1.155=-0.109
min12.264-10.632=1.632
max14.939-14.490=0.449
op=table_records(), repeat
mean41.928-41.928=0.000
σ0.002-0.002=0.000
min41.927-41.927=0.000
max41.932-41.931=0.001
mean4276.625-4116.875=159.750
σ1866.644-1556.523=310.121
min2051.000-2059.000=-8.000
max7545.000-6051.000=1494.000
mean12.072-12.385=-0.314
σ2.088-1.911=0.178
min10.194-10.583=-0.389
max15.588-14.730=0.858
op=vertical_lookup_list() on indexed, after table_records()
mean27.605-27.605=-0.000
σ0.002-0.003=-0.001
min27.603
max27.607-27.611=-0.004
mean4276.625-4116.875=159.750
σ1866.644-1556.523=310.121
min2051.000-2059.000=-8.000
max7545.000-6051.000=1494.000
mean8.848-8.217=0.630
σ1.229-0.360=0.869
min7.817-7.704=0.113
max11.745-8.596=3.149
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.400
σ0.002
min26.399
max26.403
mean4276.625-4116.875=159.750
σ1866.644-1556.523=310.121
min2051.000-2059.000=-8.000
max7545.000-6051.000=1494.000
mean6.837-6.300=0.537
σ1.179-0.186=0.993
min6.074-5.995=0.079
max9.676-6.472=3.204
op=vertical_lookup_row_list() on 2 indexed
mean25.666-25.666=0.000
σ0.002-0.002=-0.000
min25.665-25.664=0.000
max25.670-25.670=0.000
mean4276.625-4116.875=159.750
σ1866.644-1556.523=310.121
min2051.000-2059.000=-8.000
max7545.000-6051.000=1494.000
mean5.716-5.395=0.321
σ0.496-0.115=0.381
min5.306-5.130=0.176
max6.889-5.508=1.382
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.968-25.968=0.000
σ0.002-0.002=0.000
min25.967-25.967=-0.000
max25.971
mean4276.625-4116.875=159.750
σ1866.644-1556.523=310.121
min2051.000-2059.000=-8.000
max7545.000-6051.000=1494.000
mean5.535-5.218=0.317
σ0.496-0.059=0.437
min5.205-5.152=0.052
max6.735-5.349=1.386

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean2524.593-2524.594=-0.000
σ0.000-0.000=-0.000
min2524.593-2524.594=-0.001
max2524.594-2524.594=-0.001
mean7545.000-6051.000=1494.000
σ0.000
min7545.000-6051.000=1494.000
max7545.000-6051.000=1494.000
mean1104.951-1565.822=-460.872
σ159.812-808.338=-648.526
min1042.636-1052.079=-9.444
max1500.215-3039.454=-1539.239
op=load_product(), with indexes
mean8744.559-8744.559=-0.001
σ0.032-0.033=-0.000
min8744.545-8744.545=-0.001
max8744.639-8744.640=-0.002
mean3807.500-3846.125=-38.625
σ1585.551-1625.606=-40.055
min2051.000-2059.000=-8.000
max5944.000-6051.000=-107.000
mean4651.472-4454.424=197.048
σ1543.313-1522.523=20.790
min3527.403-3558.200=-30.797
max7403.603-7764.050=-360.447

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense
mean62.078-62.078=0.000
σ0.002-0.002=-0.000
min62.076-62.075=0.001
max62.080
mean4276.625-4116.875=159.750
σ1866.644-1556.523=310.121
min2051.000-2059.000=-8.000
max7545.000-6051.000=1494.000
mean86.034-78.029=8.005
σ12.805-5.575=7.230
min72.983-70.680=2.303
max108.036-84.922=23.114
op=vertical_lookup_list() on unindexed missing Int
mean40.944-40.944=0.000
σ0.001-0.000=0.000
min40.943-40.944=-0.000
max40.945-40.945=0.000
mean7545.000-6051.000=1494.000
σ0.000
min7545.000-6051.000=1494.000
max7545.000-6051.000=1494.000
mean207.868-96.407=111.461
σ311.258-9.067=302.191
min82.789-83.675=-0.886
max977.932-105.891=872.041
op=vertical_lookup_row_list() on unindexed small unique Int
mean66.686-66.686=-0.000
σ0.000-0.000=-0.000
min66.685
max66.687
mean7545.000-6051.000=1494.000
σ0.000
min7545.000-6051.000=1494.000
max7545.000-6051.000=1494.000
mean147.637-223.452=-75.816
σ18.273-225.268=-206.995
min116.942-124.609=-7.668
max166.728-779.999=-613.272

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean11656.833-11656.833=0.000
σ4.318-4.318=-0.000
min11655.305-11655.305=0.000
max11667.521
mean4276.625-4116.875=159.750
σ1866.644-1556.523=310.121
min2051.000-2059.000=-8.000
max7545.000-6051.000=1494.000
mean6537.071-6303.223=233.849
σ1229.132-1295.185=-66.054
min4957.566-4899.198=58.368
max8282.784-8209.832=72.952
op=vertical_lookup_list() on indexed
mean11560.068-11560.069=-0.001
σ0.001-0.002=-0.001
min11560.066-11560.066=-0.001
max11560.069-11560.073=-0.004
mean7545.000-6051.000=1494.000
σ0.000
min7545.000-6051.000=1494.000
max7545.000-6051.000=1494.000
mean5221.230-6780.384=-1559.154
σ1039.306-1552.700=-513.393
min4362.176-4434.404=-72.228
max6721.288-8202.408=-1481.120

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean172.587-172.587=-0.000
σ0.023-0.027=-0.003
min172.562-172.558=0.004
max172.622-172.624=-0.002
mean2051.000-2059.000=-8.000
σ0.000
min2051.000-2059.000=-8.000
max2051.000-2059.000=-8.000
mean363.356-361.810=1.546
σ44.082-43.797=0.285
min320.251-326.398=-6.147
max445.204-454.610=-9.405
op=pointedIndexFromKvs
rows=400000
mean357.371-357.367=0.003
σ0.047-0.046=0.001
min357.321
max357.432-357.432=0.001
mean2051.000-2059.000=-8.000
σ0.000
min2051.000-2059.000=-8.000
max2051.000-2059.000=-8.000
mean758.377-762.021=-3.643
σ55.479-28.704=26.775
min635.185-698.560=-63.375
max809.184-788.455=20.729

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean1008.183-1008.186=-0.004
σ0.899-0.869=0.030
min1007.693-1007.684=0.008
max1011.920-1011.778=0.141
mean1037.150-1040.850=-3.700
σ304.391-302.915=1.476
min544.000-559.000=-15.000
max1465.000-1467.000=-2.000
mean1.545-1.536=0.009
σ0.136-0.085=0.051
min1.435-1.398=0.037
max2.039-1.750=0.289

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.884-0.883=0.001
σ0.002-0.002=-0.000
min0.881
max0.886
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.426-0.445=-0.018
σ0.098-0.156=-0.058
min0.364-0.366=-0.002
max0.598-0.723=-0.125
name=error_case.art
mean2.334-2.333=0.001
σ0.003-0.003=0.000
min2.332
max2.339-2.337=0.002
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.883-0.914=-0.031
σ0.077-0.137=-0.059
min0.834-0.819=0.015
max1.019-1.155=-0.136
name=heavy_item.art
mean1.522-1.521=0.001
σ0.002-0.002=0.001
min1.520
max1.526-1.524=0.002
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.534-0.545=-0.011
σ0.053-0.048=0.005
min0.493-0.511=-0.017
max0.624-0.625=-0.001
name=nested_use.art
mean2.356-2.355=0.000
σ0.002-0.002=0.001
min2.355
max2.360-2.358=0.001
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.837-0.848=-0.011
σ0.041-0.036=0.005
min0.809-0.827=-0.018
max0.908-0.911=-0.004

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean963.548-963.543=0.005
σ0.032-0.025=0.007
min963.528-963.523=0.005
max963.605-963.583=0.022
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.801-0.813=-0.012
σ0.031-0.066=-0.035
min0.769-0.704=0.065
max0.839-0.866=-0.027
name=cdr_enhanced_4k_loc.art
mean884.854-884.853=0.001
σ0.009-0.005=0.004
min884.847
max884.868-884.857=0.010
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.729-0.690=0.039
σ0.033-0.034=-0.001
min0.671-0.662=0.009
max0.747-0.745=0.002

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.706-7.706=-0.000
σ0.002-0.002=-0.000
min7.705
max7.709
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean4.275-4.307=-0.032
σ0.088-0.081=0.008
min4.130-4.165=-0.034
max4.567-4.607=-0.039
name=record_render_large_syntax_err.art
mean7.655-7.655=0.000
σ0.002-0.002=0.000
min7.654
max7.659
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean4.256-4.287=-0.031
σ0.068-0.115=-0.047
min4.121-4.118=0.003
max4.486-4.866=-0.380
name=record_render_success.art
mean2.288
σ0.002
min2.287
max2.291
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.737-0.765=-0.028
σ0.017-0.017=0.001
min0.711-0.734=-0.023
max0.861-0.868=-0.007
name=record_render_syntax_err.art
mean2.163
σ0.002
min2.163
max2.166
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.702-0.724=-0.021
σ0.012-0.025=-0.013
min0.676-0.683=-0.007
max0.761-0.911=-0.151

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean297.302-297.460=-0.159
σ0.258-0.236=0.021
min297.106-297.294=-0.188
max297.980-298.109=-0.129
mean2051.000-2059.000=-8.000
σ0.000
min2051.000-2059.000=-8.000
max2051.000-2059.000=-8.000
mean0.558-0.572=-0.014
σ0.031-0.028=0.003
min0.512-0.510=0.003
max0.590-0.608=-0.018

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean290.084-290.086=-0.002
σ0.037-0.033=0.004
min290.031-290.039=-0.009
max290.122-290.118=0.004
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.401-0.399=0.002
σ0.006-0.003=0.003
min0.396-0.396=0.000
max0.412-0.404=0.007
iters=5
new-facts=1
prior-facts=6402
mean1300.002-1299.943=0.060
σ0.091-0.117=-0.025
min1299.900-1299.806=0.093
max1300.128-1300.068=0.060
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean2.047-1.869=0.178
σ0.058-0.073=-0.015
min1.987-1.807=0.180
max2.131-1.973=0.157
iters=5
new-facts=6402
prior-facts=1
mean1542.952-1542.924=0.028
σ3.991-4.025=-0.035
min1540.970-1540.892=0.078
max1550.083-1550.118=-0.036
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean2.272-2.193=0.079
σ0.218-0.120=0.099
min2.109-2.032=0.077
max2.654-2.365=0.289
iters=5
new-facts=6402
prior-facts=6402
mean1861.943-1861.955=-0.012
σ0.055-0.045=0.010
min1861.869-1861.905=-0.036
max1862.010-1862.013=-0.002
mean1728.600-1730.600=-2.000
σ234.317-237.341=-3.024
min1465.000-1467.000=-2.000
max2051.000-2059.000=-8.000
mean2.938-2.881=0.057
σ0.099-0.083=0.016
min2.813-2.805=0.008
max3.071-3.015=0.056

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.638-1.751=-0.113
σ0.080-0.223=-0.143
min1.543-1.367=0.176
max1.753-1.911=-0.159

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean446.186-446.147=0.039
σ577.139-577.064=0.075
min157.614-157.613=0.001
max1311.895-1311.743=0.152
mean96.250
σ8.500
min92.000
max109.000
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean0.547-0.543=0.005
σ0.570-0.560=0.010
min0.261-0.259=0.001
max1.402-1.383=0.020
policies=20
mean1576.420-1576.404=0.016
σ0.345-0.345=-0.001
min1576.185-1576.170=0.016
max1576.932-1576.917=0.015
mean920.000
σ0.000
min920.000
max920.000
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean2.622-2.614=0.008
σ0.046-0.013=0.033
min2.592-2.603=-0.010
max2.690-2.630=0.060
policies=200
mean15772.316-15772.332=-0.016
σ0.084-0.112=-0.028
min15772.252-15772.219=0.033
max15772.431-15772.485=-0.053
mean9200.000
σ0.000
min9200.000
max9200.000
mean1465.000-1467.000=-2.000
σ0.000
min1465.000-1467.000=-2.000
max1465.000-1467.000=-2.000
mean26.372-26.426=-0.054
σ0.017-0.064=-0.046
min26.358-26.371=-0.012
max26.395-26.517=-0.122