Compare commits

Home / Compare 96532f0e .. 9e3354a0

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.057-54.057=-0.001
σ0.368-0.370=-0.002
min53.913-53.912=0.001
max55.102-55.109=-0.007
mean417.000-414.000=3.000
σ0.000
min417.000-414.000=3.000
max417.000-414.000=3.000
mean0.138-0.139=-0.001
σ0.005-0.004=0.001
min0.131-0.133=-0.002
max0.145-0.146=-0.001
cache=y
dps=1
iters=10
policies=100
mean43.395-43.394=0.001
σ0.006-0.005=0.001
min43.386-43.386=0.000
max43.404-43.404=-0.000
mean417.000-414.000=3.000
σ0.000
min417.000-414.000=3.000
max417.000-414.000=3.000
mean0.122-0.117=0.005
σ0.004-0.003=0.002
min0.113-0.115=-0.001
max0.129-0.121=0.008

CSV benchmark: ci/scenario_aon

Factorallocated MiBpeak allocated MiBtime_s
fixture=catalog
iterations=5
operation_display=column_set() on area_code (indexed)
profile=catalog
rows=catalog
scenario=ci/scenario_aon
scenario_id=ci/scenario_aon
step_id=column_set.area_code
mean228.284-228.293=-0.009
σ0.022-0.019=0.002
min228.249-228.262=-0.013
max228.303-228.315=-0.012
mean26.600-27.000=-0.400
σ15.421-15.182=0.239
min4.000
max46.000-45.000=1.000
mean0.271-0.274=-0.003
σ0.058-0.056=0.002
min0.211-0.211=0.000
max0.349-0.328=0.021
fixture=catalog
iterations=5
operation_display=column_set() on dpx_subarea_name (no index)
profile=catalog
rows=catalog
scenario=ci/scenario_aon
scenario_id=ci/scenario_aon
step_id=column_set.dpx_subarea_name
mean8.892-8.890=0.002
σ0.004-0.003=0.001
min8.885-8.887=-0.002
max8.896-8.894=0.001
mean1.573-1.571=0.002
σ0.092-0.095=-0.004
min1.412-1.404=0.008
max1.638-1.640=-0.002
mean0.014-0.015=-0.000
σ0.002-0.002=-0.000
min0.013-0.013=0.000
max0.018-0.017=0.000
fixture=catalog
iterations=5
operation_display=load_product()
profile=catalog
rows=catalog
scenario=ci/scenario_aon
scenario_id=ci/scenario_aon
step_id=load_product
mean178.288-178.285=0.004
σ0.301-0.307=-0.005
min177.749-177.736=0.013
max178.434-178.434=-0.000
mean20.838-20.848=-0.010
σ2.265-2.487=-0.222
min18.000-17.000=1.000
max23.016-23.058=-0.042
mean0.153-0.146=0.008
σ0.033-0.022=0.012
min0.123-0.126=-0.002
max0.199-0.177=0.022
fixture=catalog
iterations=5
operation_display=vertical_lookup_row_list() on area + subarea code (both indexed)
profile=catalog
rows=catalog
scenario=ci/scenario_aon
scenario_id=ci/scenario_aon
step_id=vertical_lookup_row_list.area.subarea
mean0.000
σ0.000
min0.000
max0.000
mean0.000
σ0.000
min0.000
max0.000
mean0.001-0.001=0.000
σ0.000-0.000=0.000
min0.001-0.001=-0.000
max0.002-0.001=0.000

CSV benchmark: ci/scenario_synthetic

Factorallocated MiBpeak allocated MiBtime_s
fixture=synthetic/mixed_indexes
iterations=5
operation_display=load_product()
profile=synthetic/mixed_indexes
rows=200000
scenario=ci/scenario_synthetic
scenario_id=ci/scenario_synthetic
step_id=load_product
mean8292.280-8292.278=0.002
σ1.465-1.378=0.088
min8291.599-8291.621=-0.023
max8294.901-8294.742=0.159
mean457.203-457.114=0.088
σ78.789-77.695=1.093
min381.000-384.000=-3.000
max543.000-542.000=1.000
mean4.518-4.928=-0.410
σ1.584-1.501=0.083
min3.319-3.298=0.021
max7.099-6.849=0.250
fixture=synthetic/mixed_indexes
iterations=5
operation_display=table_records()
profile=synthetic/mixed_indexes
rows=200000
scenario=ci/scenario_synthetic
scenario_id=ci/scenario_synthetic
step_id=table_records
mean14174.842-14174.837=0.005
σ4.999-5.072=-0.072
min14172.578-14172.523=0.055
max14183.785-14183.909=-0.124
mean409.400-405.600=3.800
σ190.576-187.505=3.071
min91.000-92.000=-1.000
max598.000-588.000=10.000
mean9.173-9.750=-0.578
σ1.794-1.947=-0.153
min6.418-6.526=-0.108
max10.818-11.406=-0.588
fixture=synthetic/mixed_indexes
iterations=5
operation_display=table_records(), repeat
profile=synthetic/mixed_indexes
rows=200000
scenario=ci/scenario_synthetic
scenario_id=ci/scenario_synthetic
step_id=table_records.repeat
mean27.914-27.914=0.000
σ0.000-0.000=0.000
min27.914-27.914=-0.000
max27.915-27.915=0.000
mean0.119-0.118=0.000
σ0.000-0.000=-0.000
min0.118-0.118=0.001
max0.119-0.119=0.000
mean0.010-0.009=0.000
σ0.002-0.001=0.000
min0.008-0.009=-0.000
max0.012-0.011=0.001
fixture=synthetic/mixed_indexes
iterations=5
operation_display=vertical_lookup_list() on indexed
profile=synthetic/mixed_indexes
rows=200000
scenario=ci/scenario_synthetic
scenario_id=ci/scenario_synthetic
step_id=vertical_lookup_list.indexed
mean19.579-19.579=0.000
σ0.003-0.003=0.000
min19.576-19.576=-0.000
max19.582-19.582=0.000
mean12.127-12.127=-0.000
σ0.001-0.001=0.000
min12.126-12.126=-0.000
max12.128
mean0.021-0.021=0.000
σ0.002-0.002=-0.000
min0.019-0.018=0.000
max0.023-0.024=-0.000
fixture=synthetic/mixed_indexes
iterations=5
operation_display=vertical_lookup_list() on indexed, repeat
profile=synthetic/mixed_indexes
rows=200000
scenario=ci/scenario_synthetic
scenario_id=ci/scenario_synthetic
step_id=vertical_lookup_list.indexed.repeat
mean13.287-13.287=-0.000
σ0.002-0.002=0.000
min13.285
max13.289-13.289=0.000
mean9.280-9.280=-0.000
σ0.001-0.001=0.000
min9.280
max9.281-9.281=0.000
mean0.009-0.009=0.001
σ0.001-0.001=0.000
min0.009-0.008=0.001
max0.010-0.010=0.001

Internal benchmark: pointedIndexFromKvs

Factortime_s
benchmark=pointed_index_from_kvs
iterations=5
rows=100000
mean0.012-0.012=-0.001
σ0.026-0.027=-0.001
min0.000-0.000=-0.000
max0.058-0.061=-0.003

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean1006.857-1006.859=-0.002
σ0.751-0.751=-0.000
min1006.386-1006.385=0.001
max1009.959-1009.961=-0.002
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean1.559-1.555=0.004
σ0.154-0.102=0.052
min1.338-1.472=-0.134
max1.903-1.791=0.113

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.883
σ0.003
min0.882
max0.887
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.307-0.442=-0.135
σ0.119-0.201=-0.083
min0.239-0.332=-0.093
max0.518-0.801=-0.283
name=error_case.art
mean2.339
σ0.003
min2.337
max2.342
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.642-0.921=-0.280
σ0.075-0.166=-0.091
min0.595-0.837=-0.242
max0.774-1.217=-0.443
name=heavy_item.art
mean1.524
σ0.002
min1.522
max1.526
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.380-0.536=-0.156
σ0.033-0.047=-0.014
min0.351-0.509=-0.159
max0.435-0.620=-0.185
name=nested_use.art
mean2.361
σ0.002
min2.360
max2.364
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.603-0.830=-0.227
σ0.038-0.044=-0.006
min0.571-0.795=-0.224
max0.669-0.905=-0.236

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.831-964.831=0.000
σ0.026-0.027=-0.001
min964.813-964.813=0.000
max964.874-964.877=-0.003
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.811-0.751=0.060
σ0.052-0.055=-0.003
min0.737-0.682=0.055
max0.855-0.827=0.028
name=cdr_enhanced_4k_loc.art
mean885.511-885.510=0.001
σ0.008-0.006=0.002
min885.504
max885.522-885.519=0.004
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.732-0.678=0.054
σ0.006-0.037=-0.031
min0.724-0.629=0.095
max0.738-0.729=0.009

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.716-7.716=0.000
σ0.002-0.002=0.000
min7.715
max7.722-7.719=0.002
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean4.286-3.492=0.794
σ0.094-0.090=0.003
min4.131-3.330=0.802
max4.753-3.795=0.958
name=record_render_large_syntax_err.art
mean7.665-7.665=-0.000
σ0.002-0.002=-0.000
min7.664
max7.668-7.669=-0.001
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean3.535-3.474=0.061
σ0.200-0.069=0.132
min3.345-3.334=0.011
max4.359-3.748=0.611
name=record_render_success.art
mean2.291
σ0.002
min2.290
max2.294
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.736-0.728=0.007
σ0.018-0.019=-0.000
min0.712-0.701=0.011
max0.842-0.854=-0.012
name=record_render_syntax_err.art
mean2.166
σ0.002
min2.165
max2.169
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.692-0.688=0.004
σ0.015-0.015=0.000
min0.665-0.663=0.002
max0.759-0.794=-0.035

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean295.427-295.556=-0.129
σ0.186-0.244=-0.058
min295.289-295.393=-0.104
max295.911-296.223=-0.312
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.563-0.557=0.006
σ0.067-0.030=0.037
min0.486-0.514=-0.028
max0.714-0.588=0.125

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean285.395-285.396=-0.001
σ0.027-0.028=-0.001
min285.372-285.365=0.007
max285.429-285.428=0.002
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.393-0.389=0.004
σ0.007-0.011=-0.005
min0.386-0.380=0.006
max0.401-0.408=-0.008
iters=5
new-facts=1
prior-facts=6402
mean1295.846-1295.844=0.003
σ0.084-0.065=0.018
min1295.773-1295.782=-0.009
max1295.974-1295.951=0.023
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean2.411-2.180=0.231
σ0.216-0.150=0.065
min2.187-1.933=0.254
max2.717-2.319=0.398
iters=5
new-facts=6402
prior-facts=1
mean1539.937-1539.937=0.001
σ6.855-6.871=-0.016
min1536.682-1536.658=0.024
max1552.197-1552.225=-0.028
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean2.277-2.262=0.015
σ0.167-0.219=-0.052
min2.033-2.085=-0.052
max2.484-2.633=-0.148
iters=5
new-facts=6402
prior-facts=6402
mean1857.683-1857.682=0.001
σ0.057-0.062=-0.005
min1857.629-1857.612=0.016
max1857.767-1857.759=0.008
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean3.085-3.110=-0.025
σ0.236-0.289=-0.053
min2.867-2.844=0.022
max3.486-3.473=0.013

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.474-1.531=-0.057
σ0.253-0.310=-0.056
min1.154-1.137=0.017
max1.722-1.778=-0.055

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean445.708-445.707=0.000
σ580.933-580.933=0.000
min155.238-155.238=-0.000
max1317.107-1317.107=0.000
mean96.250
σ8.500
min92.000
max109.000
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean0.555-0.536=0.018
σ0.563-0.556=0.006
min0.273-0.258=0.015
max1.399-1.371=0.028
policies=20
mean1552.674-1552.662=0.012
σ0.354-0.340=0.014
min1552.435-1552.436=-0.001
max1553.200-1553.168=0.032
mean920.000
σ0.000
min920.000
max920.000
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean2.631-2.616=0.016
σ0.066-0.060=0.006
min2.577-2.571=0.006
max2.728-2.704=0.024
policies=200
mean15534.728-15534.736=-0.007
σ0.091-0.145=-0.054
min15534.650-15534.605=0.045
max15534.860-15534.914=-0.054
mean9200.000
σ0.000
min9200.000
max9200.000
mean4260.000-4253.000=7.000
σ0.000
min4260.000-4253.000=7.000
max4260.000-4253.000=7.000
mean26.174-26.225=-0.051
σ0.121-0.128=-0.007
min26.069-26.115=-0.046
max26.325-26.409=-0.084