Compare commits

Home / Compare c991b0a9 .. d5c7f691

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.082-5.301=-0.219
σ0.000
min5.082-5.301=-0.219
max5.082-5.301=-0.219

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean4.988-5.176=-0.188
σ0.000
min4.988-5.176=-0.188
max4.988-5.176=-0.188

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean5.395-5.676=-0.281
σ0.000
min5.395-5.676=-0.281
max5.395-5.676=-0.281

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.633-446.883=-0.250
σ0.000
min446.633-446.883=-0.250
max446.633-446.883=-0.250

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.238-5.488=-0.250
σ0.000
min5.238-5.488=-0.250
max5.238-5.488=-0.250

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.395-6.645=-0.250
σ0.000
min6.395-6.645=-0.250
max6.395-6.645=-0.250

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.717-54.772=-0.054
σ0.366-0.363=0.004
min54.580-54.615=-0.035
max55.759-55.802=-0.043
mean420.000-426.000=-6.000
σ0.000
min420.000-426.000=-6.000
max420.000-426.000=-6.000
mean0.140-0.140=0.000
σ0.004-0.006=-0.002
min0.133-0.134=-0.000
max0.145-0.150=-0.004
cache=y
dps=1
iters=10
policies=100
mean43.777-43.779=-0.002
σ0.008-0.008=0.001
min43.767-43.768=-0.001
max43.793-43.792=0.001
mean420.000-426.000=-6.000
σ0.000
min420.000-426.000=-6.000
max420.000-426.000=-6.000
mean0.123-0.123=-0.000
σ0.006-0.003=0.003
min0.116-0.121=-0.005
max0.133-0.130=0.003

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.604-36.617=-0.013
σ0.000-0.000=-0.000
min36.604-36.617=-0.013
max36.605-36.618=-0.013
mean4225.875-4587.750=-361.875
σ1967.775-2245.180=-277.405
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean12.344-151.845=-139.502
σ0.345-395.607=-395.262
min12.036-11.755=0.281
max13.139-1130.923=-1117.784
op=column_set() on indexed, repeat
mean36.605-36.617=-0.013
σ0.000-0.000=-0.000
min36.604-36.617=-0.013
max36.605-36.618=-0.013
mean4451.625-4587.750=-136.125
σ2236.584-2245.180=-8.596
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean13.016-13.948=-0.932
σ0.269-4.397=-4.128
min12.803-11.463=1.340
max13.643-24.729=-11.085
op=table_records(), no indexes
mean41.939-41.948=-0.008
σ0.002-0.002=0.000
min41.938-41.946=-0.009
max41.942-41.950=-0.008
mean7432.000-7373.000=59.000
σ0.000
min7432.000-7373.000=59.000
max7432.000-7373.000=59.000
mean11.140-10.684=0.456
σ0.155-0.117=0.038
min10.837-10.490=0.347
max11.323-10.846=0.477
op=table_records(), repeat
mean41.925-41.934=-0.008
σ0.001-0.001=-0.000
min41.924-41.933=-0.008
max41.928-41.937=-0.008
mean4225.875-4179.625=46.250
σ1967.775-1942.967=24.808
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean11.568-11.440=0.128
σ0.934-2.053=-1.119
min11.050-10.506=0.544
max13.814-16.508=-2.694
op=vertical_lookup_list() on indexed, after table_records()
mean27.633-27.655=-0.022
σ0.137-0.137=0.000
min27.584-27.606=-0.022
max27.971-27.993=-0.022
mean4225.875-4179.625=46.250
σ1967.775-1942.967=24.808
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean8.883-9.126=-0.243
σ0.651-1.147=-0.496
min8.049-8.271=-0.223
max10.140-11.895=-1.755
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.384-26.406=-0.022
σ0.000-0.000=0.000
min26.384-26.406=-0.022
max26.384-26.406=-0.022
mean4225.875-4179.625=46.250
σ1967.775-1942.967=24.808
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean6.729-7.154=-0.425
σ0.512-0.914=-0.401
min5.980-6.632=-0.652
max7.736-9.401=-1.665
op=vertical_lookup_row_list() on 2 indexed
mean25.662-25.681=-0.018
σ0.004-0.003=0.001
min25.661-25.680=-0.019
max25.673-25.688=-0.015
mean4451.625-4587.750=-136.125
σ2236.584-2245.180=-8.596
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean6.426-6.341=0.085
σ1.018-0.992=0.026
min5.897-5.811=0.086
max8.927-8.767=0.160
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.961-25.984=-0.024
σ0.001-0.000=0.000
min25.961-25.984=-0.024
max25.962-25.985=-0.023
mean4451.625-4587.750=-136.125
σ2236.584-2245.180=-8.596
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean6.279-6.166=0.113
σ0.988-1.065=-0.077
min5.871-5.611=0.260
max8.722-8.781=-0.059

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.608-1021.627=-0.019
σ0.002
min1021.606-1021.625=-0.019
max1021.611-1021.629=-0.019
mean7432.000-7373.000=59.000
σ0.000
min7432.000-7373.000=59.000
max7432.000-7373.000=59.000
mean361.710-370.483=-8.773
σ42.165-90.053=-47.888
min343.300-335.870=7.430
max465.978-593.306=-127.328
op=load_product(), with indexes
mean3495.855-3495.878=-0.024
σ0.454-0.468=-0.014
min3495.694-3495.713=-0.019
max3496.978-3497.037=-0.059
mean3794.000-3756.125=37.875
σ2326.905-2297.667=29.239
min720.000
max7432.000-7373.000=59.000
mean1983.144-1718.451=264.694
σ907.456-673.280=234.176
min1399.766-1348.893=50.873
max3732.152-2888.162=843.990

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense, minor warmup
mean107.841-107.854=-0.013
σ0.000-0.000=-0.000
min107.840-107.853=-0.013
max107.841-107.855=-0.013
mean4225.875-4407.000=-181.125
σ1967.775-2214.272=-246.496
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean131.191-196.177=-64.986
σ4.648-161.029=-156.381
min126.084-125.228=0.856
max139.028-591.602=-452.574
op=column_set() on unindexed, dense, repeat
mean62.068-62.081=-0.013
σ0.001-0.000=0.001
min62.065-62.081=-0.017
max62.068-62.082=-0.013
mean4225.875-4407.000=-181.125
σ1967.775-2214.272=-246.496
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean42.820-42.344=0.475
σ1.282-1.369=-0.088
min41.458-40.764=0.694
max45.230-43.992=1.238
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.926-40.948=-0.022
σ0.002
min40.925-40.947=-0.022
max40.928-40.950=-0.022
mean7432.000-7373.000=59.000
σ0.000
min7432.000-7373.000=59.000
max7432.000-7373.000=59.000
mean22.521-23.189=-0.668
σ1.148-2.869=-1.722
min21.840-21.461=0.379
max25.220-29.396=-4.177
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.294-71.308=-0.014
σ0.002-0.002=-0.000
min71.292-71.307=-0.014
max71.296-71.311=-0.014
mean7432.000-7373.000=59.000
σ0.000
min7432.000-7373.000=59.000
max7432.000-7373.000=59.000
mean155.028-57.189=97.839
σ279.431-2.677=276.754
min55.794-55.952=-0.158
max846.584-63.753=782.830

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.842-4646.850=-0.008
σ0.013-0.013=-0.000
min4646.837-4646.845=-0.008
max4646.875-4646.883=-0.008
mean4225.875-4179.625=46.250
σ1967.775-1942.967=24.808
min1482.000-1487.000=-5.000
max7432.000-7373.000=59.000
mean3185.995-3238.771=-52.775
σ535.213-817.732=-282.520
min2522.672-2448.643=74.028
max3839.866-4609.738=-769.872
op=vertical_lookup_list() on indexed
mean1973.829-1973.851=-0.022
σ0.002-0.002=-0.000
min1973.827-1973.849=-0.022
max1973.831-1973.853=-0.022
mean7432.000-7373.000=59.000
σ0.000
min7432.000-7373.000=59.000
max7432.000-7373.000=59.000
mean1337.097-999.209=337.888
σ998.492-202.400=796.092
min932.710-908.142=24.568
max3801.423-1488.522=2312.901
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.083-905.105=-0.022
σ0.002-0.002=0.000
min905.080-905.102=-0.022
max905.086-905.108=-0.022
mean7432.000-7373.000=59.000
σ0.000
min7432.000-7373.000=59.000
max7432.000-7373.000=59.000
mean425.146-501.967=-76.820
σ42.904-292.703=-249.799
min399.832-376.717=23.115
max530.776-1217.797=-687.020

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144
σ0.001
min148.144
max148.148
mean475.625-476.000=-0.375
σ0.518-0.000=0.518
min475.000-476.000=-1.000
max476.000
mean385.039-390.109=-5.070
σ43.780-32.226=11.555
min292.429-344.738=-52.309
max423.370-438.629=-15.259
op=pointedIndexFromKvs
rows=400000
mean308.493
σ0.001
min308.493
max308.497
mean563.750-563.125=0.625
σ0.707-1.553=-0.846
min562.000
max564.000-565.000=-1.000
mean727.748-716.314=11.434
σ84.646-78.757=5.890
min602.928-605.749=-2.821
max836.334-813.270=23.064

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean964.543-963.986=0.557
σ1.822-1.822=0.000
min963.840-963.283=0.557
max972.248-971.689=0.559
mean1028.800-1043.650=-14.850
σ301.074-306.341=-5.268
min549.000-594.000=-45.000
max1456.000-1493.000=-37.000
mean1.286-1.271=0.015
σ0.149-0.149=0.000
min1.186-1.168=0.018
max1.837-1.814=0.022

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913-0.913=0.001
σ0.020-0.020=-0.000
min0.903-0.903=0.001
max0.949-0.949=0.000
mean402.000
σ0.000
min402.000
max402.000
mean0.412-0.405=0.007
σ0.553-0.537=0.017
min0.132-0.133=-0.002
max1.399-1.362=0.038
name=error_case.art
mean2.346-2.340=0.006
σ0.012
min2.340-2.334=0.006
max2.368-2.362=0.006
mean402.000
σ0.000
min402.000
max402.000
mean0.473-0.476=-0.002
σ0.327-0.322=0.005
min0.317-0.317=-0.000
max1.058-1.051=0.007
name=heavy_item.art
mean1.523-1.523=0.001
σ0.002-0.002=-0.000
min1.522-1.521=0.001
max1.526-1.525=0.001
mean402.000
σ0.000
min402.000
max402.000
mean0.220-0.215=0.006
σ0.021-0.021=-0.000
min0.197-0.202=-0.005
max0.255-0.252=0.003
name=nested_use.art
mean2.364-2.358=0.006
σ0.002
min2.363-2.357=0.006
max2.367-2.361=0.006
mean402.000
σ0.000
min402.000
max402.000
mean0.333-0.330=0.003
σ0.021-0.015=0.006
min0.312-0.316=-0.004
max0.368-0.355=0.013

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.979-964.971=0.008
σ0.009-0.011=-0.001
min964.974-964.965=0.008
max964.996-964.989=0.006
mean431.400-431.800=-0.400
σ0.894-1.789=-0.894
min431.000
max433.000-435.000=-2.000
mean0.570-0.575=-0.005
σ0.032-0.031=0.001
min0.527-0.537=-0.010
max0.609-0.609=0.000
name=cdr_enhanced_4k_loc.art
mean886.001-885.898=0.103
σ0.002-0.002=-0.000
min886.000-885.897=0.103
max886.004-885.901=0.103
mean440.000-443.000=-3.000
σ0.000
min440.000-443.000=-3.000
max440.000-443.000=-3.000
mean0.473-0.446=0.028
σ0.042-0.074=-0.032
min0.419-0.338=0.081
max0.514-0.516=-0.002

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.719-7.710=0.009
σ0.002
min7.718-7.709=0.009
max7.722-7.713=0.009
mean440.000-443.000=-3.000
σ0.000
min440.000-443.000=-3.000
max440.000-443.000=-3.000
mean1.962-1.911=0.051
σ0.071-0.083=-0.013
min1.813-1.765=0.048
max2.224-2.257=-0.033
name=record_render_large_syntax_err.art
mean7.667-7.658=0.009
σ0.002
min7.666-7.657=0.009
max7.670-7.661=0.009
mean440.000-443.000=-3.000
σ0.000
min440.000-443.000=-3.000
max440.000-443.000=-3.000
mean1.957-1.908=0.048
σ0.071-0.065=0.006
min1.831-1.766=0.065
max2.257-2.133=0.124
name=record_render_success.art
mean2.290-2.290=0.001
σ0.002
min2.290-2.289=0.001
max2.294-2.293=0.001
mean440.000-443.000=-3.000
σ0.000
min440.000-443.000=-3.000
max440.000-443.000=-3.000
mean0.301-0.300=0.001
σ0.008-0.009=-0.001
min0.280-0.287=-0.007
max0.344-0.359=-0.015
name=record_render_syntax_err.art
mean2.166-2.165=0.001
σ0.002
min2.165-2.164=0.001
max2.169-2.168=0.001
mean440.000-443.000=-3.000
σ0.000
min440.000-443.000=-3.000
max440.000-443.000=-3.000
mean0.293-0.289=0.004
σ0.010-0.007=0.003
min0.279-0.275=0.004
max0.337-0.320=0.017

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.985-11.489=0.496
σ12.533-11.676=0.856
min0.574-0.574=-0.000
max110.921-102.949=7.972
mean469.506-453.462=16.044
σ20.153-16.451=3.702
min408.000-410.000=-2.000
max484.000-472.000=12.000
mean0.027-0.026=0.001
σ0.031-0.029=0.002
min0.001-0.001=-0.000
max0.270-0.270=-0.000

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean278.789-278.630=0.159
σ1.053-1.095=-0.042
min278.294-278.159=0.135
max281.779-281.736=0.043
mean441.300-440.900=0.400
σ15.727-16.031=-0.304
min417.000-414.000=3.000
max467.000-465.000=2.000
mean0.565-0.583=-0.018
σ0.042-0.017=0.025
min0.474-0.544=-0.071
max0.619-0.613=0.006

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean169.257-173.738=-4.481
σ0.000-0.000=0.000
min169.257-173.738=-4.481
max169.257-173.738=-4.481
mean874.000-841.000=33.000
σ0.000-2.138=-2.138
min874.000-839.000=35.000
max874.000-843.000=31.000
mean223.535-205.939=17.596
σ23.658-22.008=1.649
min190.524-181.115=9.409
max242.867-239.725=3.142
phase=macro expansion
mean5.900-6.130=-0.230
σ0.000
min5.900-6.130=-0.230
max5.900-6.130=-0.230
mean874.000-840.500=33.500
σ0.000-2.070=-2.070
min874.000-839.000=35.000
max874.000-843.000=31.000
mean7.431-9.116=-1.685
σ1.604-1.058=0.546
min5.851-7.345=-1.494
max8.960-10.294=-1.335
phase=parser
mean1042.134-1042.431=-0.297
σ0.000-0.000=-0.000
min1042.133-1042.430=-0.297
max1042.134-1042.431=-0.297
mean874.000-840.500=33.500
σ0.000-2.070=-2.070
min874.000-839.000=35.000
max874.000-843.000=31.000
mean511.219-553.462=-42.243
σ110.165-69.241=40.924
min374.372-420.662=-46.291
max592.877-607.099=-14.222
phase=processProdDef
mean2243.717-2243.978=-0.261
σ0.000
min2243.717-2243.978=-0.261
max2243.717-2243.978=-0.261
mean874.000-841.000=33.000
σ0.000-2.138=-2.138
min874.000-839.000=35.000
max874.000-843.000=31.000
mean2675.174-2512.858=162.315
σ335.713-214.631=121.082
min2308.432-2147.045=161.387
max3207.502-2870.586=336.917
phase=renaming
mean21.436-21.671=-0.235
σ0.000
min21.436-21.671=-0.235
max21.436-21.671=-0.235
mean874.000-840.500=33.500
σ0.000-2.070=-2.070
min874.000-839.000=35.000
max874.000-843.000=31.000
mean37.418-36.630=0.788
σ5.015-6.146=-1.131
min28.922-26.523=2.399
max41.869-40.521=1.348
phase=type checker
mean958.596-989.755=-31.159
σ0.047-0.066=-0.019
min958.573-989.658=-31.085
max958.698-989.845=-31.147
mean874.000-840.500=33.500
σ0.000-2.070=-2.070
min874.000-839.000=35.000
max874.000-843.000=31.000
mean659.427-705.746=-46.319
σ39.647-75.037=-35.390
min636.684-640.137=-3.453
max754.014-879.385=-125.371

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean282.589-282.617=-0.027
σ0.036-0.034=0.002
min282.537-282.568=-0.031
max282.623-282.653=-0.029
mean441.000-445.400=-4.400
σ2.739-0.548=2.191
min439.000-445.000=-6.000
max444.000-446.000=-2.000
mean0.381-0.388=-0.007
σ0.009-0.011=-0.002
min0.375-0.375=-0.001
max0.393-0.397=-0.004
iters=5
new-facts=1
prior-facts=6402
mean1267.800-1267.302=0.497
σ0.091-0.080=0.011
min1267.692-1267.230=0.462
max1267.940-1267.430=0.510
mean1076.800-1115.200=-38.400
σ185.198-170.946=14.251
min837.000-898.000=-61.000
max1319.000-1329.000=-10.000
mean2.064-1.891=0.173
σ0.235-0.102=0.132
min1.786-1.785=0.002
max2.387-2.061=0.326
iters=5
new-facts=6402
prior-facts=1
mean1506.813-1506.335=0.477
σ2.960-2.945=0.015
min1505.288-1504.837=0.451
max1512.099-1511.595=0.504
mean630.800-633.600=-2.800
σ93.377-88.124=5.253
min520.000-499.000=21.000
max754.000-729.000=25.000
mean2.330-2.251=0.079
σ0.201-0.264=-0.063
min2.078-2.042=0.036
max2.622-2.660=-0.038
iters=5
new-facts=6402
prior-facts=6402
mean1829.797-1829.323=0.474
σ0.056-0.049=0.006
min1829.733-1829.258=0.475
max1829.855-1829.375=0.480
mean1726.600-1757.400=-30.800
σ169.100-222.416=-53.316
min1489.000-1466.000=23.000
max1885.000-1970.000=-85.000
mean3.005-3.125=-0.120
σ0.290-0.277=0.013
min2.832-2.806=0.026
max3.521-3.494=0.026

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.648-1.554=0.094
σ0.154-0.037=0.116
min1.521-1.513=0.008
max1.898-1.608=0.290

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean405.597-409.435=-3.838
σ509.661-518.108=-8.447
min150.761-150.378=0.383
max1170.088-1186.596=-16.508
mean101.750-99.750=2.000
σ7.500
min98.000-96.000=2.000
max113.000-111.000=2.000
mean441.000-443.000=-2.000
σ0.000
min441.000-443.000=-2.000
max441.000-443.000=-2.000
mean0.508-0.509=-0.001
σ0.503-0.487=0.017
min0.252-0.255=-0.003
max1.263-1.238=0.025
policies=20
mean1507.805-1503.936=3.869
σ0.269-0.255=0.015
min1507.631-1503.766=3.865
max1508.205-1504.313=3.891
mean980.000-960.000=20.000
σ0.000
min980.000-960.000=20.000
max980.000-960.000=20.000
mean449.750-448.000=1.750
σ11.087-5.774=5.313
min441.000-443.000=-2.000
max466.000-453.000=13.000
mean2.502-2.488=0.014
σ0.009-0.012=-0.003
min2.495-2.474=0.021
max2.514-2.503=0.010
policies=200
mean15086.750-15047.881=38.869
σ0.095-0.100=-0.005
min15086.659-15047.809=38.850
max15086.843-15048.029=38.814
mean9800.000-9600.000=200.000
σ0.000
min9800.000-9600.000=200.000
max9800.000-9600.000=200.000
mean498.000-500.750=-2.750
σ6.683-6.131=0.553
min492.000-493.000=-1.000
max507.000
mean25.205-25.210=-0.005
σ0.160-0.067=0.093
min25.039-25.152=-0.113
max25.421-25.285=0.136