Compare commits

Home / Compare a1e57049 .. d8c4c084

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.082-5.301=-0.219
σ0.000
min5.082-5.301=-0.219
max5.082-5.301=-0.219

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean4.988-5.176=-0.188
σ0.000
min4.988-5.176=-0.188
max4.988-5.176=-0.188

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean5.395-5.676=-0.281
σ0.000
min5.395-5.676=-0.281
max5.395-5.676=-0.281

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.633-446.883=-0.250
σ0.000
min446.633-446.883=-0.250
max446.633-446.883=-0.250

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.238-5.488=-0.250
σ0.000
min5.238-5.488=-0.250
max5.238-5.488=-0.250

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.395-6.645=-0.250
σ0.000
min6.395-6.645=-0.250
max6.395-6.645=-0.250

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.718-54.776=-0.058
σ0.367-0.364=0.003
min54.568-54.632=-0.064
max55.760-55.810=-0.050
mean416.300-417.000=-0.700
σ0.483-0.000=0.483
min416.000-417.000=-1.000
max417.000
mean0.139-0.143=-0.004
σ0.005-0.008=-0.003
min0.135-0.138=-0.003
max0.148-0.163=-0.015
cache=y
dps=1
iters=10
policies=100
mean43.777-43.779=-0.003
σ0.009-0.007=0.002
min43.759-43.768=-0.008
max43.795-43.791=0.003
mean417.000
σ0.000
min417.000
max417.000
mean0.123-0.139=-0.016
σ0.004-0.011=-0.006
min0.119-0.120=-0.001
max0.130-0.151=-0.021

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.604-36.617=-0.013
σ0.000-0.000=0.000
min36.604-36.617=-0.013
max36.605-36.618=-0.013
mean4591.125-4590.375=0.750
σ2248.114-2249.453=-1.339
min1487.000-1486.000=1.000
max7385.000
mean120.517-149.392=-28.875
σ306.172-385.929=-79.757
min11.840-11.912=-0.072
max878.253-1104.505=-226.252
op=column_set() on indexed, repeat
mean36.604-36.617=-0.013
σ0.000-0.000=0.000
min36.604-36.617=-0.013
max36.605-36.618=-0.013
mean4591.125-4590.375=0.750
σ2248.114-2249.453=-1.339
min1487.000-1486.000=1.000
max7385.000
mean13.749-14.601=-0.852
σ3.365-4.801=-1.436
min11.574-11.434=0.140
max21.994-25.272=-3.277
op=table_records(), no indexes
mean41.940-41.948=-0.008
σ0.002-0.002=-0.000
min41.938-41.946=-0.008
max41.942-41.951=-0.009
mean7385.000
σ0.000
min7385.000
max7385.000
mean11.042-11.274=-0.232
σ0.113-1.134=-1.020
min10.938-10.631=0.307
max11.271-14.059=-2.788
op=table_records(), repeat
mean41.925-41.933=-0.008
σ0.001-0.000=0.001
min41.924-41.933=-0.009
max41.928-41.934=-0.005
mean4182.250-4180.250=2.000
σ1944.318-1945.738=-1.420
min1487.000-1486.000=1.000
max7385.000
mean11.658-11.825=-0.168
σ1.722-1.680=0.042
min10.765-10.885=-0.120
max15.877-15.939=-0.062
op=vertical_lookup_list() on indexed, after table_records()
mean27.633-27.655=-0.023
σ0.137-0.136=0.000
min27.584-27.606=-0.022
max27.971-27.993=-0.022
mean4182.250-4180.250=2.000
σ1944.318-1945.738=-1.420
min1487.000-1486.000=1.000
max7385.000
mean9.041-8.953=0.088
σ1.289-1.328=-0.039
min8.298-7.670=0.628
max12.196-12.104=0.092
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.384-26.406=-0.022
σ0.000-0.000=0.000
min26.384-26.406=-0.022
max26.384-26.406=-0.022
mean4182.250-4180.250=2.000
σ1944.318-1945.738=-1.420
min1487.000-1486.000=1.000
max7385.000
mean7.054-6.750=0.303
σ0.988-1.081=-0.094
min6.210-6.146=0.064
max9.422-9.397=0.025
op=vertical_lookup_row_list() on 2 indexed
mean25.662-25.681=-0.018
σ0.004-0.003=0.001
min25.661-25.680=-0.019
max25.673-25.688=-0.015
mean4591.125-4590.375=0.750
σ2248.114-2249.453=-1.339
min1487.000-1486.000=1.000
max7385.000
mean6.433-6.493=-0.060
σ1.049-1.028=0.020
min5.847-5.815=0.031
max9.013-8.798=0.214
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.961-25.984=-0.023
σ0.001-0.000=0.000
min25.961-25.984=-0.023
max25.962-25.985=-0.023
mean4591.125-4590.375=0.750
σ2248.114-2249.453=-1.339
min1487.000-1486.000=1.000
max7385.000
mean6.276-6.438=-0.162
σ1.080-1.023=0.057
min5.777-5.824=-0.047
max8.932-8.711=0.221

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.608-1021.627=-0.019
σ0.002
min1021.606-1021.625=-0.019
max1021.611-1021.629=-0.019
mean7385.000
σ0.000
min7385.000
max7385.000
mean363.197-363.568=-0.371
σ42.355-43.364=-1.009
min345.954-345.268=0.686
max467.946-470.823=-2.877
op=load_product(), with indexes
mean3495.855-3495.878=-0.024
σ0.454-0.469=-0.015
min3495.694-3495.713=-0.019
max3496.978-3497.038=-0.060
mean3758.000-3757.250=0.750
σ2299.230-2300.229=-0.998
min720.000
max7385.000
mean1707.234-1776.147=-68.913
σ573.624-614.401=-40.777
min1397.364-1406.013=-8.649
max2831.830-2806.898=24.933

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense, minor warmup
mean107.841-107.854=-0.013
σ0.000-0.000=0.000
min107.841-107.854=-0.013
max107.842-107.855=-0.013
mean4411.625-4409.125=2.500
σ2217.739-2218.927=-1.187
min1487.000-1486.000=1.000
max7385.000
mean268.489-263.354=5.135
σ369.917-360.253=9.664
min127.104-125.057=2.047
max1182.945-1153.737=29.208
op=column_set() on unindexed, dense, repeat
mean62.068-62.081=-0.013
σ0.001-0.000=0.001
min62.065-62.081=-0.017
max62.068-62.082=-0.013
mean4411.625-4409.125=2.500
σ2217.739-2218.927=-1.187
min1487.000-1486.000=1.000
max7385.000
mean43.714-46.930=-3.216
σ2.632-9.640=-7.007
min41.186-41.441=-0.255
max49.055-70.303=-21.248
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.926-40.948=-0.022
σ0.002
min40.925-40.947=-0.022
max40.928-40.950=-0.022
mean7385.000
σ0.000
min7385.000
max7385.000
mean22.782-23.551=-0.769
σ2.657-3.120=-0.463
min21.610-21.649=-0.038
max29.334-29.599=-0.265
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.294-71.308=-0.014
σ0.002-0.002=-0.000
min71.292-71.307=-0.014
max71.296-71.311=-0.014
mean7385.000
σ0.000
min7385.000
max7385.000
mean233.401-284.900=-51.498
σ499.995-642.015=-142.020
min55.220-54.903=0.317
max1470.811-1873.776=-402.964

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.842-4646.850=-0.008
σ0.013-0.013=-0.000
min4646.837-4646.845=-0.008
max4646.875-4646.883=-0.008
mean4182.250-4180.250=2.000
σ1944.318-1945.738=-1.420
min1487.000-1486.000=1.000
max7385.000
mean3274.550-3243.843=30.708
σ746.869-731.020=15.849
min2497.456-2517.674=-20.218
max4395.059-4455.597=-60.538
op=vertical_lookup_list() on indexed
mean1973.829-1973.851=-0.022
σ0.002
min1973.827-1973.849=-0.022
max1973.832-1973.854=-0.022
mean7385.000
σ0.000
min7385.000
max7385.000
mean1049.890-1057.977=-8.087
σ247.250-191.960=55.290
min915.435-928.569=-13.134
max1633.603-1390.260=243.343
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.083-905.105=-0.022
σ0.002
min905.080-905.102=-0.022
max905.086-905.108=-0.022
mean7385.000
σ0.000
min7385.000
max7385.000
mean406.848-409.064=-2.216
σ48.623-38.539=10.084
min376.355-391.771=-15.416
max526.353-503.659=22.694

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144
σ0.001
min148.144
max148.148
mean477.500-476.000=1.500
σ0.535-0.000=0.535
min477.000-476.000=1.000
max478.000-476.000=2.000
mean393.475-380.687=12.788
σ19.067-34.091=-15.024
min364.756-316.909=47.847
max427.725-438.815=-11.090
op=pointedIndexFromKvs
rows=400000
mean308.493
σ0.001
min308.493
max308.497
mean558.250-560.000=-1.750
σ3.732-0.000=3.732
min554.000-560.000=-6.000
max562.000-560.000=2.000
mean745.838-792.372=-46.535
σ80.636-27.863=52.773
min602.467-752.692=-150.225
max870.497-823.885=46.612

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean964.531-963.971=0.560
σ1.819-1.820=-0.001
min963.826-963.263=0.563
max972.223-971.666=0.557
mean1041.650-1036.050=5.600
σ316.474-304.714=11.760
min542.000-546.000=-4.000
max1553.000-1527.000=26.000
mean1.287-1.251=0.035
σ0.131-0.068=0.063
min1.183-1.179=0.004
max1.727-1.423=0.304

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913-0.913=0.001
σ0.020-0.020=-0.000
min0.903-0.903=0.001
max0.949-0.949=0.000
mean402.000
σ0.000
min402.000
max402.000
mean0.334-0.298=0.035
σ0.298-0.294=0.003
min0.139-0.137=0.002
max0.827-0.821=0.007
name=error_case.art
mean2.346-2.340=0.006
σ0.012
min2.340-2.334=0.006
max2.368-2.362=0.006
mean402.000
σ0.000
min402.000
max402.000
mean0.494-0.484=0.009
σ0.344-0.325=0.019
min0.330-0.321=0.010
max1.109-1.065=0.043
name=heavy_item.art
mean1.523-1.523=0.001
σ0.002-0.002=-0.000
min1.522-1.521=0.001
max1.526-1.525=0.001
mean402.000
σ0.000
min402.000
max402.000
mean0.220-0.225=-0.005
σ0.027-0.023=0.003
min0.198-0.206=-0.008
max0.263-0.264=-0.000
name=nested_use.art
mean2.364-2.358=0.006
σ0.002
min2.363-2.357=0.006
max2.367-2.361=0.006
mean402.000
σ0.000
min402.000
max402.000
mean0.327-0.354=-0.027
σ0.017-0.016=0.001
min0.313-0.342=-0.030
max0.356-0.372=-0.016

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.839-964.971=-0.132
σ0.009-0.011=-0.001
min964.834-964.965=-0.132
max964.855-964.989=-0.134
mean430.600-431.600=-1.000
σ0.548-1.342=-0.794
min430.000-431.000=-1.000
max431.000-434.000=-3.000
mean0.579-0.547=0.032
σ0.034-0.052=-0.018
min0.538-0.479=0.058
max0.612-0.618=-0.006
name=cdr_enhanced_4k_loc.art
mean885.913-885.898=0.015
σ0.002-0.002=-0.000
min885.912-885.897=0.015
max885.915-885.901=0.015
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean0.454-0.461=-0.007
σ0.055-0.064=-0.010
min0.373-0.388=-0.015
max0.505-0.529=-0.023

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.718-7.710=0.009
σ0.002
min7.717-7.709=0.009
max7.721-7.713=0.009
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean2.643-2.702=-0.059
σ0.112-0.100=0.013
min2.467-2.510=-0.044
max3.041-3.059=-0.018
name=record_render_large_syntax_err.art
mean7.667-7.658=0.008
σ0.002
min7.666-7.657=0.008
max7.670-7.661=0.008
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean2.582-2.677=-0.095
σ0.081-0.085=-0.003
min2.457-2.477=-0.020
max2.910-2.945=-0.035
name=record_render_success.art
mean2.290-2.290=0.000
σ0.002
min2.289-2.289=0.000
max2.293-2.293=0.000
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean0.302-0.316=-0.014
σ0.010-0.011=-0.001
min0.290-0.303=-0.013
max0.364-0.407=-0.043
name=record_render_syntax_err.art
mean2.165-2.165=0.000
σ0.002
min2.164-2.164=0.000
max2.168-2.168=0.000
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean0.290-0.302=-0.012
σ0.008-0.008=-0.000
min0.276-0.288=-0.013
max0.325-0.341=-0.016

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.575-11.642=-0.068
σ12.313-11.796=0.517
min0.573-0.574=-0.000
max114.079-91.935=22.144
mean469.623-460.302=9.321
σ19.732-19.766=-0.034
min407.000
max483.000-476.000=7.000
mean0.026-0.027=-0.000
σ0.030-0.029=0.001
min0.001-0.001=-0.000
max0.269-0.236=0.034

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean278.957-278.598=0.359
σ1.019-1.074=-0.055
min278.559-278.093=0.465
max281.856-281.644=0.211
mean443.500-447.800=-4.300
σ17.778-20.088=-2.310
min415.000-417.000=-2.000
max473.000-471.000=2.000
mean0.585-0.588=-0.004
σ0.013-0.012=0.000
min0.568-0.576=-0.007
max0.612-0.613=-0.000

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean169.257-173.738=-4.481
σ0.000-0.000=0.000
min169.257-173.738=-4.481
max169.257-173.738=-4.481
mean851.000-844.000=7.000
σ0.000
min851.000-844.000=7.000
max851.000-844.000=7.000
mean222.372-239.682=-17.309
σ23.005-19.235=3.769
min194.022-197.531=-3.509
max245.821-254.117=-8.296
phase=macro expansion
mean5.896-6.130=-0.234
σ0.000
min5.896-6.130=-0.234
max5.896-6.130=-0.234
mean851.000-843.375=7.625
σ0.000-1.768=-1.768
min851.000-839.000=12.000
max851.000-844.000=7.000
mean7.371-9.062=-1.692
σ2.223-0.978=1.245
min5.893-7.335=-1.442
max11.833-9.703=2.130
phase=parser
mean1041.967-1042.434=-0.467
σ0.000-0.000=0.000
min1041.967-1042.433=-0.466
max1041.967-1042.434=-0.467
mean851.000-843.375=7.625
σ0.000-1.768=-1.768
min851.000-839.000=12.000
max851.000-844.000=7.000
mean545.060-565.747=-20.687
σ122.904-75.086=47.818
min376.022-417.869=-41.847
max725.940-618.704=107.236
phase=processProdDef
mean2243.715-2243.975=-0.260
σ0.000
min2243.715-2243.975=-0.260
max2243.715-2243.975=-0.260
mean851.000-844.000=7.000
σ0.000
min851.000-844.000=7.000
max851.000-844.000=7.000
mean2572.608-2999.984=-427.376
σ433.615-425.591=8.024
min1951.136-2421.614=-470.478
max3259.534-3421.270=-161.736
phase=renaming
mean21.436-21.671=-0.235
σ0.000
min21.436-21.671=-0.235
max21.436-21.671=-0.235
mean851.000-843.375=7.625
σ0.000-1.768=-1.768
min851.000-839.000=12.000
max851.000-844.000=7.000
mean29.414-36.775=-7.361
σ10.264-6.460=3.804
min20.112-26.254=-6.142
max47.999-41.249=6.750
phase=type checker
mean958.600-989.755=-31.156
σ0.047-0.066=-0.019
min958.577-989.658=-31.081
max958.702-989.845=-31.144
mean851.000-843.375=7.625
σ0.000-1.768=-1.768
min851.000-839.000=12.000
max851.000-844.000=7.000
mean685.806-714.014=-28.208
σ88.206-41.405=46.801
min608.041-644.363=-36.322
max840.675-787.700=52.975

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean282.361-282.394=-0.033
σ0.034-0.024=0.010
min282.313-282.373=-0.060
max282.397-282.425=-0.028
mean439.000-444.400=-5.400
σ2.739-0.548=2.191
min437.000-444.000=-7.000
max442.000-445.000=-3.000
mean0.383-0.381=0.001
σ0.007-0.005=0.002
min0.373-0.376=-0.003
max0.388-0.389=-0.001
iters=5
new-facts=1
prior-facts=6402
mean1267.580-1267.114=0.466
σ0.074-0.101=-0.028
min1267.472-1266.993=0.479
max1267.676-1267.270=0.406
mean1111.400-1113.200=-1.800
σ170.789-169.525=1.264
min895.000-898.000=-3.000
max1329.000-1328.000=1.000
mean2.002-2.017=-0.015
σ0.145-0.110=0.035
min1.777-1.872=-0.094
max2.122-2.137=-0.015
iters=5
new-facts=6402
prior-facts=1
mean1506.605-1506.123=0.482
σ2.958-2.939=0.019
min1505.099-1504.603=0.497
max1511.886-1511.369=0.517
mean626.800-629.000=-2.200
σ89.843-90.019=-0.177
min503.000-505.000=-2.000
max726.000-728.000=-2.000
mean2.428-2.338=0.090
σ0.364-0.251=0.113
min2.032-2.072=-0.040
max2.864-2.628=0.236
iters=5
new-facts=6402
prior-facts=6402
mean1829.569-1829.095=0.474
σ0.053-0.055=-0.003
min1829.491-1829.015=0.476
max1829.619-1829.161=0.458
mean1764.600-1756.600=8.000
σ209.681-211.213=-1.531
min1489.000-1473.000=16.000
max1964.000-1956.000=8.000
mean3.128-3.119=0.009
σ0.256-0.255=0.001
min2.845-2.829=0.016
max3.448-3.491=-0.043

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.710-1.737=-0.027
σ0.172-0.126=0.046
min1.467-1.542=-0.075
max1.872-1.838=0.034

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean393.852-397.699=-3.846
σ509.576-518.050=-8.474
min139.061-138.672=0.388
max1158.217-1174.774=-16.557
mean99.750-97.750=2.000
σ7.500
min96.000-94.000=2.000
max111.000-109.000=2.000
mean441.000-448.000=-7.000
σ0.000
min441.000-448.000=-7.000
max441.000-448.000=-7.000
mean0.474-0.503=-0.029
σ0.471-0.524=-0.054
min0.234-0.237=-0.003
max1.180-1.290=-0.110
policies=20
mean1390.729-1386.840=3.889
σ0.163-0.168=-0.005
min1390.644-1386.725=3.919
max1390.973-1387.089=3.883
mean960.000-940.000=20.000
σ0.000
min960.000-940.000=20.000
max960.000-940.000=20.000
mean446.000-449.000=-3.000
σ10.000-1.155=8.845
min441.000-448.000=-7.000
max461.000-450.000=11.000
mean2.312-2.321=-0.008
σ0.011-0.025=-0.014
min2.302-2.286=0.017
max2.325-2.340=-0.015
policies=200
mean13915.790-13876.884=38.906
σ0.137-0.111=0.025
min13915.698-13876.805=38.893
max13915.993-13877.048=38.945
mean9600.000-9400.000=200.000
σ0.000
min9600.000-9400.000=200.000
max9600.000-9400.000=200.000
mean497.000-495.250=1.750
σ4.690-4.787=-0.097
min492.000-491.000=1.000
max503.000-502.000=1.000
mean23.371-23.478=-0.107
σ0.144-0.102=0.041
min23.207-23.407=-0.200
max23.543-23.627=-0.084