Compare commits

Home / Compare 25b28e8b .. a6049d13

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.082-5.301=-0.219
σ0.000
min5.082-5.301=-0.219
max5.082-5.301=-0.219

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean4.988-5.176=-0.188
σ0.000
min4.988-5.176=-0.188
max4.988-5.176=-0.188

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean5.395-5.676=-0.281
σ0.000
min5.395-5.676=-0.281
max5.395-5.676=-0.281

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.633-446.883=-0.250
σ0.000
min446.633-446.883=-0.250
max446.633-446.883=-0.250

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.238-5.488=-0.250
σ0.000
min5.238-5.488=-0.250
max5.238-5.488=-0.250

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.395-6.676=-0.281
σ0.000
min6.395-6.676=-0.281
max6.395-6.676=-0.281

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.717-54.775=-0.058
σ0.363-0.366=-0.003
min54.582-54.633=-0.050
max55.750-55.816=-0.066
mean418.000-416.000=2.000
σ0.000
min418.000-416.000=2.000
max418.000-416.000=2.000
mean0.142-0.141=0.001
σ0.005-0.004=0.001
min0.137-0.138=-0.000
max0.151-0.148=0.002
cache=y
dps=1
iters=10
policies=100
mean43.779-43.773=0.006
σ0.007-0.005=0.002
min43.767-43.764=0.002
max43.788-43.780=0.008
mean418.000-416.000=2.000
σ0.000
min418.000-416.000=2.000
max418.000-416.000=2.000
mean0.124-0.124=0.000
σ0.004-0.004=0.001
min0.121-0.121=0.001
max0.132-0.130=0.002

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.604-36.617=-0.013
σ0.000-0.000=0.000
min36.604-36.617=-0.013
max36.605-36.618=-0.013
mean4589.500-4592.750=-3.250
σ2247.606-2248.965=-1.360
min1486.000
max7379.000-7382.000=-3.000
mean152.166-141.847=10.319
σ395.904-366.461=29.444
min11.921-11.858=0.063
max1131.979-1048.791=83.189
op=column_set() on indexed, repeat
mean36.604-36.617=-0.013
σ0.000-0.000=0.000
min36.604-36.617=-0.013
max36.605-36.618=-0.013
mean4589.500-4592.750=-3.250
σ2247.606-2248.965=-1.360
min1486.000
max7379.000-7382.000=-3.000
mean14.063-13.571=0.492
σ4.367-4.714=-0.347
min11.770-11.085=0.685
max24.818-25.111=-0.294
op=table_records(), no indexes
mean41.940-41.948=-0.008
σ0.002-0.002=-0.000
min41.938-41.946=-0.009
max41.942-41.950=-0.008
mean7379.000-7382.000=-3.000
σ0.000
min7379.000-7382.000=-3.000
max7379.000-7382.000=-3.000
mean11.610-11.170=0.440
σ1.753-0.243=1.510
min10.717-10.851=-0.134
max15.929-11.575=4.353
op=table_records(), repeat
mean41.925-41.933=-0.008
σ0.002-0.000=0.001
min41.924-41.933=-0.009
max41.929-41.934=-0.005
mean4180.375-4184.375=-4.000
σ1944.792-1946.392=-1.600
min1486.000
max7379.000-7382.000=-3.000
mean11.839-11.684=0.155
σ1.840-1.678=0.162
min10.837-10.713=0.124
max16.326-15.815=0.510
op=vertical_lookup_list() on indexed, after table_records()
mean27.633-27.655=-0.023
σ0.137-0.136=0.000
min27.584-27.606=-0.022
max27.971-27.993=-0.022
mean4180.375-4184.375=-4.000
σ1944.792-1946.392=-1.600
min1486.000
max7379.000-7382.000=-3.000
mean9.085-9.035=0.049
σ1.232-1.333=-0.101
min8.080-8.099=-0.019
max12.048-12.251=-0.203
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.384-26.406=-0.022
σ0.000-0.000=0.000
min26.384-26.406=-0.022
max26.384-26.406=-0.022
mean4180.375-4184.375=-4.000
σ1944.792-1946.392=-1.600
min1486.000
max7379.000-7382.000=-3.000
mean6.787-6.671=0.115
σ0.972-1.262=-0.290
min6.008-5.777=0.232
max9.056-9.621=-0.565
op=vertical_lookup_row_list() on 2 indexed
mean25.662-25.681=-0.018
σ0.004-0.003=0.001
min25.661-25.680=-0.019
max25.673-25.688=-0.015
mean4589.500-4592.750=-3.250
σ2247.606-2248.965=-1.360
min1486.000
max7379.000-7382.000=-3.000
mean6.331-6.243=0.089
σ0.982-0.996=-0.014
min5.860-5.826=0.034
max8.745-8.699=0.046
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.961-25.984=-0.023
σ0.000-0.000=0.000
min25.961-25.984=-0.024
max25.962-25.985=-0.023
mean4589.500-4592.750=-3.250
σ2247.606-2248.965=-1.360
min1486.000
max7379.000-7382.000=-3.000
mean6.252-6.190=0.062
σ1.087-1.026=0.061
min5.462-5.594=-0.132
max8.902-8.704=0.198

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.608-1021.627=-0.019
σ0.002-0.002=0.000
min1021.606-1021.625=-0.019
max1021.611-1021.629=-0.019
mean7379.000-7382.000=-3.000
σ0.000
min7379.000-7382.000=-3.000
max7379.000-7382.000=-3.000
mean344.088-362.486=-18.398
σ1.848-41.765=-39.917
min341.733-344.999=-3.265
max348.068-465.771=-117.703
op=load_product(), with indexes
mean3495.855-3495.878=-0.024
σ0.454-0.469=-0.015
min3495.694-3495.713=-0.019
max3496.978-3497.038=-0.060
mean3757.000-3760.000=-3.000
σ2299.600-2301.410=-1.810
min719.000-720.000=-1.000
max7379.000-7382.000=-3.000
mean1733.709-1776.557=-42.848
σ626.324-695.379=-69.055
min1386.028-1399.742=-13.714
max2776.427-3000.132=-223.705

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense, minor warmup
mean107.841-107.854=-0.013
σ0.000-0.000=0.000
min107.840-107.854=-0.013
max107.842-107.855=-0.013
mean4408.250-4412.000=-3.750
σ2216.785-2217.923=-1.138
min1486.000
max7379.000-7382.000=-3.000
mean189.263-245.400=-56.137
σ151.375-317.196=-165.822
min122.968-124.008=-1.040
max561.792-1029.346=-467.554
op=column_set() on unindexed, dense, repeat
mean62.068-62.081=-0.013
σ0.001-0.000=0.001
min62.065-62.081=-0.017
max62.068-62.082=-0.013
mean4408.250-4412.000=-3.750
σ2216.785-2217.923=-1.138
min1486.000
max7379.000-7382.000=-3.000
mean42.425-45.585=-3.160
σ1.508-6.381=-4.873
min40.671-41.117=-0.446
max44.944-60.866=-15.922
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.926-40.947=-0.022
σ0.002-0.001=0.000
min40.925-40.947=-0.022
max40.928-40.950=-0.022
mean7379.000-7382.000=-3.000
σ0.000
min7379.000-7382.000=-3.000
max7379.000-7382.000=-3.000
mean216.994-22.499=194.496
σ550.319-1.150=549.170
min21.771-21.704=0.067
max1578.963-25.130=1553.833
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.294-71.308=-0.015
σ0.002-0.002=-0.000
min71.292-71.307=-0.014
max71.296-71.311=-0.014
mean7379.000-7382.000=-3.000
σ0.000
min7379.000-7382.000=-3.000
max7379.000-7382.000=-3.000
mean64.840-298.441=-233.601
σ23.436-684.341=-660.904
min55.245-55.493=-0.249
max122.577-1992.094=-1869.517

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.842-4646.850=-0.008
σ0.014-0.013=0.000
min4646.837-4646.845=-0.008
max4646.876-4646.883=-0.008
mean4180.375-4184.375=-4.000
σ1944.792-1946.392=-1.600
min1486.000
max7379.000-7382.000=-3.000
mean3241.555-3276.127=-34.571
σ840.523-778.527=61.996
min2472.741-2511.570=-38.828
max4754.069-4539.401=214.668
op=vertical_lookup_list() on indexed
mean1973.829-1973.851=-0.022
σ0.002-0.002=-0.000
min1973.827-1973.849=-0.022
max1973.832-1973.854=-0.022
mean7379.000-7382.000=-3.000
σ0.000
min7379.000-7382.000=-3.000
max7379.000-7382.000=-3.000
mean1016.918-1173.288=-156.370
σ217.923-501.344=-283.421
min912.869-938.623=-25.754
max1532.311-2394.836=-862.525
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.083-905.104=-0.022
σ0.002-0.002=0.000
min905.080-905.102=-0.022
max905.086-905.108=-0.022
mean7379.000-7382.000=-3.000
σ0.000
min7379.000-7382.000=-3.000
max7379.000-7382.000=-3.000
mean389.504-392.346=-2.842
σ7.840-3.365=4.475
min372.940-386.920=-13.979
max401.818-396.768=5.050

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144
σ0.001
min148.144
max148.148
mean475.000-476.875=-1.875
σ0.000-1.246=-1.246
min475.000
max475.000-478.000=-3.000
mean371.018-376.077=-5.059
σ35.783-62.653=-26.870
min306.224-296.742=9.482
max418.602-446.077=-27.475
op=pointedIndexFromKvs
rows=400000
mean308.493
σ0.001
min308.493
max308.497
mean562.500-564.000=-1.500
σ0.926-0.000=0.926
min561.000-564.000=-3.000
max563.000-564.000=-1.000
mean744.982-779.351=-34.369
σ71.027-63.951=7.076
min612.414-645.597=-33.183
max825.746-842.352=-16.607

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean964.563-963.972=0.591
σ1.879-1.821=0.057
min963.842-963.264=0.578
max972.509-971.675=0.835
mean1037.900-1030.000=7.900
σ309.207-301.069=8.138
min546.000-550.000=-4.000
max1534.000-1457.000=77.000
mean1.250-1.290=-0.039
σ0.071-0.121=-0.050
min1.174-1.192=-0.018
max1.389-1.677=-0.288

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913-0.913=0.001
σ0.020-0.020=-0.000
min0.903-0.903=0.001
max0.949-0.949=0.000
mean402.000
σ0.000
min402.000
max402.000
mean0.298-0.300=-0.002
σ0.296-0.305=-0.008
min0.139-0.136=0.003
max0.824-0.840=-0.016
name=error_case.art
mean2.346-2.340=0.006
σ0.012
min2.340-2.334=0.006
max2.368-2.362=0.006
mean402.000
σ0.000
min402.000
max402.000
mean0.475-0.475=-0.000
σ0.315-0.318=-0.004
min0.320-0.320=-0.000
max1.037-1.045=-0.007
name=heavy_item.art
mean1.523-1.523=0.001
σ0.002-0.002=-0.000
min1.522-1.521=0.001
max1.526-1.525=0.001
mean402.000
σ0.000
min402.000
max402.000
mean0.213-0.219=-0.006
σ0.020-0.024=-0.004
min0.197-0.201=-0.004
max0.246-0.259=-0.012
name=nested_use.art
mean2.364-2.358=0.006
σ0.002
min2.363-2.357=0.006
max2.367-2.361=0.006
mean402.000
σ0.000
min402.000
max402.000
mean0.335-0.331=0.003
σ0.015-0.012=0.003
min0.318-0.323=-0.004
max0.356-0.352=0.004

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.979-964.971=0.008
σ0.009-0.011=-0.001
min964.974-964.965=0.008
max964.996-964.989=0.006
mean436.400-431.000=5.400
σ6.148-0.000=6.148
min431.000
max443.000-431.000=12.000
mean0.553-0.561=-0.008
σ0.049-0.051=-0.002
min0.480-0.483=-0.003
max0.593-0.615=-0.022
name=cdr_enhanced_4k_loc.art
mean886.001-885.898=0.103
σ0.002-0.002=-0.000
min886.000-885.897=0.103
max886.004-885.901=0.103
mean443.000-431.000=12.000
σ0.000
min443.000-431.000=12.000
max443.000-431.000=12.000
mean0.467-0.470=-0.004
σ0.042-0.051=-0.009
min0.424-0.394=0.030
max0.520-0.521=-0.001

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.719-7.710=0.009
σ0.002
min7.718-7.709=0.009
max7.722-7.713=0.009
mean443.000-431.000=12.000
σ0.000
min443.000-431.000=12.000
max443.000-431.000=12.000
mean2.677-2.635=0.042
σ0.106-0.087=0.019
min2.531-2.455=0.076
max3.056-2.862=0.193
name=record_render_large_syntax_err.art
mean7.667-7.658=0.009
σ0.002
min7.666-7.657=0.009
max7.670-7.661=0.009
mean443.000-431.000=12.000
σ0.000
min443.000-431.000=12.000
max443.000-431.000=12.000
mean2.666-2.603=0.064
σ0.120-0.090=0.029
min2.480-2.429=0.051
max3.413-2.911=0.501
name=record_render_success.art
mean2.290-2.290=0.001
σ0.002
min2.290-2.289=0.001
max2.294-2.293=0.001
mean443.000-431.000=12.000
σ0.000
min443.000-431.000=12.000
max443.000-431.000=12.000
mean0.310-0.311=-0.002
σ0.025-0.010=0.015
min0.291-0.294=-0.003
max0.540-0.367=0.173
name=record_render_syntax_err.art
mean2.166-2.165=0.001
σ0.002
min2.165-2.164=0.001
max2.169-2.168=0.001
mean443.000-431.000=12.000
σ0.000
min443.000-431.000=12.000
max443.000-431.000=12.000
mean0.294-0.298=-0.004
σ0.008-0.010=-0.003
min0.275-0.280=-0.005
max0.331-0.365=-0.034

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.206-11.859=-0.652
σ11.665-11.788=-0.123
min0.574-0.718=-0.145
max91.892-88.744=3.148
mean467.413-465.442=1.971
σ20.373-18.870=1.503
min406.000-407.000=-1.000
max484.000-481.000=3.000
mean0.026-0.027=-0.001
σ0.028-0.029=-0.000
min0.001-0.001=-0.000
max0.218-0.203=0.016

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean279.004-278.614=0.391
σ1.055-1.051=0.004
min278.606-278.212=0.394
max282.002-281.603=0.400
mean443.900-445.500=-1.600
σ16.196-21.230=-5.034
min426.000-416.000=10.000
max469.000-475.000=-6.000
mean0.580-0.582=-0.002
σ0.013-0.021=-0.008
min0.558-0.539=0.019
max0.593-0.612=-0.019

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean169.257-173.738=-4.481
σ0.000-0.000=0.000
min169.257-173.738=-4.481
max169.257-173.738=-4.481
mean848.000-864.000=-16.000
σ0.000
min848.000-864.000=-16.000
max848.000-864.000=-16.000
mean202.481-233.430=-30.950
σ15.433-26.918=-11.485
min188.583-197.904=-9.321
max237.333-259.144=-21.810
phase=macro expansion
mean5.896-6.130=-0.234
σ0.000-0.001=-0.001
min5.896-6.130=-0.234
max5.896-6.133=-0.237
mean848.000-864.000=-16.000
σ0.000
min848.000-864.000=-16.000
max848.000-864.000=-16.000
mean6.233-9.078=-2.845
σ1.053-1.165=-0.112
min5.797-6.336=-0.539
max8.834-9.710=-0.876
phase=parser
mean1042.134-1042.435=-0.301
σ0.000-0.001=-0.001
min1042.134-1042.432=-0.298
max1042.134-1042.436=-0.302
mean848.000-864.000=-16.000
σ0.000
min848.000-864.000=-16.000
max848.000-864.000=-16.000
mean404.630-551.548=-146.918
σ81.722-95.535=-13.812
min371.799-377.900=-6.101
max606.705-614.147=-7.441
phase=processProdDef
mean2243.717-2243.976=-0.259
σ0.000-0.001=-0.001
min2243.717-2243.976=-0.258
max2243.717-2243.979=-0.261
mean848.000-864.000=-16.000
σ0.000
min848.000-864.000=-16.000
max848.000-864.000=-16.000
mean2877.529-2595.686=281.843
σ349.611-521.480=-171.869
min2322.665-1734.264=588.401
max3230.083-3247.753=-17.671
phase=renaming
mean21.436-21.671=-0.235
σ0.000
min21.436-21.671=-0.235
max21.436-21.671=-0.235
mean848.000-864.000=-16.000
σ0.000
min848.000-864.000=-16.000
max848.000-864.000=-16.000
mean34.986-35.966=-0.980
σ4.371-7.817=-3.445
min31.986-20.785=11.201
max45.296-41.172=4.124
phase=type checker
mean958.600-989.755=-31.155
σ0.047-0.066=-0.020
min958.576-989.654=-31.078
max958.701-989.845=-31.144
mean848.000-864.000=-16.000
σ0.000
min848.000-864.000=-16.000
max848.000-864.000=-16.000
mean644.923-732.371=-87.448
σ15.168-86.308=-71.140
min627.367-670.303=-42.936
max673.192-873.506=-200.315

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean282.855-282.394=0.461
σ0.039-0.025=0.014
min282.800-282.372=0.428
max282.894-282.425=0.469
mean439.400-442.000=-2.600
σ3.286-0.000=3.286
min437.000-442.000=-5.000
max443.000-442.000=1.000
mean0.382-0.377=0.005
σ0.008-0.002=0.006
min0.377-0.375=0.002
max0.397-0.380=0.017
iters=5
new-facts=1
prior-facts=6402
mean1268.045-1267.112=0.933
σ0.080-0.103=-0.023
min1267.961-1267.001=0.959
max1268.166-1267.278=0.889
mean1082.400-1112.200=-29.800
σ179.801-170.973=8.828
min870.000-896.000=-26.000
max1323.000-1331.000=-8.000
mean1.949-1.989=-0.040
σ0.115-0.121=-0.005
min1.787-1.822=-0.035
max2.109-2.106=0.003
iters=5
new-facts=6402
prior-facts=1
mean1507.079-1506.111=0.968
σ2.960-2.956=0.004
min1505.585-1504.606=0.979
max1512.366-1511.388=0.978
mean633.400-628.200=5.200
σ89.879-90.489=-0.609
min497.000-504.000=-7.000
max731.000-728.000=3.000
mean2.339-2.303=0.035
σ0.289-0.286=0.002
min2.113-2.072=0.040
max2.670-2.776=-0.106
iters=5
new-facts=6402
prior-facts=6402
mean1830.067-1829.110=0.956
σ0.038-0.070=-0.032
min1830.008-1829.022=0.985
max1830.096-1829.206=0.890
mean1723.000-1765.600=-42.600
σ168.210-209.681=-41.472
min1486.000-1490.000=-4.000
max1881.000-1965.000=-84.000
mean2.968-3.051=-0.082
σ0.208-0.180=0.028
min2.816-2.829=-0.013
max3.335-3.290=0.045

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.753-1.790=-0.037
σ0.065-0.118=-0.053
min1.680-1.581=0.100
max1.821-1.855=-0.034

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean406.199-397.698=8.501
σ509.798-518.051=-8.253
min151.294-138.670=12.624
max1170.895-1174.774=-3.879
mean101.750-97.750=4.000
σ7.500
min98.000-94.000=4.000
max113.000-109.000=4.000
mean439.000
σ0.000
min439.000
max439.000
mean0.483-0.495=-0.012
σ0.452-0.507=-0.055
min0.249-0.240=0.009
max1.160-1.255=-0.094
policies=20
mean1513.142-1386.836=126.306
σ0.277-0.159=0.118
min1512.961-1386.727=126.234
max1513.555-1387.072=126.483
mean980.000-940.000=40.000
σ0.000
min980.000-940.000=40.000
max980.000-940.000=40.000
mean445.750-444.000=1.750
σ4.500-5.774=-1.274
min439.000
max448.000-449.000=-1.000
mean2.509-2.328=0.181
σ0.010-0.015=-0.005
min2.499-2.309=0.190
max2.523-2.345=0.177
policies=200
mean15140.031-13876.909=1263.122
σ0.089-0.101=-0.012
min15139.953-13876.828=1263.125
max15140.158-13877.055=1263.103
mean9800.000-9400.000=400.000
σ0.000
min9800.000-9400.000=400.000
max9800.000-9400.000=400.000
mean496.500-494.250=2.250
σ5.196-4.856=0.340
min491.000-488.000=3.000
max503.000-499.000=4.000
mean25.286-23.625=1.662
σ0.098-0.187=-0.089
min25.212-23.421=1.791
max25.423-23.875=1.549