Compare commits

Home / Compare f114d67d .. 2fd15579

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.082-5.301=-0.219
σ0.000
min5.082-5.301=-0.219
max5.082-5.301=-0.219

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean4.988-5.176=-0.188
σ0.000
min4.988-5.176=-0.188
max4.988-5.176=-0.188

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean5.395
σ0.000
min5.395
max5.395

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.633-558.664=-112.031
σ0.000
min446.633-558.664=-112.031
max446.633-558.664=-112.031

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.238-5.488=-0.250
σ0.000
min5.238-5.488=-0.250
max5.238-5.488=-0.250

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.395-6.645=-0.250
σ0.000
min6.395-6.645=-0.250
max6.395-6.645=-0.250

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.720-54.881=-0.161
σ0.361-0.359=0.002
min54.577-54.745=-0.168
max55.746-55.902=-0.156
mean420.000-417.000=3.000
σ0.000
min420.000-417.000=3.000
max420.000-417.000=3.000
mean0.138-0.141=-0.003
σ0.004-0.004=0.000
min0.131-0.137=-0.006
max0.145-0.148=-0.003
cache=y
dps=1
iters=10
policies=100
mean43.774-43.849=-0.075
σ0.006-0.007=-0.001
min43.764-43.836=-0.072
max43.782-43.858=-0.076
mean420.000-417.000=3.000
σ0.000
min420.000-417.000=3.000
max420.000-417.000=3.000
mean0.120-0.125=-0.005
σ0.004-0.004=-0.001
min0.116-0.121=-0.005
max0.126-0.132=-0.006

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.604-36.618=-0.014
σ0.000-0.002=-0.002
min36.604-36.616=-0.012
max36.605-36.620=-0.016
mean4427.375-4041.250=386.125
σ2226.908-2008.838=218.070
min1483.000-1216.000=267.000
max7408.000-7148.000=260.000
mean12.799-12.922=-0.122
σ1.978-0.315=1.662
min11.737-12.554=-0.817
max17.577-13.399=4.178
op=column_set() on indexed, repeat
mean36.604-36.618=-0.014
σ0.000-0.002=-0.002
min36.604-36.616=-0.012
max36.605-36.620=-0.015
mean4607.125-4041.250=565.875
σ2257.941-2008.838=249.103
min1483.000-1216.000=267.000
max7408.000-7148.000=260.000
mean143.670-12.723=130.947
σ368.162-0.673=367.489
min11.591-11.824=-0.233
max1054.810-13.784=1041.026
op=table_records(), no indexes
mean41.939-41.930=0.010
σ0.002-0.002=0.000
min41.938-41.928=0.009
max41.942-41.933=0.009
mean7408.000-7148.000=260.000
σ0.000
min7408.000-7148.000=260.000
max7408.000-7148.000=260.000
mean11.145-15.374=-4.230
σ0.183-0.184=-0.001
min10.830-15.153=-4.322
max11.375-15.642=-4.266
op=table_records(), repeat
mean41.925-41.929=-0.004
σ0.000-0.002=-0.002
min41.924-41.927=-0.003
max41.925-41.932=-0.007
mean4198.250-4041.250=157.000
σ1953.999-2008.838=-54.839
min1483.000-1216.000=267.000
max7408.000-7148.000=260.000
mean11.602-13.615=-2.012
σ1.765-1.889=-0.124
min10.688-10.758=-0.070
max15.933-16.383=-0.450
op=vertical_lookup_list() on indexed, after table_records()
mean27.633-27.654=-0.021
σ0.137-0.136=0.001
min27.584-27.604=-0.020
max27.971-27.991=-0.020
mean4198.250-4041.250=157.000
σ1953.999-2008.838=-54.839
min1483.000-1216.000=267.000
max7408.000-7148.000=260.000
mean9.100-9.605=-0.505
σ1.293-1.283=0.011
min7.989-8.591=-0.601
max12.220-12.677=-0.457
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.384-26.401=-0.017
σ0.000-0.002=-0.002
min26.384-26.400=-0.016
max26.384-26.404=-0.020
mean4198.250-4041.250=157.000
σ1953.999-2008.838=-54.839
min1483.000-1216.000=267.000
max7408.000-7148.000=260.000
mean7.211-8.017=-0.806
σ1.003-1.500=-0.496
min6.280-6.994=-0.713
max9.609-10.704=-1.095
op=vertical_lookup_row_list() on 2 indexed
mean25.662-25.668=-0.006
σ0.004-0.004=0.000
min25.661-25.665=-0.005
max25.673-25.678=-0.005
mean4607.125-4041.250=565.875
σ2257.941-2008.838=249.103
min1483.000-1216.000=267.000
max7408.000-7148.000=260.000
mean6.639-6.038=0.600
σ1.114-0.108=1.006
min6.012-5.866=0.146
max9.119-6.163=2.955
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.961-25.970=-0.009
σ0.000-0.002=-0.001
min25.961-25.968=-0.008
max25.962-25.972=-0.010
mean4607.125-4041.250=565.875
σ2257.941-2008.838=249.103
min1483.000-1216.000=267.000
max7408.000-7148.000=260.000
mean6.448-5.855=0.592
σ1.117-0.100=1.017
min5.779-5.674=0.106
max8.866-5.989=2.877

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.610-1021.613=-0.003
σ0.002-0.002=0.000
min1021.608-1021.612=-0.004
max1021.613-1021.616=-0.003
mean7408.000-7148.000=260.000
σ0.000
min7408.000-7148.000=260.000
max7408.000-7148.000=260.000
mean346.211-366.098=-19.887
σ3.825-53.388=-49.564
min341.251-344.524=-3.272
max354.793-498.137=-143.344
op=load_product(), with indexes
mean3495.857-7917.792=-4421.935
σ0.455-0.469=-0.014
min3495.696-7917.624=-4421.929
max3496.983-7918.954=-4421.970
mean3771.125-3272.875=498.250
σ2310.280-1823.353=486.927
min720.000-809.000=-89.000
max7408.000-5619.000=1789.000
mean1719.999-3806.810=-2086.811
σ560.280-1320.083=-759.804
min1387.621-3082.815=-1695.194
max2811.418-6199.142=-3387.724

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense
mean62.077
σ0.002
min62.076
max62.080
mean4041.250
σ2008.838
min1216.000
max7148.000
mean91.506
σ19.702
min75.383
max127.516
op=column_set() on unindexed, dense, minor warmup
mean107.841
σ0.000
min107.840
max107.841
mean4198.250
σ1953.999
min1483.000
max7408.000
mean136.261
σ12.792
min126.249
max166.958
op=column_set() on unindexed, dense, repeat
mean62.068
σ0.001
min62.065
max62.069
mean4427.375
σ2226.908
min1483.000
max7408.000
mean166.759
σ353.262
min40.527
max1041.034
op=vertical_lookup_list() on unindexed missing Int
mean40.946
σ0.002
min40.945
max40.950
mean7148.000
σ0.000
min7148.000
max7148.000
mean61.265
σ0.323
min60.918
max61.671
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.926
σ0.002
min40.924
max40.929
mean7408.000
σ0.000
min7408.000
max7408.000
mean104.012
σ230.460
min21.622
max674.365
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.294-66.688=4.606
σ0.002-0.002=-0.000
min71.292-66.686=4.606
max71.296-66.691=4.606
mean7408.000-7148.000=260.000
σ0.000
min7408.000-7148.000=260.000
max7408.000-7148.000=260.000
mean59.773-100.160=-40.387
σ9.223-0.693=8.529
min55.757-99.234=-43.477
max82.533-101.288=-18.754

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.842-14217.171=-9570.328
σ0.013-4.304=-4.291
min4646.837-14215.575=-9568.738
max4646.875-14227.821=-9580.946
mean4198.250-4041.250=157.000
σ1953.999-2008.838=-54.839
min1483.000-1216.000=267.000
max7408.000-7148.000=260.000
mean3156.464-8175.550=-5019.085
σ601.794-2157.102=-1555.308
min2520.646-6002.891=-3482.245
max4118.876-11067.801=-6948.925
op=vertical_lookup_list() on indexed
mean1973.829-13159.109=-11185.279
σ0.002-0.002=0.000
min1973.827-13159.106=-11185.279
max1973.831-13159.110=-11185.279
mean7408.000-7148.000=260.000
σ0.000
min7408.000-7148.000=260.000
max7408.000-7148.000=260.000
mean1034.237-5090.331=-4056.094
σ194.501-172.373=22.128
min924.593-4962.508=-4037.915
max1496.686-5381.260=-3884.574
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.082
σ0.002
min905.080
max905.086
mean7408.000
σ0.000
min7408.000
max7408.000
mean404.649
σ40.779
min381.279
max505.132

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144-172.559=-24.414
σ0.001
min148.144-172.558=-24.414
max148.148-172.562=-24.414
mean475.875-477.000=-1.125
σ0.991-0.000=0.991
min475.000-477.000=-2.000
max477.000
mean375.353-382.309=-6.956
σ31.720-19.092=12.628
min314.831-350.784=-35.952
max425.647-394.179=31.468
op=pointedIndexFromKvs
rows=400000
mean308.493-357.322=-48.828
σ0.001
min308.493-357.321=-48.828
max308.497-357.325=-48.828
mean560.000-561.000=-1.000
σ0.000
min560.000-561.000=-1.000
max560.000-561.000=-1.000
mean747.186-704.380=42.806
σ47.890-73.371=-25.480
min664.840-591.987=72.853
max813.614-790.846=22.769

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean964.530-1024.299=-59.770
σ1.822-0.759=1.063
min963.823-1023.827=-60.004
max972.234-1027.437=-55.204
mean1039.850-1037.800=2.050
σ293.473-301.078=-7.606
min552.000-562.000=-10.000
max1432.000-1460.000=-28.000
mean1.265-1.536=-0.272
σ0.102-0.068=0.034
min1.174-1.361=-0.187
max1.511-1.642=-0.131

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.914-0.913=0.002
σ0.020
min0.904-0.903=0.002
max0.951-0.949=0.002
mean402.000
σ0.000
min402.000
max402.000
mean0.392-0.442=-0.050
σ0.495-0.625=-0.130
min0.138-0.132=0.006
max1.272-1.557=-0.285
name=error_case.art
mean2.347-2.339=0.008
σ0.012
min2.341-2.333=0.008
max2.369-2.361=0.008
mean402.000
σ0.000
min402.000
max402.000
mean0.486-0.719=-0.233
σ0.322-0.292=0.030
min0.331-0.562=-0.231
max1.061-1.239=-0.177
name=heavy_item.art
mean1.525-1.522=0.003
σ0.002-0.002=-0.000
min1.524-1.521=0.003
max1.528-1.525=0.003
mean402.000
σ0.000
min402.000
max402.000
mean0.219-0.381=-0.163
σ0.027-0.041=-0.014
min0.201-0.356=-0.154
max0.266-0.451=-0.186
name=nested_use.art
mean2.365-2.357=0.008
σ0.002
min2.364-2.356=0.008
max2.368-2.360=0.008
mean402.000
σ0.000
min402.000
max402.000
mean0.451-0.584=-0.133
σ0.248-0.030=0.218
min0.326-0.560=-0.234
max0.895-0.634=0.261

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean966.287-964.840=1.446
σ0.009-0.011=-0.001
min966.281-964.835=1.447
max966.303-964.859=1.444
mean432.600-435.200=-2.600
σ0.894-4.550=-3.655
min431.000
max433.000-440.000=-7.000
mean0.553-0.608=-0.055
σ0.073-0.075=-0.002
min0.454-0.548=-0.094
max0.613-0.739=-0.126
name=cdr_enhanced_4k_loc.art
mean886.717-885.820=0.897
σ0.002-0.002=-0.000
min886.716-885.819=0.897
max886.720-885.823=0.897
mean433.000-440.000=-7.000
σ0.000
min433.000-440.000=-7.000
max433.000-440.000=-7.000
mean0.485-0.463=0.022
σ0.035-0.032=0.003
min0.434-0.426=0.009
max0.520-0.495=0.025

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.722-7.709=0.013
σ0.002
min7.721-7.708=0.013
max7.725-7.712=0.013
mean433.000-440.000=-7.000
σ0.000
min433.000-440.000=-7.000
max433.000-440.000=-7.000
mean2.654-2.634=0.019
σ0.100-0.123=-0.023
min2.488-2.457=0.031
max3.002-3.248=-0.246
name=record_render_large_syntax_err.art
mean7.670-7.657=0.013
σ0.002
min7.670-7.656=0.013
max7.673-7.660=0.013
mean433.000-440.000=-7.000
σ0.000
min433.000-440.000=-7.000
max433.000-440.000=-7.000
mean2.617-2.605=0.011
σ0.090-0.094=-0.004
min2.466-2.428=0.038
max2.876-2.841=0.035
name=record_render_success.art
mean2.294-2.289=0.004
σ0.002
min2.293-2.289=0.004
max2.297-2.293=0.004
mean433.000-440.000=-7.000
σ0.000
min433.000-440.000=-7.000
max433.000-440.000=-7.000
mean0.303-0.303=-0.000
σ0.009-0.011=-0.002
min0.287-0.286=0.001
max0.356-0.372=-0.015
name=record_render_syntax_err.art
mean2.169-2.165=0.004
σ0.002
min2.168-2.164=0.004
max2.172-2.168=0.004
mean433.000-440.000=-7.000
σ0.000
min433.000-440.000=-7.000
max433.000-440.000=-7.000
mean0.290-0.303=-0.013
σ0.008-0.070=-0.061
min0.276-0.274=0.002
max0.331-0.757=-0.426

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.219
σ11.625
min0.573
max90.260
mean467.711
σ18.963
min408.000
max483.000
mean0.026
σ0.028
min0.001
max0.222

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean278.777-298.896=-20.119
σ1.032-1.002=0.031
min278.383-298.502=-20.119
max281.712-301.743=-20.031
mean441.300-441.200=0.100
σ17.250-14.359=2.891
min414.000-418.000=-4.000
max465.000-467.000=-2.000
mean0.568-0.598=-0.030
σ0.028-0.024=0.003
min0.527-0.546=-0.019
max0.610-0.621=-0.011

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean169.258-236.678=-67.420
σ0.000-0.000=0.000
min169.258-236.678=-67.420
max169.258-236.678=-67.420
mean698.000-1065.000=-367.000
σ0.000-33.941=-33.941
min698.000-981.000=-283.000
max698.000-1077.000=-379.000
mean231.524-204.754=26.770
σ21.844-2.560=19.284
min194.421-200.470=-6.049
max250.121-208.766=41.355
phase=macro expansion
mean5.900-6.130=-0.229
σ0.000
min5.900-6.130=-0.229
max5.900-6.130=-0.229
mean698.000-1065.000=-367.000
σ0.000-33.941=-33.941
min698.000-981.000=-283.000
max698.000-1077.000=-379.000
mean5.862-10.272=-4.410
σ0.063-2.103=-2.040
min5.780-6.365=-0.584
max5.976-13.111=-7.135
phase=parser
mean1043.350-1042.363=0.987
σ0.000-0.000=-0.000
min1043.350-1042.362=0.988
max1043.351-1042.363=0.988
mean698.000-1065.000=-367.000
σ0.000-33.941=-33.941
min698.000-981.000=-283.000
max698.000-1077.000=-379.000
mean498.648-534.641=-35.993
σ94.950-108.425=-13.475
min407.183-379.637=27.545
max613.643-708.169=-94.526
phase=processProdDef
mean2243.722-4850.764=-2607.042
σ0.000
min2243.722-4850.764=-2607.042
max2243.722-4850.764=-2607.042
mean698.000-1077.000=-379.000
σ0.000
min698.000-1077.000=-379.000
max698.000-1077.000=-379.000
mean2949.579-4957.461=-2007.882
σ335.370-326.121=9.249
min2208.441-4349.627=-2141.186
max3279.957-5330.964=-2051.006
phase=renaming
mean21.438-21.671=-0.233
σ0.000
min21.438-21.671=-0.233
max21.438-21.671=-0.233
mean698.000-1065.000=-367.000
σ0.000-33.941=-33.941
min698.000-981.000=-283.000
max698.000-1077.000=-379.000
mean34.531-39.943=-5.411
σ1.695-8.042=-6.347
min32.254-21.633=10.621
max36.938-48.137=-11.198
phase=type checker
mean958.597-990.450=-31.853
σ0.047-0.071=-0.024
min958.573-990.344=-31.771
max958.698-990.563=-31.865
mean698.000-1065.000=-367.000
σ0.000-33.941=-33.941
min698.000-981.000=-283.000
max698.000-1077.000=-379.000
mean664.546-745.459=-80.914
σ12.819-82.410=-69.590
min645.561-643.765=1.796
max679.296-894.559=-215.263

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean282.367-293.821=-11.455
σ0.035-0.038=-0.003
min282.312-293.768=-11.456
max282.397-293.860=-11.463
mean443.600-443.000=0.600
σ0.548-0.000=0.548
min443.000
max444.000-443.000=1.000
mean0.391-0.400=-0.009
σ0.010-0.005=0.005
min0.374-0.394=-0.020
max0.398-0.406=-0.008
iters=5
new-facts=1
prior-facts=6402
mean1267.620-1320.211=-52.591
σ0.126-0.047=0.080
min1267.437-1320.137=-52.700
max1267.776-1320.257=-52.481
mean1092.400-1067.600=24.800
σ169.214-164.483=4.730
min890.000-845.000=45.000
max1322.000-1268.000=54.000
mean2.172-2.249=-0.077
σ0.277-0.124=0.153
min1.954-2.147=-0.193
max2.654-2.456=0.197
iters=5
new-facts=6402
prior-facts=1
mean1506.545-1560.617=-54.072
σ2.924-6.226=-3.302
min1505.037-1557.635=-52.599
max1511.763-1571.750=-59.987
mean611.600-619.600=-8.000
σ92.985-82.984=10.002
min499.000-494.000=5.000
max731.000-717.000=14.000
mean2.223-2.304=-0.081
σ0.135-0.188=-0.053
min2.076-2.181=-0.105
max2.389-2.623=-0.234
iters=5
new-facts=6402
prior-facts=6402
mean1829.568-1882.168=-52.600
σ0.051-0.056=-0.005
min1829.510-1882.105=-52.595
max1829.615-1882.241=-52.626
mean1722.000-1708.600=13.400
σ190.604-258.907=-68.303
min1458.000-1396.000=62.000
max1898.000-2056.000=-158.000
mean2.927-2.929=-0.002
σ0.275-0.076=0.199
min2.605-2.842=-0.237
max3.361-3.013=0.348

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.657-1.841=-0.184
σ0.207-0.050=0.157
min1.316-1.755=-0.439
max1.845-1.874=-0.029

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean393.630-452.082=-58.451
σ509.937-582.721=-72.784
min138.659-160.720=-22.061
max1158.536-1326.164=-167.628
mean97.750-102.250=-4.500
σ7.500-8.500=-1.000
min94.000-98.000=-4.000
max109.000-115.000=-6.000
mean435.000-441.000=-6.000
σ0.000
min435.000-441.000=-6.000
max435.000-441.000=-6.000
mean0.479-0.541=-0.062
σ0.492-0.542=-0.050
min0.229-0.268=-0.039
max1.217-1.354=-0.136
policies=20
mean1386.719-1607.527=-220.808
σ0.134-0.413=-0.278
min1386.622-1607.247=-220.625
max1386.918-1608.139=-221.221
mean940.000-980.000=-40.000
σ0.000
min940.000-980.000=-40.000
max940.000-980.000=-40.000
mean443.000-445.000=-2.000
σ9.238-4.619=4.619
min435.000-441.000=-6.000
max451.000-449.000=2.000
mean2.282-2.694=-0.412
σ0.007-0.017=-0.010
min2.272-2.677=-0.405
max2.289-2.714=-0.424
policies=200
mean13875.869-16083.364=-2207.495
σ0.104-0.167=-0.062
min13875.773-16083.200=-2207.426
max13876.017-16083.533=-2207.516
mean9400.000-9800.000=-400.000
σ0.000
min9400.000-9800.000=-400.000
max9400.000-9800.000=-400.000
mean493.250-493.750=-0.500
σ4.425-4.031=0.394
min489.000
max498.000
mean23.252-27.146=-3.894
σ0.149-0.033=0.115
min23.120-27.115=-3.995
max23.394-27.193=-3.799