Compare commits

Home / Compare cc634dd7 .. a7c86e55

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.082-5.301=-0.219
σ0.000
min5.082-5.301=-0.219
max5.082-5.301=-0.219

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean4.988-5.176=-0.188
σ0.000
min4.988-5.176=-0.188
max4.988-5.176=-0.188

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean5.395
σ0.000
min5.395
max5.395

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.633-558.633=-112.000
σ0.000
min446.633-558.633=-112.000
max446.633-558.633=-112.000

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.238-5.488=-0.250
σ0.000
min5.238-5.488=-0.250
max5.238-5.488=-0.250

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.395-6.645=-0.250
σ0.000
min6.395-6.645=-0.250
max6.395-6.645=-0.250

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.717-54.880=-0.163
σ0.365-0.358=0.006
min54.584-54.741=-0.157
max55.754-55.900=-0.145
mean416.000-416.700=-0.700
σ0.000-0.483=-0.483
min416.000
max416.000-417.000=-1.000
mean0.140-0.139=0.000
σ0.005-0.004=0.001
min0.133-0.136=-0.003
max0.147-0.147=-0.001
cache=y
dps=1
iters=10
policies=100
mean43.778-43.848=-0.070
σ0.008-0.007=0.000
min43.767-43.838=-0.072
max43.789-43.861=-0.072
mean416.000-417.000=-1.000
σ0.000
min416.000-417.000=-1.000
max416.000-417.000=-1.000
mean0.125-0.124=0.001
σ0.004-0.004=-0.000
min0.122-0.120=0.002
max0.132-0.132=0.000

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.604-36.618=-0.014
σ0.000-0.002=-0.002
min36.604-36.616=-0.012
max36.605-36.621=-0.016
mean4581.625-4042.500=539.125
σ2257.603-2007.721=249.882
min1485.000-1216.000=269.000
max7406.000-7139.000=267.000
mean14.000-12.720=1.279
σ3.967-0.212=3.755
min11.740-12.457=-0.716
max22.618-13.107=9.510
op=column_set() on indexed, repeat
mean36.604-36.618=-0.014
σ0.000-0.002=-0.002
min36.604-36.616=-0.012
max36.605-36.620=-0.016
mean4581.625-4042.500=539.125
σ2257.603-2007.721=249.882
min1485.000-1216.000=269.000
max7406.000-7139.000=267.000
mean14.324-12.576=1.748
σ3.927-0.628=3.298
min11.728-11.760=-0.033
max22.695-13.367=9.328
op=table_records(), no indexes
mean41.939-41.930=0.009
σ0.002-0.002=0.001
min41.938-41.928=0.009
max41.943-41.932=0.010
mean7406.000-7139.000=267.000
σ0.000
min7406.000-7139.000=267.000
max7406.000-7139.000=267.000
mean10.976-15.087=-4.111
σ0.174-0.219=-0.046
min10.652-14.807=-4.155
max11.127-15.536=-4.408
op=table_records(), repeat
mean41.925-41.930=-0.005
σ0.000-0.002=-0.002
min41.924-41.927=-0.003
max41.925-41.932=-0.007
mean4165.500-4042.500=123.000
σ1948.247-2007.721=-59.474
min1485.000-1216.000=269.000
max7406.000-7139.000=267.000
mean11.624-14.176=-2.552
σ1.871-2.455=-0.584
min10.747-10.932=-0.185
max16.239-18.798=-2.559
op=vertical_lookup_list() on indexed, after table_records()
mean27.633-27.654=-0.021
σ0.136-0.136=0.000
min27.584-27.604=-0.020
max27.971-27.991=-0.020
mean4165.500-4042.500=123.000
σ1948.247-2007.721=-59.474
min1485.000-1216.000=269.000
max7406.000-7139.000=267.000
mean9.094-9.491=-0.397
σ1.290-1.388=-0.097
min8.303-8.515=-0.211
max12.251-12.833=-0.582
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.384-26.401=-0.017
σ0.000-0.002=-0.002
min26.384-26.400=-0.016
max26.384-26.404=-0.020
mean4165.500-4042.500=123.000
σ1948.247-2007.721=-59.474
min1485.000-1216.000=269.000
max7406.000-7139.000=267.000
mean7.224-7.344=-0.120
σ0.912-1.280=-0.368
min6.570-6.578=-0.007
max9.423-10.430=-1.006
op=vertical_lookup_row_list() on 2 indexed
mean25.662-25.668=-0.006
σ0.004-0.004=0.000
min25.661-25.666=-0.005
max25.673-25.678=-0.005
mean4581.625-4042.500=539.125
σ2257.603-2007.721=249.882
min1485.000-1216.000=269.000
max7406.000-7139.000=267.000
mean6.483-6.030=0.453
σ1.033-0.064=0.969
min5.740-5.921=-0.181
max8.646-6.127=2.519
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.961-25.970=-0.009
σ0.001-0.002=-0.001
min25.961-25.969=-0.008
max25.962-25.972=-0.010
mean4581.625-4042.500=539.125
σ2257.603-2007.721=249.882
min1485.000-1216.000=269.000
max7406.000-7139.000=267.000
mean6.316-5.801=0.515
σ1.039-0.180=0.859
min5.613-5.405=0.208
max8.684-6.000=2.685

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.608-1021.613=-0.005
σ0.002-0.002=0.000
min1021.606-1021.612=-0.006
max1021.611-1021.616=-0.005
mean7406.000-7139.000=267.000
σ0.000
min7406.000-7139.000=267.000
max7406.000-7139.000=267.000
mean376.150-375.181=0.969
σ80.883-85.018=-4.135
min344.729-343.332=1.396
max576.283-585.559=-9.276
op=load_product(), with indexes
mean3495.854-7917.792=-4421.938
σ0.453-0.469=-0.016
min3495.694-7917.624=-4421.930
max3496.977-7918.954=-4421.977
mean3745.875-3275.250=470.625
σ2299.833-1825.405=474.428
min720.000-810.000=-90.000
max7406.000-5622.000=1784.000
mean1646.193-3583.393=-1937.200
σ536.376-1007.733=-471.357
min1400.207-3077.694=-1677.487
max2958.324-5901.757=-2943.434

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense
mean62.077
σ0.002
min62.076
max62.080
mean4042.500
σ2007.721
min1216.000
max7139.000
mean85.557
σ19.829
min70.831
max123.287
op=column_set() on unindexed, dense, minor warmup
mean107.841
σ0.000
min107.841
max107.842
mean4400.625
σ2229.029
min1485.000
max7406.000
mean284.551
σ420.422
min125.262
max1324.427
op=column_set() on unindexed, dense, repeat
mean62.068
σ0.001
min62.065
max62.069
mean4581.625
σ2257.603
min1485.000
max7406.000
mean193.075
σ414.152
min41.184
max1217.804
op=vertical_lookup_list() on unindexed missing Int
mean40.946
σ0.002
min40.945
max40.949
mean7139.000
σ0.000
min7139.000
max7139.000
mean60.509
σ0.388
min59.995
max60.979
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.926
σ0.002
min40.925
max40.928
mean7406.000
σ0.000
min7406.000
max7406.000
mean22.566
σ1.911
min21.414
max26.098
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.294-66.688=4.606
σ0.002-0.002=-0.000
min71.292-66.686=4.606
max71.296-66.691=4.605
mean7406.000-7139.000=267.000
σ0.000
min7406.000-7139.000=267.000
max7406.000-7139.000=267.000
mean277.129-98.910=178.218
σ621.623-0.717=620.906
min55.590-97.984=-42.394
max1815.553-100.069=1715.484

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.842-14217.171=-9570.329
σ0.014-4.304=-4.290
min4646.837-14215.575=-9568.738
max4646.876-14227.820=-9580.944
mean4165.500-4042.500=123.000
σ1948.247-2007.721=-59.474
min1485.000-1216.000=269.000
max7406.000-7139.000=267.000
mean3137.392-7956.007=-4818.615
σ835.535-1865.162=-1029.627
min2521.886-5954.659=-3432.773
max4759.232-10475.838=-5716.606
op=vertical_lookup_list() on indexed
mean1973.829-13159.108=-11185.279
σ0.002-0.002=0.000
min1973.827-13159.106=-11185.279
max1973.832-13159.110=-11185.278
mean7406.000-7139.000=267.000
σ0.000
min7406.000-7139.000=267.000
max7406.000-7139.000=267.000
mean1104.768-5103.603=-3998.834
σ300.592-197.871=102.721
min923.950-4969.090=-4045.140
max1765.458-5440.038=-3674.580
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.082
σ0.002
min905.080
max905.086
mean7406.000
σ0.000
min7406.000
max7406.000
mean408.702
σ39.596
min387.139
max505.495

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144-172.559=-24.414
σ0.001
min148.144-172.558=-24.414
max148.148-172.562=-24.414
mean478.000
σ0.000
min478.000
max478.000
mean387.799-370.675=17.124
σ22.775-35.110=-12.335
min361.714-291.912=69.801
max423.452-390.663=32.789
op=pointedIndexFromKvs
rows=400000
mean308.493-357.322=-48.828
σ0.001
min308.493-357.321=-48.828
max308.497-357.325=-48.828
mean563.500-560.000=3.500
σ0.926-0.000=0.926
min562.000-560.000=2.000
max564.000-560.000=4.000
mean777.704-751.837=25.866
σ42.358-45.070=-2.712
min721.580-642.716=78.863
max841.741-780.601=61.141

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean964.562-1024.299=-59.737
σ1.879-0.759=1.120
min963.843-1023.827=-59.984
max972.510-1027.437=-54.927
mean1041.900-1040.150=1.750
σ305.833-291.001=14.833
min587.000-561.000=26.000
max1490.000-1434.000=56.000
mean1.275-1.541=-0.266
σ0.145-0.076=0.069
min1.082-1.480=-0.398
max1.740-1.767=-0.027

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913-0.913=0.001
σ0.020-0.020=-0.000
min0.903-0.903=0.001
max0.949-0.949=0.000
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.292-0.463=-0.171
σ0.289-0.489=-0.200
min0.136-0.162=-0.026
max0.804-1.330=-0.527
name=error_case.art
mean2.346-2.339=0.007
σ0.012
min2.340-2.333=0.007
max2.368-2.361=0.007
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.472-0.820=-0.348
σ0.317-0.508=-0.192
min0.316-0.571=-0.255
max1.038-1.729=-0.690
name=heavy_item.art
mean1.523-1.522=0.001
σ0.002-0.002=-0.000
min1.522-1.521=0.001
max1.526-1.525=0.001
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.214-0.391=-0.177
σ0.021-0.029=-0.009
min0.199-0.368=-0.170
max0.250-0.441=-0.191
name=nested_use.art
mean2.364-2.357=0.007
σ0.002
min2.363-2.356=0.007
max2.367-2.360=0.007
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.331-0.586=-0.255
σ0.020-0.029=-0.009
min0.312-0.562=-0.251
max0.362-0.636=-0.274

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.979-964.840=0.139
σ0.009-0.011=-0.001
min964.974-964.835=0.139
max964.996-964.859=0.137
mean431.000-429.000=2.000
σ0.000
min431.000-429.000=2.000
max431.000-429.000=2.000
mean0.542-0.572=-0.030
σ0.041-0.074=-0.033
min0.492-0.483=0.009
max0.582-0.679=-0.097
name=cdr_enhanced_4k_loc.art
mean886.001-885.820=0.181
σ0.002-0.002=-0.000
min886.000-885.819=0.181
max886.004-885.823=0.181
mean431.000-429.000=2.000
σ0.000
min431.000-429.000=2.000
max431.000-429.000=2.000
mean0.473-0.453=0.020
σ0.028-0.035=-0.007
min0.456-0.416=0.040
max0.523-0.501=0.022

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.719-7.709=0.010
σ0.002
min7.718-7.708=0.010
max7.722-7.712=0.010
mean431.000-429.000=2.000
σ0.000
min431.000-429.000=2.000
max431.000-429.000=2.000
mean2.696-2.634=0.063
σ0.110-0.093=0.017
min2.465-2.466=-0.000
max2.959-2.991=-0.032
name=record_render_large_syntax_err.art
mean7.667-7.657=0.010
σ0.002
min7.666-7.656=0.010
max7.670-7.660=0.010
mean431.000-429.000=2.000
σ0.000
min431.000-429.000=2.000
max431.000-429.000=2.000
mean2.621-2.635=-0.014
σ0.081-0.107=-0.026
min2.465-2.432=0.033
max2.915-3.140=-0.225
name=record_render_success.art
mean2.290-2.289=0.001
σ0.002
min2.290-2.289=0.001
max2.294-2.293=0.001
mean431.000-429.000=2.000
σ0.000
min431.000-429.000=2.000
max431.000-429.000=2.000
mean0.306-0.307=-0.001
σ0.015-0.009=0.005
min0.283-0.292=-0.009
max0.359-0.367=-0.008
name=record_render_syntax_err.art
mean2.166-2.165=0.001
σ0.002
min2.165-2.164=0.001
max2.169-2.168=0.001
mean431.000-429.000=2.000
σ0.000
min431.000-429.000=2.000
max431.000-429.000=2.000
mean0.295-0.297=-0.001
σ0.042-0.008=0.034
min0.277-0.282=-0.005
max0.692-0.339=0.354

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.801
σ11.829
min0.574
max109.330
mean456.713
σ17.129
min408.000
max479.000
mean0.027
σ0.029
min0.001
max0.259

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean279.028-298.637=-19.610
σ1.064-1.000=0.065
min278.592-298.190=-19.598
max282.051-301.469=-19.417
mean441.700-441.500=0.200
σ15.464-12.295=3.169
min417.000-425.000=-8.000
max467.000-462.000=5.000
mean0.573-0.606=-0.032
σ0.028-0.018=0.011
min0.513-0.587=-0.074
max0.611-0.644=-0.032

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean169.257-236.678=-67.420
σ0.000-0.000=0.000
min169.257-236.678=-67.420
max169.257-236.678=-67.420
mean851.000-1065.000=-214.000
σ0.000-33.941=-33.941
min851.000-981.000=-130.000
max851.000-1077.000=-226.000
mean245.626-206.949=38.677
σ10.794-3.538=7.256
min221.507-198.643=22.864
max253.232-209.726=43.506
phase=macro expansion
mean5.896-6.130=-0.234
σ0.000
min5.896-6.130=-0.234
max5.896-6.130=-0.234
mean842.125-1065.000=-222.875
σ25.102-33.941=-8.839
min780.000-981.000=-201.000
max851.000-1077.000=-226.000
mean7.905-8.065=-0.160
σ1.321-2.148=-0.827
min5.984-6.360=-0.376
max9.170-12.704=-3.534
phase=parser
mean1042.133-1042.363=-0.230
σ0.000-0.000=-0.000
min1042.133-1042.362=-0.229
max1042.134-1042.363=-0.229
mean842.125-1065.000=-222.875
σ25.102-33.941=-8.839
min780.000-981.000=-201.000
max851.000-1077.000=-226.000
mean518.771-491.964=26.807
σ96.205-109.474=-13.269
min407.941-371.833=36.108
max619.212-725.292=-106.080
phase=processProdDef
mean2243.715-4850.764=-2607.049
σ0.000-0.000=-0.000
min2243.715-4850.764=-2607.049
max2243.715-4850.765=-2607.050
mean851.000-1077.000=-226.000
σ0.000
min851.000-1077.000=-226.000
max851.000-1077.000=-226.000
mean3030.714-4568.491=-1537.777
σ256.071-489.623=-233.553
min2681.535-3784.059=-1102.524
max3333.293-5302.566=-1969.272
phase=renaming
mean21.436-21.671=-0.235
σ0.000
min21.436-21.671=-0.235
max21.436-21.671=-0.235
mean842.125-1065.000=-222.875
σ25.102-33.941=-8.839
min780.000-981.000=-201.000
max851.000-1077.000=-226.000
mean40.764-28.840=11.924
σ5.589-10.416=-4.827
min32.717-20.279=12.438
max47.465-48.638=-1.173
phase=type checker
mean958.600-990.450=-31.850
σ0.047-0.071=-0.024
min958.577-990.344=-31.768
max958.702-990.563=-31.862
mean842.125-1065.000=-222.875
σ25.102-33.941=-8.839
min780.000-981.000=-201.000
max851.000-1077.000=-226.000
mean659.624-732.040=-72.416
σ14.675-114.030=-99.354
min637.524-640.846=-3.322
max680.044-887.069=-207.025

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean282.856-293.820=-10.964
σ0.036-0.038=-0.002
min282.804-293.767=-10.963
max282.891-293.860=-10.969
mean438.800-441.200=-2.400
σ3.834-1.643=2.191
min436.000-440.000=-4.000
max443.000
mean0.378-0.393=-0.015
σ0.002-0.002=-0.000
min0.376-0.392=-0.016
max0.381-0.397=-0.016
iters=5
new-facts=1
prior-facts=6402
mean1268.147-1320.197=-52.050
σ0.080-0.040=0.041
min1268.080-1320.160=-52.080
max1268.270-1320.243=-51.973
mean1092.600-1096.800=-4.200
σ170.132-196.710=-26.578
min891.000-841.000=50.000
max1325.000-1357.000=-32.000
mean2.229-2.259=-0.030
σ0.131-0.085=0.046
min2.094-2.155=-0.061
max2.399-2.367=0.032
iters=5
new-facts=6402
prior-facts=1
mean1507.012-1560.681=-53.669
σ2.979-6.187=-3.209
min1505.459-1557.771=-52.313
max1512.327-1571.747=-59.420
mean624.200-631.800=-7.600
σ87.622-96.552=-8.929
min493.000-519.000=-26.000
max726.000-756.000=-30.000
mean2.185-2.461=-0.276
σ0.257-0.233=0.024
min1.939-2.191=-0.252
max2.606-2.681=-0.075
iters=5
new-facts=6402
prior-facts=6402
mean1830.067-1882.169=-52.102
σ0.060-0.066=-0.006
min1830.000-1882.097=-52.097
max1830.130-1882.251=-52.121
mean1715.200-1732.000=-16.800
σ186.868-186.744=0.124
min1453.000-1488.000=-35.000
max1884.000-1909.000=-25.000
mean3.031-3.144=-0.113
σ0.337-0.347=-0.010
min2.749-2.889=-0.141
max3.510-3.744=-0.234

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.756-1.859=-0.103
σ0.110-0.049=0.061
min1.647-1.800=-0.153
max1.900-1.937=-0.037

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean406.199-452.079=-45.880
σ509.796-582.724=-72.928
min151.295-160.713=-9.418
max1170.892-1326.165=-155.273
mean101.750-102.250=-0.500
σ7.500-8.500=-1.000
min98.000
max113.000-115.000=-2.000
mean444.000-442.000=2.000
σ0.000
min444.000-442.000=2.000
max444.000-442.000=2.000
mean0.516-0.522=-0.007
σ0.520-0.503=0.017
min0.250-0.264=-0.014
max1.296-1.277=0.020
policies=20
mean1513.152-1607.501=-94.348
σ0.276-0.413=-0.137
min1512.975-1607.221=-94.246
max1513.561-1608.112=-94.551
mean980.000
σ0.000
min980.000
max980.000
mean446.000-450.000=-4.000
σ2.309-9.238=-6.928
min444.000-442.000=2.000
max448.000-458.000=-10.000
mean2.512-2.683=-0.171
σ0.014-0.017=-0.003
min2.498-2.666=-0.168
max2.530-2.705=-0.175
policies=200
mean15139.936-16083.400=-943.464
σ0.137-0.186=-0.049
min15139.749-16083.212=-943.463
max15140.044-16083.642=-943.598
mean9800.000
σ0.000
min9800.000
max9800.000
mean496.500-498.000=-1.500
σ6.608-3.367=3.242
min489.000-494.000=-5.000
max503.000-502.000=1.000
mean25.380-27.069=-1.689
σ0.118-0.065=0.053
min25.258-26.987=-1.729
max25.542-27.146=-1.604