Compare commits

Home / Compare dae43b63 .. 13a96a26

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.301
σ0.000
min5.301
max5.301

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean5.176
σ0.000
min5.176
max5.176

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean5.676
σ0.000
min5.676
max5.676

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.883-558.664=-111.781
σ0.000
min446.883-558.664=-111.781
max446.883-558.664=-111.781

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.488
σ0.000
min5.488
max5.488

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.645
σ0.000
min6.645
max6.645

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.860-54.890=-0.030
σ0.357-0.357=0.001
min54.724-54.754=-0.030
max55.876-55.905=-0.029
mean416.000-415.000=1.000
σ0.000
min416.000-415.000=1.000
max416.000-415.000=1.000
mean0.137-0.150=-0.013
σ0.006-0.011=-0.004
min0.131-0.135=-0.005
max0.149-0.160=-0.011
cache=y
dps=1
iters=10
policies=100
mean43.847-43.858=-0.011
σ0.007-0.006=0.001
min43.834-43.848=-0.014
max43.859-43.866=-0.007
mean416.000-415.000=1.000
σ0.000
min416.000-415.000=1.000
max416.000-415.000=1.000
mean0.125-0.133=-0.008
σ0.005-0.009=-0.004
min0.118-0.122=-0.004
max0.133-0.147=-0.014

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.617-36.618=-0.001
σ0.000-0.002=-0.002
min36.617-36.616=0.001
max36.618-36.620=-0.003
mean4419.500-4041.625=377.875
σ2227.590-2006.816=220.774
min1487.000-1219.000=268.000
max7405.000-7140.000=265.000
mean12.868-13.475=-0.607
σ3.446-1.750=1.696
min11.386-12.445=-1.060
max21.391-17.748=3.642
op=column_set() on indexed, repeat
mean36.618-36.618=-0.001
σ0.000-0.002=-0.002
min36.617-36.616=0.001
max36.618-36.620=-0.002
mean4601.125-4041.625=559.500
σ2258.516-2006.816=251.699
min1487.000-1219.000=268.000
max7405.000-7140.000=265.000
mean179.920-13.690=166.230
σ471.166-1.836=469.330
min11.248-12.223=-0.975
max1345.971-18.075=1327.896
op=table_records(), no indexes
mean41.948-41.930=0.018
σ0.002-0.002=-0.000
min41.946-41.928=0.018
max41.950-41.933=0.017
mean7405.000-7140.000=265.000
σ0.000
min7405.000-7140.000=265.000
max7405.000-7140.000=265.000
mean10.539-15.090=-4.551
σ0.124-0.115=0.010
min10.334-14.962=-4.628
max10.760-15.296=-4.536
op=table_records(), repeat
mean41.933-41.929=0.004
σ0.000-0.002=-0.002
min41.933-41.927=0.005
max41.934-41.932=0.001
mean4190.250-4041.625=148.625
σ1954.021-2006.816=-52.795
min1487.000-1219.000=268.000
max7405.000-7140.000=265.000
mean11.022-14.648=-3.626
σ1.780-1.372=0.408
min10.214-11.965=-1.751
max15.416-16.282=-0.866
op=vertical_lookup_list() on indexed, after table_records()
mean27.655-27.653=0.002
σ0.136-0.136=0.000
min27.606-27.604=0.002
max27.993-27.991=0.002
mean4190.250-4041.625=148.625
σ1954.021-2006.816=-52.795
min1487.000-1219.000=268.000
max7405.000-7140.000=265.000
mean8.534-9.893=-1.358
σ1.258-1.525=-0.267
min7.585-8.630=-1.045
max11.598-12.544=-0.946
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.406-26.401=0.005
σ0.000-0.002=-0.002
min26.406-26.400=0.006
max26.406-26.404=0.002
mean4190.250-4041.625=148.625
σ1954.021-2006.816=-52.795
min1487.000-1219.000=268.000
max7405.000-7140.000=265.000
mean6.563-7.648=-1.085
σ0.932-1.306=-0.374
min5.953-6.681=-0.728
max8.844-10.506=-1.663
op=vertical_lookup_row_list() on 2 indexed
mean25.681-25.668=0.013
σ0.003-0.004=-0.001
min25.680-25.665=0.014
max25.688-25.678=0.010
mean4601.125-4041.625=559.500
σ2258.516-2006.816=251.699
min1487.000-1219.000=268.000
max7405.000-7140.000=265.000
mean6.062-6.219=-0.158
σ0.975-0.570=0.405
min5.438-5.652=-0.213
max7.967-7.576=0.391
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.984-25.970=0.015
σ0.000-0.002=-0.001
min25.984-25.968=0.016
max25.985-25.972=0.013
mean4601.125-4041.625=559.500
σ2258.516-2006.816=251.699
min1487.000-1219.000=268.000
max7405.000-7140.000=265.000
mean5.881-6.021=-0.139
σ1.015-0.587=0.427
min5.266-5.607=-0.342
max8.119-7.452=0.667

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.627-1021.613=0.014
σ0.002-0.002=0.000
min1021.625-1021.612=0.013
max1021.630-1021.616=0.014
mean7405.000-7140.000=265.000
σ0.000
min7405.000-7140.000=265.000
max7405.000-7140.000=265.000
mean346.308-362.790=-16.482
σ1.850-42.668=-40.818
min342.373-345.159=-2.786
max348.289-468.301=-120.012
op=load_product(), with indexes
mean3495.878-7917.793=-4421.914
σ0.469-0.469=-0.001
min3495.713-7917.624=-4421.912
max3497.038-7918.954=-4421.916
mean3765.625-3274.500=491.125
σ2308.847-1823.440=485.407
min721.000-810.000=-89.000
max7405.000-5622.000=1783.000
mean1763.119-3686.639=-1923.520
σ629.015-1083.458=-454.444
min1388.788-3086.782=-1697.995
max2793.689-6179.639=-3385.950

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense
mean62.077
σ0.002
min62.076
max62.080
mean4041.625
σ2006.816
min1219.000
max7140.000
mean88.802
σ18.457
min71.172
max113.883
op=column_set() on unindexed, dense, minor warmup
mean107.854
σ0.000
min107.854
max107.854
mean4190.250
σ1954.021
min1487.000
max7405.000
mean134.913
σ17.303
min124.710
max177.349
op=column_set() on unindexed, dense, repeat
mean62.082
σ0.000
min62.081
max62.082
mean4419.500
σ2227.590
min1487.000
max7405.000
mean181.375
σ390.860
min41.103
max1148.696
op=vertical_lookup_list() on unindexed missing Int
mean40.946
σ0.002
min40.945
max40.950
mean7140.000
σ0.000
min7140.000
max7140.000
mean61.562
σ2.476
min60.314
max67.633
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.948
σ0.002
min40.947
max40.951
mean7405.000
σ0.000
min7405.000
max7405.000
mean113.468
σ261.077
min20.856
max759.601
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.308-66.688=4.620
σ0.002-0.002=-0.001
min71.307-66.686=4.620
max71.311-66.691=4.620
mean7405.000-7140.000=265.000
σ0.000
min7405.000-7140.000=265.000
max7405.000-7140.000=265.000
mean57.301-262.855=-205.554
σ6.320-366.960=-360.639
min54.868-97.886=-43.018
max72.939-1141.187=-1068.248

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.850-14217.171=-9570.321
σ0.014-4.303=-4.289
min4646.845-14215.576=-9568.731
max4646.884-14227.820=-9580.936
mean4190.250-4041.625=148.625
σ1954.021-2006.816=-52.795
min1487.000-1219.000=268.000
max7405.000-7140.000=265.000
mean3146.092-7651.513=-4505.421
σ782.769-1952.033=-1169.263
min2518.774-6010.842=-3492.067
max4607.487-11234.867=-6627.381
op=vertical_lookup_list() on indexed
mean1973.851-13159.109=-11185.257
σ0.002-0.002=0.000
min1973.849-13159.106=-11185.256
max1973.853-13159.110=-11185.256
mean7405.000-7140.000=265.000
σ0.000
min7405.000-7140.000=265.000
max7405.000-7140.000=265.000
mean965.172-5425.132=-4459.960
σ71.421-1008.866=-937.445
min922.783-4997.880=-4075.097
max1099.293-7890.995=-6791.702
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.105
σ0.002
min905.102
max905.108
mean7405.000
σ0.000
min7405.000
max7405.000
mean386.579
σ4.502
min376.939
max391.770

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144-172.559=-24.414
σ0.001
min148.144-172.558=-24.414
max148.148-172.562=-24.414
mean476.000-477.000=-1.000
σ0.000
min476.000-477.000=-1.000
max476.000-477.000=-1.000
mean361.498-390.133=-28.635
σ44.082-5.849=38.233
min301.525-380.224=-78.699
max434.412-396.695=37.717
op=pointedIndexFromKvs
rows=400000
mean308.493-357.322=-48.828
σ0.001
min308.493-357.321=-48.828
max308.497-357.325=-48.828
mean565.000-562.000=3.000
σ0.000
min565.000-562.000=3.000
max565.000-562.000=3.000
mean768.285-743.496=24.789
σ41.236-66.306=-25.070
min703.012-591.985=111.027
max819.336-799.223=20.113

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean1024.564-1024.299=0.264
σ0.761-0.758=0.002
min1024.092-1023.826=0.265
max1027.706-1027.435=0.270
mean1035.150-1053.450=-18.300
σ298.917-312.440=-13.523
min552.000-570.000=-18.000
max1438.000-1511.000=-73.000
mean1.366-1.550=-0.184
σ0.069-0.060=0.009
min1.293-1.480=-0.188
max1.573-1.733=-0.160

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913-0.913=0.000
σ0.020
min0.903-0.903=0.000
max0.949-0.949=0.000
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.446-0.326=0.120
σ0.468-0.342=0.126
min0.172-0.142=0.030
max1.280-0.932=0.348
name=error_case.art
mean2.340-2.339=0.001
σ0.012
min2.334-2.333=0.001
max2.362-2.361=0.001
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.825-0.414=0.410
σ0.520-0.190=0.330
min0.576-0.319=0.257
max1.754-0.754=1.000
name=heavy_item.art
mean1.523-1.522=0.000
σ0.002
min1.521-1.521=0.000
max1.525-1.525=0.000
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.376-0.221=0.155
σ0.032-0.020=0.012
min0.349-0.205=0.144
max0.430-0.255=0.175
name=nested_use.art
mean2.358-2.357=0.001
σ0.002
min2.357-2.356=0.001
max2.361-2.360=0.001
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.589-0.332=0.258
σ0.027-0.022=0.005
min0.564-0.314=0.250
max0.634-0.369=0.265

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.971-964.840=0.130
σ0.011
min964.965-964.835=0.130
max964.989-964.859=0.130
mean431.600-433.600=-2.000
σ1.342-4.099=-2.757
min431.000-430.000=1.000
max434.000-438.000=-4.000
mean0.571-0.530=0.041
σ0.109-0.065=0.044
min0.466-0.442=0.025
max0.749-0.602=0.147
name=cdr_enhanced_4k_loc.art
mean885.898-885.820=0.078
σ0.002
min885.897-885.819=0.078
max885.901-885.823=0.078
mean441.000-440.000=1.000
σ0.000
min441.000-440.000=1.000
max441.000-440.000=1.000
mean0.466-0.514=-0.048
σ0.060-0.022=0.038
min0.372-0.477=-0.105
max0.521-0.528=-0.008

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.710-7.709=0.001
σ0.002
min7.709-7.708=0.001
max7.713-7.712=0.001
mean441.000-440.000=1.000
σ0.000
min441.000-440.000=1.000
max441.000-440.000=1.000
mean1.977-2.620=-0.643
σ0.068-0.117=-0.049
min1.847-2.487=-0.640
max2.278-3.399=-1.121
name=record_render_large_syntax_err.art
mean7.658-7.657=0.001
σ0.002
min7.657-7.656=0.001
max7.661-7.660=0.001
mean441.000-440.000=1.000
σ0.000
min441.000-440.000=1.000
max441.000-440.000=1.000
mean1.955-2.598=-0.643
σ0.064-0.102=-0.037
min1.839-2.450=-0.611
max2.202-2.996=-0.795
name=record_render_success.art
mean2.290-2.289=0.000
σ0.002
min2.289-2.289=0.000
max2.293-2.293=0.000
mean441.000-440.000=1.000
σ0.000
min441.000-440.000=1.000
max441.000-440.000=1.000
mean0.303-0.303=-0.000
σ0.008-0.012=-0.004
min0.287-0.289=-0.001
max0.351-0.368=-0.016
name=record_render_syntax_err.art
mean2.165-2.165=0.000
σ0.002
min2.164-2.164=0.000
max2.168-2.168=0.000
mean441.000-440.000=1.000
σ0.000
min441.000-440.000=1.000
max441.000-440.000=1.000
mean0.290-0.292=-0.002
σ0.006-0.008=-0.002
min0.277-0.279=-0.002
max0.319-0.341=-0.022

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.610
σ11.416
min0.574
max87.127
mean467.563
σ21.142
min408.000
max483.000
mean0.027
σ0.028
min0.001
max0.222

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean298.720-298.730=-0.010
σ0.989-1.035=-0.045
min298.298-298.299=-0.000
max301.524-301.666=-0.142
mean442.800-445.800=-3.000
σ17.831-17.850=-0.019
min415.000-417.000=-2.000
max462.000-469.000=-7.000
mean0.602-0.601=0.001
σ0.021-0.025=-0.004
min0.559-0.553=0.006
max0.641-0.639=0.002

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean173.738-236.678=-62.940
σ0.000-0.000=0.000
min173.738-236.678=-62.940
max173.738-236.678=-62.940
mean844.000-1067.625=-223.625
σ0.000-26.517=-26.517
min844.000-1002.000=-158.000
max844.000-1077.000=-233.000
mean224.416-201.791=22.625
σ28.197-4.104=24.093
min193.725-197.078=-3.353
max263.615-206.945=56.670
phase=macro expansion
mean6.133-6.130=0.004
σ0.000
min6.133-6.130=0.004
max6.133-6.130=0.004
mean843.375-1067.625=-224.250
σ1.768-26.517=-24.749
min839.000-1002.000=-163.000
max844.000-1077.000=-233.000
mean6.812-8.459=-1.647
σ1.089-2.138=-1.049
min6.260-6.411=-0.151
max9.496-12.875=-3.380
phase=parser
mean1042.431-1042.361=0.071
σ0.000-0.000=0.000
min1042.431-1042.360=0.070
max1042.431-1042.361=0.071
mean843.375-1067.625=-224.250
σ1.768-26.517=-24.749
min839.000-1002.000=-163.000
max844.000-1077.000=-233.000
mean571.450-509.787=61.663
σ52.343-107.912=-55.570
min449.199-413.877=35.322
max607.833-755.118=-147.284
phase=processProdDef
mean2243.980-4850.768=-2606.788
σ0.000
min2243.980-4850.768=-2606.788
max2243.980-4850.768=-2606.788
mean844.000-1077.000=-233.000
σ0.000
min844.000-1077.000=-233.000
max844.000-1077.000=-233.000
mean2650.825-4702.815=-2051.990
σ402.780-683.401=-280.620
min2288.236-3517.916=-1229.680
max3470.878-5496.386=-2025.508
phase=renaming
mean21.671-21.675=-0.004
σ0.000
min21.671-21.675=-0.004
max21.671-21.675=-0.004
mean843.375-1067.625=-224.250
σ1.768-26.517=-24.749
min839.000-1002.000=-163.000
max844.000-1077.000=-233.000
mean23.157-31.199=-8.042
σ6.781-10.319=-3.537
min20.479-20.735=-0.256
max39.933-48.841=-8.908
phase=type checker
mean989.752-990.450=-0.698
σ0.066-0.071=-0.005
min989.654-990.344=-0.690
max989.841-990.563=-0.722
mean843.375-1067.625=-224.250
σ1.768-26.517=-24.749
min839.000-1002.000=-163.000
max844.000-1077.000=-233.000
mean690.386-710.609=-20.224
σ56.258-105.444=-49.186
min642.812-638.885=3.927
max820.836-906.354=-85.518

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean293.821-293.822=-0.001
σ0.037-0.038=-0.001
min293.768-293.770=-0.001
max293.859-293.862=-0.003
mean443.000-439.800=3.200
σ0.000-1.095=-1.095
min443.000-439.000=4.000
max443.000-441.000=2.000
mean0.392-0.393=-0.001
σ0.005-0.001=0.004
min0.389-0.391=-0.003
max0.400-0.394=0.006
iters=5
new-facts=1
prior-facts=6402
mean1320.290-1320.222=0.067
σ0.139-0.034=0.104
min1320.167-1320.166=0.001
max1320.519-1320.254=0.265
mean1115.000-1081.400=33.600
σ178.027-170.866=7.160
min896.000-877.000=19.000
max1349.000-1311.000=38.000
mean1.958-2.255=-0.297
σ0.036-0.097=-0.061
min1.928-2.158=-0.230
max2.012-2.413=-0.402
iters=5
new-facts=6402
prior-facts=1
mean1559.279-1560.592=-1.313
σ2.967-6.213=-3.245
min1557.709-1557.611=0.098
max1564.574-1571.701=-7.127
mean623.600-609.000=14.600
σ87.540-93.995=-6.454
min493.000-496.000=-3.000
max726.000-732.000=-6.000
mean2.210-2.225=-0.015
σ0.127-0.125=0.002
min2.086-2.023=0.063
max2.416-2.353=0.063
iters=5
new-facts=6402
prior-facts=6402
mean1882.369-1882.171=0.198
σ0.075-0.059=0.016
min1882.265-1882.097=0.168
max1882.459-1882.248=0.211
mean1754.600-1722.000=32.600
σ210.681-174.452=36.229
min1473.000-1464.000=9.000
max1954.000-1875.000=79.000
mean3.204-3.096=0.109
σ0.469-0.254=0.215
min2.776-2.890=-0.114
max3.800-3.491=0.309

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.808-1.805=0.002
σ0.054-0.106=-0.053
min1.761-1.632=0.129
max1.879-1.920=-0.041

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean419.944-452.082=-32.138
σ518.103-582.723=-64.619
min160.888-160.719=0.169
max1197.099-1326.166=-129.067
mean101.750-102.250=-0.500
σ7.500-8.500=-1.000
min98.000
max113.000-115.000=-2.000
mean446.000-442.000=4.000
σ0.000
min446.000-442.000=4.000
max446.000-442.000=4.000
mean0.506-0.550=-0.044
σ0.486-0.546=-0.060
min0.256-0.273=-0.018
max1.234-1.368=-0.134
policies=20
mean1609.209-1607.534=1.675
σ0.367-0.428=-0.061
min1608.960-1607.242=1.717
max1609.754-1608.168=1.585
mean980.000
σ0.000
min980.000
max980.000
mean449.250-450.500=-1.250
σ6.500-9.815=-3.315
min446.000-442.000=4.000
max459.000
mean2.599-2.705=-0.106
σ0.014-0.017=-0.003
min2.588-2.691=-0.102
max2.619-2.723=-0.104
policies=200
mean16100.656-16083.545=17.112
σ0.061-0.079=-0.018
min16100.581-16083.497=17.084
max16100.723-16083.663=17.060
mean9800.000
σ0.000
min9800.000
max9800.000
mean497.750-500.000=-2.250
σ4.573-4.690=-0.117
min492.000-495.000=-3.000
max503.000-506.000=-3.000
mean26.057-27.226=-1.170
σ0.071-0.074=-0.003
min25.972-27.158=-1.186
max26.145-27.302=-1.157