Compare commits

Home / Compare 095244dc .. c9ab4e67

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.301
σ0.000
min5.301
max5.301

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean5.176
σ0.000
min5.176
max5.176

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean5.676
σ0.000
min5.676
max5.676

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.883-558.633=-111.750
σ0.000
min446.883-558.633=-111.750
max446.883-558.633=-111.750

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.488
σ0.000
min5.488
max5.488

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.645
σ0.000
min6.645
max6.645

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.861-54.881=-0.020
σ0.360-0.360=-0.000
min54.725-54.744=-0.019
max55.884-55.904=-0.020
mean415.000-419.000=-4.000
σ0.000
min415.000-419.000=-4.000
max415.000-419.000=-4.000
mean0.142-0.139=0.003
σ0.004-0.005=-0.001
min0.138-0.132=0.006
max0.148-0.147=0.002
cache=y
dps=1
iters=10
policies=100
mean43.848-43.851=-0.003
σ0.007-0.006=0.001
min43.837-43.843=-0.006
max43.857-43.861=-0.004
mean415.000-419.000=-4.000
σ0.000
min415.000-419.000=-4.000
max415.000-419.000=-4.000
mean0.126-0.128=-0.002
σ0.004-0.009=-0.005
min0.123-0.116=0.007
max0.133-0.148=-0.015

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.617-36.618=-0.001
σ0.000-0.002=-0.002
min36.617-36.616=0.001
max36.618-36.620=-0.003
mean4586.250-4040.875=545.375
σ2258.230-2008.944=249.286
min1485.000-1216.000=269.000
max7409.000-7151.000=258.000
mean13.445-12.887=0.559
σ3.642-1.694=1.947
min11.487-11.964=-0.478
max20.998-17.019=3.979
op=column_set() on indexed, repeat
mean36.617-36.618=-0.001
σ0.000-0.002=-0.002
min36.617-36.616=0.001
max36.618-36.620=-0.003
mean4586.250-4040.875=545.375
σ2258.230-2008.944=249.286
min1485.000-1216.000=269.000
max7409.000-7151.000=258.000
mean13.653-12.608=1.045
σ3.169-1.658=1.512
min11.070-11.337=-0.266
max20.414-16.491=3.923
op=table_records(), no indexes
mean41.948-41.930=0.018
σ0.002-0.002=0.000
min41.946-41.928=0.018
max41.951-41.933=0.018
mean7409.000-7151.000=258.000
σ0.000
min7409.000-7151.000=258.000
max7409.000-7151.000=258.000
mean10.416-14.682=-4.266
σ0.207-0.250=-0.043
min10.075-14.171=-4.096
max10.722-15.012=-4.289
op=table_records(), repeat
mean41.933-41.929=0.004
σ0.000-0.002=-0.002
min41.933-41.927=0.005
max41.934-41.932=0.002
mean4170.875-4040.875=130.000
σ1949.331-2008.944=-59.612
min1485.000-1216.000=269.000
max7409.000-7151.000=258.000
mean11.069-13.767=-2.699
σ1.846-1.492=0.355
min10.180-10.509=-0.329
max15.600-15.249=0.351
op=vertical_lookup_list() on indexed, after table_records()
mean27.655-27.654=0.002
σ0.136-0.136=0.000
min27.606-27.604=0.002
max27.993-27.991=0.002
mean4170.875-4040.875=130.000
σ1949.331-2008.944=-59.612
min1485.000-1216.000=269.000
max7409.000-7151.000=258.000
mean8.551-9.062=-0.511
σ1.365-1.255=0.110
min7.402-7.808=-0.406
max11.849-11.710=0.139
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.406-26.401=0.005
σ0.000-0.002=-0.002
min26.406-26.400=0.006
max26.406-26.404=0.002
mean4170.875-4040.875=130.000
σ1949.331-2008.944=-59.612
min1485.000-1216.000=269.000
max7409.000-7151.000=258.000
mean6.623-6.998=-0.376
σ0.976-1.072=-0.096
min5.885-6.008=-0.123
max8.987-9.205=-0.218
op=vertical_lookup_row_list() on 2 indexed
mean25.681-25.668=0.013
σ0.003-0.004=-0.001
min25.680-25.666=0.014
max25.688-25.678=0.010
mean4586.250-4040.875=545.375
σ2258.230-2008.944=249.286
min1485.000-1216.000=269.000
max7409.000-7151.000=258.000
mean6.080-5.794=0.286
σ0.931-0.578=0.353
min5.573-5.337=0.236
max8.212-7.168=1.044
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.984-25.970=0.015
σ0.000-0.002=-0.001
min25.984-25.968=0.016
max25.985-25.972=0.013
mean4586.250-4040.875=545.375
σ2258.230-2008.944=249.286
min1485.000-1216.000=269.000
max7409.000-7151.000=258.000
mean6.018-5.504=0.514
σ0.985-0.459=0.527
min5.375-5.249=0.126
max8.191-6.630=1.561

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.627-1021.613=0.014
σ0.002-0.002=0.000
min1021.625-1021.612=0.013
max1021.629-1021.616=0.013
mean7409.000-7151.000=258.000
σ0.000
min7409.000-7151.000=258.000
max7409.000-7151.000=258.000
mean361.776-365.015=-3.238
σ41.474-54.226=-12.752
min345.348-342.712=2.636
max464.394-499.036=-34.643
op=load_product(), with indexes
mean3495.878-7917.792=-4421.914
σ0.469-0.469=-0.001
min3495.713-7917.624=-4421.912
max3497.038-7918.953=-4421.916
mean3750.125-3272.500=477.625
σ2301.687-1821.892=479.795
min720.000-810.000=-90.000
max7409.000-5617.000=1792.000
mean1615.948-3573.564=-1957.615
σ458.692-987.069=-528.378
min1392.545-3073.891=-1681.346
max2734.288-5817.322=-3083.034

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense
mean62.077
σ0.002
min62.076
max62.080
mean4040.875
σ2008.944
min1216.000
max7151.000
mean83.485
σ14.406
min70.829
max107.653
op=column_set() on unindexed, dense, minor warmup
mean107.854
σ0.000
min107.854
max107.855
mean4405.500
σ2229.402
min1485.000
max7409.000
mean246.168
σ306.423
min121.288
max1002.784
op=column_set() on unindexed, dense, repeat
mean62.082
σ0.000
min62.081
max62.082
mean4586.250
σ2258.230
min1485.000
max7409.000
mean196.044
σ428.173
min40.577
max1255.569
op=vertical_lookup_list() on unindexed missing Int
mean40.946
σ0.002
min40.944
max40.950
mean7151.000
σ0.000
min7151.000
max7151.000
mean61.309
σ1.210
min60.643
max64.279
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.948
σ0.002
min40.947
max40.950
mean7409.000
σ0.000
min7409.000
max7409.000
mean21.549
σ1.050
min21.009
max24.124
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.308-66.688=4.620
σ0.002-0.002=-0.000
min71.307-66.686=4.621
max71.311-66.691=4.620
mean7409.000-7151.000=258.000
σ0.000
min7409.000-7151.000=258.000
max7409.000-7151.000=258.000
mean279.001-99.140=179.861
σ630.496-0.327=630.169
min54.829-98.478=-43.648
max1839.393-99.615=1739.778

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.850-14217.171=-9570.321
σ0.014-4.303=-4.290
min4646.845-14215.576=-9568.731
max4646.884-14227.820=-9580.936
mean4170.875-4040.875=130.000
σ1949.331-2008.944=-59.612
min1485.000-1216.000=269.000
max7409.000-7151.000=258.000
mean3169.247-7964.130=-4794.884
σ832.831-1975.150=-1142.319
min2499.113-5974.409=-3475.296
max4735.141-11133.046=-6397.905
op=vertical_lookup_list() on indexed
mean1973.851-13159.109=-11185.257
σ0.002-0.002=0.000
min1973.849-13159.106=-11185.256
max1973.854-13159.110=-11185.256
mean7409.000-7151.000=258.000
σ0.000
min7409.000-7151.000=258.000
max7409.000-7151.000=258.000
mean1031.738-5278.711=-4246.973
σ183.377-626.579=-443.202
min905.508-4980.348=-4074.840
max1385.316-6776.850=-5391.534
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.105
σ0.002
min905.102
max905.108
mean7409.000
σ0.000
min7409.000
max7409.000
mean393.998
σ3.754
min388.620
max400.019

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144-172.559=-24.414
σ0.001
min148.144-172.558=-24.414
max148.148-172.562=-24.414
mean479.000-477.000=2.000
σ0.000
min479.000-477.000=2.000
max479.000-477.000=2.000
mean362.907-376.603=-13.696
σ50.068-20.030=30.039
min284.711-350.462=-65.751
max439.075-395.002=44.073
op=pointedIndexFromKvs
rows=400000
mean308.493-357.322=-48.828
σ0.001
min308.493-357.321=-48.828
max308.497-357.325=-48.828
mean562.000
σ0.000
min562.000
max562.000
mean764.104-720.891=43.213
σ33.574-77.524=-43.950
min731.108-611.669=119.439
max826.539-799.028=27.511

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean1024.563-1024.299=0.264
σ0.760-0.759=0.001
min1024.095-1023.824=0.271
max1027.704-1027.437=0.268
mean1038.450-1058.350=-19.900
σ307.782-311.355=-3.572
min543.000-597.000=-54.000
max1524.000-1562.000=-38.000
mean1.389-1.550=-0.161
σ0.103-0.053=0.051
min1.274-1.494=-0.221
max1.661-1.696=-0.035

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913-0.913=0.000
σ0.020
min0.903-0.903=0.000
max0.949-0.949=0.000
mean401.000-402.000=-1.000
σ0.000
min401.000-402.000=-1.000
max401.000-402.000=-1.000
mean0.312-0.297=0.016
σ0.313-0.296=0.017
min0.141-0.135=0.006
max0.868-0.821=0.047
name=error_case.art
mean2.340-2.339=0.001
σ0.012
min2.334-2.333=0.001
max2.362-2.361=0.001
mean401.000-402.000=-1.000
σ0.000
min401.000-402.000=-1.000
max401.000-402.000=-1.000
mean0.434-0.472=-0.038
σ0.195-0.320=-0.125
min0.332-0.320=0.012
max0.782-1.045=-0.263
name=heavy_item.art
mean1.523-1.522=0.000
σ0.002
min1.521-1.521=0.000
max1.525-1.525=0.000
mean401.000-402.000=-1.000
σ0.000
min401.000-402.000=-1.000
max401.000-402.000=-1.000
mean0.225-0.221=0.004
σ0.022-0.022=0.000
min0.210-0.203=0.007
max0.265-0.259=0.006
name=nested_use.art
mean2.358-2.357=0.001
σ0.002
min2.357-2.356=0.001
max2.361-2.360=0.001
mean401.000-402.000=-1.000
σ0.000
min401.000-402.000=-1.000
max401.000-402.000=-1.000
mean0.347-0.334=0.013
σ0.018-0.023=-0.005
min0.330-0.314=0.016
max0.377-0.372=0.006

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.971-964.840=0.130
σ0.011
min964.965-964.835=0.130
max964.989-964.859=0.130
mean430.000-436.600=-6.600
σ0.000-6.066=-6.066
min430.000-431.000=-1.000
max430.000-443.000=-13.000
mean0.579-0.543=0.035
σ0.032-0.056=-0.023
min0.533-0.474=0.059
max0.604-0.602=0.003
name=cdr_enhanced_4k_loc.art
mean885.898-885.820=0.078
σ0.002
min885.897-885.819=0.078
max885.901-885.823=0.078
mean430.000-443.000=-13.000
σ0.000
min430.000-443.000=-13.000
max430.000-443.000=-13.000
mean0.463-0.464=-0.001
σ0.051-0.062=-0.011
min0.384-0.390=-0.005
max0.516-0.529=-0.013

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.710-7.709=0.001
σ0.002
min7.709-7.708=0.001
max7.713-7.712=0.001
mean430.000-443.000=-13.000
σ0.000
min430.000-443.000=-13.000
max430.000-443.000=-13.000
mean2.657-1.968=0.689
σ0.128-0.081=0.047
min2.468-1.788=0.680
max3.341-2.277=1.064
name=record_render_large_syntax_err.art
mean7.658-7.657=0.001
σ0.002
min7.657-7.656=0.001
max7.661-7.660=0.001
mean430.000-443.000=-13.000
σ0.000
min430.000-443.000=-13.000
max430.000-443.000=-13.000
mean2.588-2.044=0.543
σ0.080-0.276=-0.196
min2.451-1.823=0.628
max2.871-2.878=-0.008
name=record_render_success.art
mean2.290-2.289=0.000
σ0.002
min2.289-2.289=0.000
max2.293-2.293=0.000
mean430.000-443.000=-13.000
σ0.000
min430.000-443.000=-13.000
max430.000-443.000=-13.000
mean0.312-0.305=0.007
σ0.017-0.009=0.007
min0.298-0.291=0.007
max0.424-0.370=0.054
name=record_render_syntax_err.art
mean2.165-2.165=0.000
σ0.002
min2.164-2.164=0.000
max2.168-2.168=0.000
mean430.000-443.000=-13.000
σ0.000
min430.000-443.000=-13.000
max430.000-443.000=-13.000
mean0.292-0.289=0.004
σ0.007-0.010=-0.003
min0.278-0.269=0.009
max0.335-0.335=0.000

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.995
σ11.838
min0.573
max83.974
mean461.592
σ16.664
min408.000
max474.000
mean0.027
σ0.029
min0.001
max0.201

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean298.782-298.709=0.073
σ1.004-1.006=-0.002
min298.379-298.329=0.050
max301.631-301.569=0.062
mean437.100-442.400=-5.300
σ15.029-13.509=1.520
min413.000-425.000=-12.000
max459.000-467.000=-8.000
mean0.580-0.596=-0.015
σ0.046-0.034=0.012
min0.502-0.513=-0.011
max0.638-0.640=-0.003

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean173.738-236.678=-62.940
σ0.000-0.000=-0.000
min173.738-236.678=-62.940
max173.738-236.678=-62.940
mean844.000-1064.125=-220.125
σ0.000-33.588=-33.588
min844.000-981.000=-137.000
max844.000-1076.000=-232.000
mean223.925-206.372=17.554
σ28.094-3.564=24.529
min196.888-202.774=-5.887
max256.881-212.850=44.031
phase=macro expansion
mean6.133-6.130=0.004
σ0.000
min6.133-6.130=0.004
max6.133-6.130=0.004
mean843.375-1064.125=-220.750
σ1.768-33.588=-31.820
min839.000-981.000=-142.000
max844.000-1076.000=-232.000
mean9.282-9.314=-0.032
σ0.782-2.582=-1.800
min7.381-6.397=0.983
max9.753-13.097=-3.344
phase=parser
mean1042.434-1042.363=0.071
σ0.000-0.000=0.000
min1042.433-1042.362=0.071
max1042.434-1042.363=0.071
mean843.375-1064.125=-220.750
σ1.768-33.588=-31.820
min839.000-981.000=-142.000
max844.000-1076.000=-232.000
mean585.840-535.812=50.028
σ45.406-78.374=-32.968
min475.246-376.139=99.107
max614.881-626.943=-12.062
phase=processProdDef
mean2243.977-4850.764=-2606.788
σ0.000
min2243.977-4850.764=-2606.788
max2243.977-4850.764=-2606.788
mean844.000-1076.000=-232.000
σ0.000
min844.000-1076.000=-232.000
max844.000-1076.000=-232.000
mean2634.258-4769.586=-2135.328
σ549.506-408.059=141.447
min1735.854-4164.411=-2428.556
max3204.292-5459.050=-2254.759
phase=renaming
mean21.671-21.671=-0.000
σ0.000
min21.671-21.671=-0.000
max21.671-21.671=-0.000
mean843.375-1064.125=-220.750
σ1.768-33.588=-31.820
min839.000-981.000=-142.000
max844.000-1076.000=-232.000
mean38.798-33.504=5.294
σ5.061-12.413=-7.351
min26.407-20.830=5.577
max41.767-49.131=-7.364
phase=type checker
mean989.752-990.450=-0.698
σ0.066-0.071=-0.005
min989.654-990.344=-0.690
max989.841-990.563=-0.722
mean843.375-1064.125=-220.750
σ1.768-33.588=-31.820
min839.000-981.000=-142.000
max844.000-1076.000=-232.000
mean740.881-760.576=-19.695
σ86.030-185.177=-99.147
min675.296-640.163=35.133
max893.677-1133.651=-239.975

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean293.820-293.822=-0.002
σ0.038-0.042=-0.005
min293.767-293.764=0.003
max293.857-293.871=-0.013
mean444.400-439.800=4.600
σ0.548-1.095=-0.548
min444.000-439.000=5.000
max445.000-441.000=4.000
mean0.396-0.411=-0.014
σ0.003-0.008=-0.004
min0.392-0.398=-0.006
max0.402-0.416=-0.014
iters=5
new-facts=1
prior-facts=6402
mean1320.291-1320.192=0.099
σ0.142-0.088=0.054
min1320.163-1320.084=0.080
max1320.526-1320.299=0.228
mean1117.400-1093.800=23.600
σ178.357-197.366=-19.008
min899.000-837.000=62.000
max1352.000-1355.000=-3.000
mean1.946-2.118=-0.173
σ0.052-0.101=-0.050
min1.885-2.018=-0.133
max2.016-2.276=-0.260
iters=5
new-facts=6402
prior-facts=1
mean1559.279-1560.604=-1.325
σ2.999-6.226=-3.227
min1557.659-1557.634=0.025
max1564.629-1571.737=-7.108
mean627.200-628.000=-0.800
σ86.791-97.373=-10.582
min498.000-511.000=-13.000
max729.000-751.000=-22.000
mean2.244-2.297=-0.053
σ0.216-0.065=0.151
min2.091-2.204=-0.114
max2.620-2.386=0.233
iters=5
new-facts=6402
prior-facts=6402
mean1882.371-1882.162=0.209
σ0.077-0.067=0.010
min1882.287-1882.091=0.197
max1882.470-1882.249=0.221
mean1759.400-1731.000=28.400
σ211.015-188.343=22.672
min1477.000-1486.000=-9.000
max1959.000-1910.000=49.000
mean3.256-3.158=0.097
σ0.410-0.403=0.007
min2.774-2.834=-0.060
max3.692-3.853=-0.160

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.669-1.676=-0.007
σ0.133-0.151=-0.018
min1.554-1.439=0.116
max1.851-1.816=0.036

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean419.769-452.082=-32.313
σ518.096-582.722=-64.626
min160.716-160.719=-0.003
max1196.914-1326.165=-129.252
mean101.750-102.250=-0.500
σ7.500-8.500=-1.000
min98.000
max113.000-115.000=-2.000
mean447.000-443.000=4.000
σ0.000
min447.000-443.000=4.000
max447.000-443.000=4.000
mean0.502-0.544=-0.041
σ0.481-0.543=-0.062
min0.256-0.266=-0.010
max1.223-1.357=-0.134
policies=20
mean1607.500-1607.522=-0.022
σ0.356-0.401=-0.044
min1607.248-1607.251=-0.003
max1608.026-1608.116=-0.090
mean980.000
σ0.000
min980.000
max980.000
mean450.250-450.500=-0.250
σ6.500-8.660=-2.160
min447.000-443.000=4.000
max460.000-458.000=2.000
mean2.591-2.694=-0.103
σ0.018-0.018=-0.000
min2.566-2.678=-0.112
max2.607-2.717=-0.110
policies=200
mean16083.518-16083.361=0.157
σ0.049-0.165=-0.116
min16083.466-16083.218=0.247
max16083.583-16083.571=0.011
mean9800.000
σ0.000
min9800.000
max9800.000
mean499.500-496.500=3.000
σ5.972-3.416=2.557
min493.000
max507.000-501.000=6.000
mean26.141-27.137=-0.996
σ0.039-0.029=0.010
min26.101-27.115=-1.014
max26.189-27.178=-0.989