Compare commits

Home / Compare a8b47ac2 .. 50911553

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.082-5.301=-0.219
σ0.000
min5.082-5.301=-0.219
max5.082-5.301=-0.219

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean4.988-5.176=-0.188
σ0.000
min4.988-5.176=-0.188
max4.988-5.176=-0.188

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean5.395
σ0.000
min5.395
max5.395

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.633-558.633=-112.000
σ0.000
min446.633-558.633=-112.000
max446.633-558.633=-112.000

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.238-5.488=-0.250
σ0.000
min5.238-5.488=-0.250
max5.238-5.488=-0.250

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.395-6.645=-0.250
σ0.000
min6.395-6.645=-0.250
max6.395-6.645=-0.250

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.715-54.882=-0.167
σ0.362-0.358=0.004
min54.583-54.748=-0.166
max55.745-55.901=-0.156
mean421.000-418.700=2.300
σ0.000-0.483=-0.483
min421.000-418.000=3.000
max421.000-419.000=2.000
mean0.139-0.142=-0.003
σ0.004-0.004=0.000
min0.133-0.139=-0.005
max0.145-0.149=-0.005
cache=y
dps=1
iters=10
policies=100
mean43.778-43.850=-0.072
σ0.009-0.006=0.003
min43.762-43.839=-0.077
max43.789-43.860=-0.071
mean421.000-419.000=2.000
σ0.000
min421.000-419.000=2.000
max421.000-419.000=2.000
mean0.122-0.125=-0.002
σ0.004-0.004=0.000
min0.115-0.122=-0.006
max0.130-0.132=-0.002

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.604-36.618=-0.014
σ0.000-0.002=-0.002
min36.604-36.616=-0.012
max36.605-36.620=-0.015
mean4590.625-4052.250=538.375
σ2248.286-2017.618=230.668
min1486.000-1218.000=268.000
max7384.000-7150.000=234.000
mean174.709-13.542=161.167
σ458.088-1.594=456.493
min11.835-12.593=-0.758
max1308.409-17.417=1290.992
op=column_set() on indexed, repeat
mean36.604-36.618=-0.014
σ0.000-0.002=-0.002
min36.604-36.616=-0.012
max36.604-36.620=-0.016
mean4590.625-4052.250=538.375
σ2248.286-2017.618=230.668
min1486.000-1218.000=268.000
max7384.000-7150.000=234.000
mean13.382-13.541=-0.160
σ3.006-1.788=1.218
min11.374-11.838=-0.464
max18.698-17.529=1.168
op=table_records(), no indexes
mean41.939-41.930=0.010
σ0.002-0.002=0.000
min41.938-41.928=0.009
max41.942-41.933=0.009
mean7384.000-7150.000=234.000
σ0.000
min7384.000-7150.000=234.000
max7384.000-7150.000=234.000
mean11.693-15.237=-3.544
σ1.970-0.202=1.768
min10.766-14.903=-4.136
max16.558-15.538=1.020
op=table_records(), repeat
mean41.925-41.929=-0.004
σ0.001-0.002=-0.001
min41.924-41.927=-0.003
max41.928-41.932=-0.004
mean4181.125-3893.250=287.875
σ1944.663-1920.985=23.678
min1486.000-1218.000=268.000
max7384.000-7150.000=234.000
mean11.576-14.655=-3.079
σ1.849-1.437=0.412
min10.574-11.358=-0.785
max16.113-15.981=0.132
op=vertical_lookup_list() on indexed, after table_records()
mean27.633-27.654=-0.021
σ0.137-0.136=0.000
min27.584-27.604=-0.020
max27.971-27.991=-0.020
mean4181.125-3893.250=287.875
σ1944.663-1920.985=23.678
min1486.000-1218.000=268.000
max7384.000-7150.000=234.000
mean8.888-9.704=-0.816
σ1.316-1.221=0.095
min7.971-8.447=-0.476
max12.091-12.437=-0.345
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.384-26.401=-0.017
σ0.000-0.002=-0.002
min26.384-26.400=-0.016
max26.384-26.404=-0.020
mean4181.125-3893.250=287.875
σ1944.663-1920.985=23.678
min1486.000-1218.000=268.000
max7384.000-7150.000=234.000
mean6.958-6.971=-0.013
σ0.985-1.407=-0.422
min6.272-5.675=0.597
max9.339-10.171=-0.832
op=vertical_lookup_row_list() on 2 indexed
mean25.662-25.668=-0.006
σ0.004-0.004=0.000
min25.661-25.665=-0.005
max25.673-25.677=-0.005
mean4590.625-4052.250=538.375
σ2248.286-2017.618=230.668
min1486.000-1218.000=268.000
max7384.000-7150.000=234.000
mean6.414-6.142=0.272
σ0.894-0.560=0.333
min5.821-5.681=0.140
max8.280-7.446=0.834
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.961-25.970=-0.009
σ0.000-0.002=-0.001
min25.961-25.968=-0.008
max25.962-25.972=-0.010
mean4590.625-4052.250=538.375
σ2248.286-2017.618=230.668
min1486.000-1218.000=268.000
max7384.000-7150.000=234.000
mean6.339-5.837=0.502
σ0.984-0.574=0.410
min5.704-5.224=0.480
max8.500-7.173=1.326

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.608-1021.613=-0.005
σ0.002-0.002=0.000
min1021.606-1021.612=-0.006
max1021.611-1021.616=-0.005
mean7384.000-7150.000=234.000
σ0.000
min7384.000-7150.000=234.000
max7384.000-7150.000=234.000
mean344.236-428.230=-83.994
σ2.020-231.891=-229.871
min340.985-345.130=-4.145
max346.823-1002.127=-655.304
op=load_product(), with indexes
mean3495.855-7917.793=-4421.938
σ0.454-0.469=-0.015
min3495.694-7917.625=-4421.931
max3496.978-7918.954=-4421.976
mean3757.625-3283.000=474.625
σ2299.416-1838.442=460.974
min720.000-809.000=-89.000
max7384.000-5656.000=1728.000
mean1699.115-3741.629=-2042.515
σ537.847-1132.796=-594.949
min1377.289-3084.650=-1707.361
max2743.767-6197.515=-3453.748

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense
mean62.078
σ0.002
min62.076
max62.080
mean4052.250
σ2017.618
min1218.000
max7150.000
mean197.434
σ311.911
min70.156
max968.079
op=column_set() on unindexed, dense, minor warmup
mean107.841
σ0.000
min107.841
max107.841
mean4410.125
σ2217.836
min1486.000
max7384.000
mean254.994
σ337.978
min123.931
max1090.264
op=column_set() on unindexed, dense, repeat
mean62.068
σ0.001
min62.065
max62.068
mean4410.125
σ2217.836
min1486.000
max7384.000
mean45.783
σ8.803
min41.157
max67.310
op=vertical_lookup_list() on unindexed missing Int
mean40.946
σ0.002
min40.945
max40.950
mean7150.000
σ0.000
min7150.000
max7150.000
mean60.751
σ0.459
min60.164
max61.376
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.926
σ0.002
min40.925
max40.928
mean7384.000
σ0.000
min7384.000
max7384.000
mean183.691
σ457.417
min21.437
max1315.740
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.294-66.688=4.606
σ0.002-0.002=-0.000
min71.292-66.686=4.606
max71.296-66.691=4.606
mean7384.000-7150.000=234.000
σ0.000
min7384.000-7150.000=234.000
max7384.000-7150.000=234.000
mean63.949-99.289=-35.340
σ18.619-1.053=17.566
min56.025-97.811=-41.786
max109.930-100.695=9.235

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.842-14217.171=-9570.329
σ0.014-4.304=-4.290
min4646.837-14215.575=-9568.738
max4646.876-14227.821=-9580.946
mean4181.125-3893.250=287.875
σ1944.663-1920.985=23.678
min1486.000-1218.000=268.000
max7384.000-7150.000=234.000
mean3312.854-7506.448=-4193.594
σ819.236-2140.243=-1321.007
min2486.286-5882.710=-3396.424
max4543.946-10962.122=-6418.176
op=vertical_lookup_list() on indexed
mean1973.829-13159.108=-11185.279
σ0.002-0.002=0.000
min1973.827-13159.106=-11185.279
max1973.832-13159.110=-11185.278
mean7384.000-7150.000=234.000
σ0.000
min7384.000-7150.000=234.000
max7384.000-7150.000=234.000
mean1028.777-5157.313=-4128.536
σ199.657-240.408=-40.751
min910.651-4952.316=-4041.665
max1409.223-5479.291=-4070.068
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.083
σ0.002
min905.080
max905.086
mean7384.000
σ0.000
min7384.000
max7384.000
mean384.261
σ5.985
min371.442
max390.595

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144-172.559=-24.414
σ0.001
min148.144-172.558=-24.414
max148.148-172.562=-24.414
mean475.250-478.000=-2.750
σ1.165-0.000=1.165
min473.000-478.000=-5.000
max476.000-478.000=-2.000
mean375.122-368.108=7.014
σ13.470-33.398=-19.928
min358.007-303.423=54.584
max391.306-394.768=-3.462
op=pointedIndexFromKvs
rows=400000
mean308.493-357.322=-48.828
σ0.001
min308.493-357.321=-48.828
max308.497-357.325=-48.828
mean558.000-562.000=-4.000
σ0.000
min558.000-562.000=-4.000
max558.000-562.000=-4.000
mean734.088-710.707=23.381
σ63.128-64.610=-1.482
min619.838-625.491=-5.653
max814.407-784.832=29.574

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean964.562-1024.299=-59.737
σ1.881-0.758=1.123
min963.842-1023.827=-59.985
max972.518-1027.433=-54.915
mean1035.850-1037.300=-1.450
σ309.128-303.207=5.921
min545.000-553.000=-8.000
max1533.000-1461.000=72.000
mean1.236-1.574=-0.338
σ0.056-0.085=-0.029
min1.151-1.490=-0.339
max1.339-1.774=-0.435

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913-0.913=0.001
σ0.020-0.020=-0.000
min0.903-0.903=0.001
max0.949-0.949=0.000
mean402.000
σ0.000
min402.000
max402.000
mean0.463-0.423=0.040
σ0.459-0.560=-0.101
min0.141-0.147=-0.006
max1.255-1.422=-0.167
name=error_case.art
mean2.346-2.339=0.007
σ0.012
min2.340-2.333=0.007
max2.368-2.361=0.007
mean402.000
σ0.000
min402.000
max402.000
mean0.482-0.471=0.011
σ0.320-0.319=0.001
min0.324-0.314=0.010
max1.055-1.042=0.013
name=heavy_item.art
mean1.523-1.522=0.001
σ0.002-0.002=-0.000
min1.522-1.521=0.001
max1.526-1.525=0.001
mean402.000
σ0.000
min402.000
max402.000
mean0.234-0.229=0.006
σ0.041-0.023=0.018
min0.196-0.208=-0.012
max0.294-0.254=0.041
name=nested_use.art
mean2.364-2.357=0.007
σ0.002
min2.363-2.356=0.007
max2.367-2.360=0.007
mean402.000
σ0.000
min402.000
max402.000
mean0.333-0.333=-0.000
σ0.020-0.021=-0.001
min0.312-0.318=-0.005
max0.365-0.369=-0.005

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.979-964.840=0.139
σ0.009-0.011=-0.001
min964.974-964.835=0.139
max964.996-964.859=0.137
mean431.000-436.200=-5.200
σ0.000-1.789=-1.789
min431.000-433.000=-2.000
max431.000-437.000=-6.000
mean0.582-0.523=0.058
σ0.035-0.033=0.002
min0.520-0.500=0.019
max0.607-0.579=0.029
name=cdr_enhanced_4k_loc.art
mean886.001-885.820=0.181
σ0.002-0.002=-0.000
min886.000-885.819=0.181
max886.004-885.823=0.181
mean431.000-438.000=-7.000
σ0.000
min431.000-438.000=-7.000
max431.000-438.000=-7.000
mean0.498-0.503=-0.005
σ0.031-0.023=0.008
min0.448-0.465=-0.017
max0.521-0.520=0.001

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.719-7.709=0.010
σ0.002
min7.718-7.708=0.010
max7.722-7.712=0.010
mean431.000-438.000=-7.000
σ0.000
min431.000-438.000=-7.000
max431.000-438.000=-7.000
mean2.616-2.602=0.014
σ0.090-0.088=0.002
min2.480-2.402=0.078
max2.958-2.855=0.103
name=record_render_large_syntax_err.art
mean7.667-7.657=0.010
σ0.002
min7.666-7.656=0.010
max7.670-7.660=0.010
mean431.000-438.000=-7.000
σ0.000
min431.000-438.000=-7.000
max431.000-438.000=-7.000
mean2.587-2.592=-0.005
σ0.107-0.099=0.009
min2.445-2.460=-0.015
max3.346-3.184=0.161
name=record_render_success.art
mean2.290-2.289=0.001
σ0.002
min2.290-2.289=0.001
max2.294-2.293=0.001
mean431.000-438.000=-7.000
σ0.000
min431.000-438.000=-7.000
max431.000-438.000=-7.000
mean0.302-0.308=-0.006
σ0.010-0.014=-0.004
min0.289-0.291=-0.002
max0.359-0.388=-0.029
name=record_render_syntax_err.art
mean2.166-2.165=0.001
σ0.002
min2.165-2.164=0.001
max2.169-2.168=0.001
mean431.000-438.000=-7.000
σ0.000
min431.000-438.000=-7.000
max431.000-438.000=-7.000
mean0.289-0.291=-0.002
σ0.007-0.009=-0.001
min0.277-0.276=0.001
max0.327-0.329=-0.002

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.785
σ12.406
min0.574
max125.115
mean459.393
σ20.202
min406.000
max475.000
mean0.027
σ0.030
min0.001
max0.319

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean279.006-298.684=-19.678
σ1.044-1.014=0.030
min278.608-298.319=-19.711
max281.976-301.569=-19.593
mean444.400-441.600=2.800
σ18.928-14.152=4.776
min412.000-417.000=-5.000
max476.000-465.000=11.000
mean0.586-0.585=0.001
σ0.013-0.031=-0.018
min0.557-0.534=0.023
max0.604-0.642=-0.038

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean169.257-236.678=-67.420
σ0.000-0.000=0.000
min169.257-236.678=-67.420
max169.257-236.678=-67.420
mean848.500-923.375=-74.875
σ0.535-10.253=-9.719
min848.000-898.000=-50.000
max849.000-927.000=-78.000
mean210.538-210.268=0.270
σ24.943-2.167=22.775
min189.003-208.014=-19.011
max249.465-213.383=36.082
phase=macro expansion
mean5.899-6.130=-0.230
σ0.000
min5.899-6.130=-0.230
max5.899-6.130=-0.230
mean848.375-920.500=-72.125
σ0.518-18.385=-17.867
min848.000-875.000=-27.000
max849.000-927.000=-78.000
mean7.133-8.663=-1.531
σ2.317-1.550=0.766
min5.911-6.299=-0.387
max12.226-10.000=2.226
phase=parser
mean1042.130-1042.364=-0.233
σ0.000-0.000=-0.000
min1042.130-1042.364=-0.234
max1042.131-1042.364=-0.234
mean848.375-920.500=-72.125
σ0.518-18.385=-17.867
min848.000-875.000=-27.000
max849.000-927.000=-78.000
mean453.153-504.559=-51.406
σ130.904-78.983=51.921
min370.583-379.083=-8.500
max718.994-638.844=80.150
phase=processProdDef
mean2243.720-4850.763=-2607.043
σ0.000-0.000=-0.000
min2243.720-4850.763=-2607.043
max2243.720-4850.763=-2607.043
mean848.500-927.000=-78.500
σ0.535-0.000=0.535
min848.000-927.000=-79.000
max849.000-927.000=-78.000
mean2724.207-5325.675=-2601.468
σ372.799-321.263=51.536
min2234.681-4986.209=-2751.529
max3201.065-6037.104=-2836.039
phase=renaming
mean21.436-21.671=-0.235
σ0.000
min21.436-21.671=-0.235
max21.436-21.671=-0.235
mean848.375-920.500=-72.125
σ0.518-18.385=-17.867
min848.000-875.000=-27.000
max849.000-927.000=-78.000
mean39.367-32.891=6.476
σ5.427-9.072=-3.645
min30.222-20.674=9.548
max47.381-41.204=6.177
phase=type checker
mean958.596-990.450=-31.854
σ0.047-0.071=-0.024
min958.573-990.344=-31.772
max958.698-990.563=-31.866
mean848.375-923.250=-74.875
σ0.518-10.607=-10.089
min848.000-897.000=-49.000
max849.000-927.000=-78.000
mean699.630-707.808=-8.178
σ69.878-71.294=-1.415
min636.913-663.189=-26.276
max845.992-882.041=-36.049

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean282.869-293.843=-10.973
σ0.029-0.026=0.003
min282.840-293.814=-10.974
max282.917-293.882=-10.965
mean440.600-443.000=-2.400
σ3.209-0.000=3.209
min437.000-443.000=-6.000
max444.000-443.000=1.000
mean0.379-0.395=-0.016
σ0.006-0.006=0.000
min0.374-0.391=-0.018
max0.387-0.405=-0.019
iters=5
new-facts=1
prior-facts=6402
mean1268.114-1320.252=-52.138
σ0.105-0.057=0.048
min1268.004-1320.180=-52.176
max1268.282-1320.316=-52.035
mean1110.400-1073.200=37.200
σ184.260-168.721=15.539
min889.000-844.000=45.000
max1360.000-1275.000=85.000
mean1.884-2.261=-0.377
σ0.118-0.110=0.007
min1.801-2.160=-0.358
max2.092-2.441=-0.350
iters=5
new-facts=6402
prior-facts=1
mean1507.100-1560.635=-53.535
σ2.967-6.240=-3.273
min1505.604-1557.670=-52.067
max1512.399-1571.793=-59.394
mean628.800-618.600=10.200
σ91.026-83.146=7.880
min504.000-493.000=11.000
max728.000-717.000=11.000
mean2.373-2.336=0.038
σ0.280-0.210=0.070
min2.128-2.212=-0.084
max2.784-2.708=0.075
iters=5
new-facts=6402
prior-facts=6402
mean1830.109-1882.152=-52.043
σ0.069-0.039=0.030
min1830.034-1882.097=-52.062
max1830.185-1882.184=-51.999
mean1757.600-1698.800=58.800
σ212.713-247.193=-34.480
min1476.000-1393.000=83.000
max1960.000-2025.000=-65.000
mean3.141-2.953=0.188
σ0.282-0.111=0.172
min2.818-2.840=-0.021
max3.512-3.109=0.403

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.676-1.710=-0.035
σ0.122-0.158=-0.036
min1.549-1.441=0.109
max1.844-1.819=0.025

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean406.197-452.081=-45.884
σ509.796-582.723=-72.927
min151.295-160.716=-9.421
max1170.891-1326.166=-155.274
mean101.750-102.250=-0.500
σ7.500-8.500=-1.000
min98.000
max113.000-115.000=-2.000
mean446.000-442.000=4.000
σ0.000
min446.000-442.000=4.000
max446.000-442.000=4.000
mean0.517-0.532=-0.015
σ0.525-0.518=0.007
min0.249-0.267=-0.018
max1.304-1.308=-0.004
policies=20
mean1513.141-1607.527=-94.386
σ0.271-0.412=-0.140
min1512.967-1607.248=-94.281
max1513.543-1608.137=-94.594
mean980.000
σ0.000
min980.000
max980.000
mean448.250-446.250=2.000
σ1.500-8.500=-7.000
min446.000-442.000=4.000
max449.000-459.000=-10.000
mean2.495-2.690=-0.195
σ0.006-0.019=-0.013
min2.490-2.670=-0.180
max2.503-2.714=-0.211
policies=200
mean15140.032-16083.475=-943.444
σ0.108-0.086=0.023
min15139.914-16083.415=-943.501
max15140.176-16083.602=-943.427
mean9800.000
σ0.000
min9800.000
max9800.000
mean498.750-494.750=4.000
σ6.702-3.862=2.840
min492.000-491.000=1.000
max508.000-499.000=9.000
mean25.216-27.145=-1.929
σ0.133-0.036=0.097
min25.083-27.096=-2.013
max25.390-27.184=-1.794