Compare commits

Home / Compare cdcc1243 .. c3926a13

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.082
σ0.000
min5.082
max5.082

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean4.988
σ0.000
min4.988
max4.988

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean4.863-5.395=-0.531
σ0.000
min4.863-5.395=-0.531
max4.863-5.395=-0.531

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.102-446.633=-0.531
σ0.000
min446.102-446.633=-0.531
max446.102-446.633=-0.531

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.238
σ0.000
min5.238
max5.238

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.395
σ0.000
min6.395
max6.395

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean57.822-54.716=3.107
σ0.361-0.364=-0.002
min57.674-54.582=3.092
max58.850-55.750=3.100
mean422.000-415.000=7.000
σ0.000
min422.000-415.000=7.000
max422.000-415.000=7.000
mean0.143-0.141=0.002
σ0.006-0.004=0.002
min0.136-0.137=-0.001
max0.156-0.147=0.008
cache=y
dps=1
iters=10
policies=100
mean46.897-43.777=3.119
σ0.004-0.009=-0.005
min46.891-43.765=3.127
max46.902-43.791=3.111
mean422.700-415.000=7.700
σ0.483-0.000=0.483
min422.000-415.000=7.000
max423.000-415.000=8.000
mean0.128-0.121=0.006
σ0.004-0.005=-0.001
min0.122-0.116=0.006
max0.134-0.132=0.002

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.605-36.604=0.001
σ0.000-0.000=-0.000
min36.605-36.604=0.001
max36.606-36.605=0.001
mean4581.000-4427.250=153.750
σ2255.544-2223.828=31.716
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean13.502-12.306=1.196
σ2.708-0.525=2.183
min11.908-11.791=0.117
max18.397-13.078=5.319
op=column_set() on indexed, repeat
mean36.605-36.605=0.000
σ0.000-0.000=-0.000
min36.605-36.604=0.001
max36.605-36.605=-0.000
mean4581.000-4606.125=-25.125
σ2255.544-2255.030=0.513
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean13.567-136.984=-123.417
σ2.356-352.107=-349.750
min11.243-11.914=-0.671
max17.747-1008.403=-990.657
op=table_records(), no indexes
mean41.940-41.940=0.000
σ0.002-0.002=0.000
min41.938-41.938=0.000
max41.943-41.942=0.000
mean7402.000-7401.000=1.000
σ0.000
min7402.000-7401.000=1.000
max7402.000-7401.000=1.000
mean11.197-11.249=-0.051
σ0.287-0.495=-0.208
min10.859-10.647=0.211
max11.806-12.296=-0.490
op=table_records(), repeat
mean41.925-41.925=0.000
σ0.000-0.000=0.000
min41.924-41.924=0.000
max41.926-41.925=0.000
mean4165.125-4198.875=-33.750
σ1946.672-1952.012=-5.341
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean11.731-11.428=0.303
σ1.899-0.830=1.069
min10.777-10.903=-0.126
max16.396-13.419=2.978
op=vertical_lookup_list() on indexed, after table_records()
mean27.634-27.633=0.001
σ0.137-0.136=0.000
min27.585-27.584=0.001
max27.972-27.971=0.001
mean4165.125-4198.875=-33.750
σ1946.672-1952.012=-5.341
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean9.180-8.681=0.499
σ1.406-0.611=0.795
min8.062-7.844=0.218
max12.562-9.911=2.651
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.385-26.384=0.001
σ0.000-0.000=0.000
min26.385-26.384=0.001
max26.385-26.384=0.001
mean4165.125-4198.875=-33.750
σ1946.672-1952.012=-5.341
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean6.998-6.491=0.507
σ0.974-0.621=0.353
min6.044-5.929=0.115
max9.212-7.888=1.324
op=vertical_lookup_row_list() on 2 indexed
mean25.663-25.662=0.001
σ0.004-0.004=0.000
min25.662-25.661=0.001
max25.674-25.673=0.001
mean4581.000-4606.125=-25.125
σ2255.544-2255.030=0.513
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean6.357-6.393=-0.036
σ0.541-1.112=-0.571
min5.941-5.796=0.146
max7.348-9.111=-1.762
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.962-25.961=0.002
σ0.001-0.001=-0.000
min25.962-25.961=0.002
max25.964-25.962=0.002
mean4581.000-4606.125=-25.125
σ2255.544-2255.030=0.513
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean6.166-6.131=0.035
σ0.621-1.110=-0.489
min5.358-5.129=0.229
max7.120-8.788=-1.668

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.599-1021.608=-0.009
σ0.002-0.002=-0.000
min1021.597-1021.606=-0.009
max1021.602-1021.611=-0.009
mean7402.000-7401.000=1.000
σ0.000
min7402.000-7401.000=1.000
max7402.000-7401.000=1.000
mean364.699-345.903=18.796
σ39.919-2.877=37.041
min345.879-342.340=3.539
max463.109-350.467=112.643
op=load_product(), with indexes
mean3495.846-3495.855=-0.009
σ0.455-0.454=0.001
min3495.685-3495.694=-0.009
max3496.972-3496.978=-0.006
mean3745.750-3771.125=-25.375
σ2298.461-2308.407=-9.946
min720.000-721.000=-1.000
max7402.000-7401.000=1.000
mean1636.580-1747.296=-110.715
σ498.481-651.980=-153.500
min1397.408-1378.125=19.283
max2853.908-2823.346=30.562

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense, minor warmup
mean107.842-107.841=0.001
σ0.000-0.000=0.000
min107.841-107.840=0.001
max107.842-107.841=0.001
mean4399.750-4198.875=200.875
σ2226.927-1952.012=274.914
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean268.078-128.569=139.509
σ378.192-5.885=372.307
min123.789-123.699=0.090
max1202.914-141.403=1061.510
op=column_set() on unindexed, dense, repeat
mean62.069-62.068=0.001
σ0.001
min62.065-62.065=0.001
max62.070-62.069=0.001
mean4581.000-4427.250=153.750
σ2255.544-2223.828=31.716
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean150.891-93.672=57.219
σ295.416-144.737=150.679
min41.195-40.837=0.358
max881.698-451.860=429.838
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.927-40.926=0.001
σ0.002-0.002=-0.000
min40.926-40.925=0.001
max40.930-40.929=0.000
mean7402.000-7401.000=1.000
σ0.000
min7402.000-7401.000=1.000
max7402.000-7401.000=1.000
mean22.101-106.713=-84.612
σ1.193-236.411=-235.218
min21.388-21.472=-0.084
max24.997-691.770=-666.773
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.294-71.293=0.001
σ0.002-0.002=-0.000
min71.293-71.292=0.001
max71.297-71.296=0.001
mean7402.000-7401.000=1.000
σ0.000
min7402.000-7401.000=1.000
max7402.000-7401.000=1.000
mean287.644-59.019=228.625
σ653.017-9.514=643.503
min55.234-54.774=0.460
max1903.772-82.471=1821.302

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.843-4646.842=0.001
σ0.014-0.013=0.000
min4646.837-4646.837=0.000
max4646.876-4646.875=0.001
mean4165.125-4198.875=-33.750
σ1946.672-1952.012=-5.341
min1486.000-1484.000=2.000
max7402.000-7401.000=1.000
mean3150.523-3193.585=-43.062
σ849.030-618.534=230.497
min2485.317-2483.354=1.963
max4352.816-4000.962=351.853
op=vertical_lookup_list() on indexed
mean1973.830-1973.829=0.001
σ0.002-0.002=0.000
min1973.829-1973.827=0.001
max1973.833-1973.831=0.002
mean7402.000-7401.000=1.000
σ0.000
min7402.000-7401.000=1.000
max7402.000-7401.000=1.000
mean1065.484-1023.854=41.630
σ256.178-204.588=51.590
min914.905-916.948=-2.043
max1664.499-1485.185=179.314
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.084-905.083=0.001
σ0.002
min905.082-905.080=0.001
max905.087-905.086=0.001
mean7402.000-7401.000=1.000
σ0.000
min7402.000-7401.000=1.000
max7402.000-7401.000=1.000
mean392.123-408.246=-16.123
σ3.720-42.436=-38.716
min387.405-388.401=-0.996
max399.953-512.970=-113.016

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144
σ0.001
min148.144
max148.148
mean476.500-478.000=-1.500
σ0.535-0.000=0.535
min476.000-478.000=-2.000
max477.000-478.000=-1.000
mean386.208-404.486=-18.278
σ30.082-21.718=8.364
min362.893-381.511=-18.618
max440.576-441.432=-0.856
op=pointedIndexFromKvs
rows=400000
mean308.493
σ0.001
min308.493
max308.497
mean562.750-559.250=3.500
σ0.463-1.753=-1.290
min562.000-558.000=4.000
max563.000-562.000=1.000
mean736.549-743.796=-7.247
σ58.112-85.802=-27.690
min623.846-615.602=8.244
max799.298-886.201=-86.902

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean976.918-964.563=12.356
σ2.437-1.880=0.557
min976.074-963.842=12.232
max987.245-972.517=14.728
mean1045.750-1046.900=-1.150
σ309.118-321.178=-12.060
min540.000-545.000=-5.000
max1543.000-1569.000=-26.000
mean1.293-1.246=0.047
σ0.167-0.078=0.089
min1.201-1.179=0.022
max1.970-1.505=0.464

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913
σ0.020
min0.903
max0.949
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.284-0.389=-0.105
σ0.291-0.460=-0.169
min0.129-0.152=-0.023
max0.800-1.209=-0.409
name=error_case.art
mean2.346-2.346=-0.000
σ0.012
min2.340-2.340=-0.000
max2.368-2.368=-0.000
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.478-0.416=0.062
σ0.334-0.194=0.139
min0.323-0.323=-0.000
max1.075-0.763=0.312
name=heavy_item.art
mean1.523
σ0.002
min1.522
max1.526
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.218-0.219=-0.001
σ0.017-0.016=0.001
min0.206-0.209=-0.003
max0.247-0.248=-0.001
name=nested_use.art
mean2.364-2.364=-0.000
σ0.002
min2.363-2.363=-0.000
max2.367-2.367=-0.000
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.341-0.343=-0.002
σ0.022-0.018=0.004
min0.319-0.325=-0.006
max0.375-0.371=0.004

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.978-964.979=-0.001
σ0.009
min964.973-964.974=-0.001
max964.995-964.996=-0.001
mean431.000-430.000=1.000
σ0.000
min431.000-430.000=1.000
max431.000-430.000=1.000
mean0.532-0.525=0.008
σ0.061-0.061=0.000
min0.464-0.431=0.033
max0.607-0.603=0.004
name=cdr_enhanced_4k_loc.art
mean886.000-886.001=-0.001
σ0.002
min885.999-886.000=-0.001
max886.003-886.004=-0.001
mean431.000-430.000=1.000
σ0.000
min431.000-430.000=1.000
max431.000-430.000=1.000
mean0.425-0.464=-0.039
σ0.043-0.053=-0.010
min0.378-0.387=-0.009
max0.466-0.528=-0.062

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.719
σ0.002
min7.718
max7.722
mean431.000-430.000=1.000
σ0.000
min431.000-430.000=1.000
max431.000-430.000=1.000
mean2.037-2.639=-0.602
σ0.232-0.213=0.019
min1.828-1.926=-0.097
max2.734-2.982=-0.249
name=record_render_large_syntax_err.art
mean7.667
σ0.002
min7.666
max7.670
mean431.000-430.000=1.000
σ0.000
min431.000-430.000=1.000
max431.000-430.000=1.000
mean2.571-2.598=-0.027
σ0.077-0.092=-0.016
min2.442-2.419=0.023
max2.837-2.999=-0.162
name=record_render_success.art
mean2.290
σ0.002
min2.290
max2.294
mean431.000-430.000=1.000
σ0.000
min431.000-430.000=1.000
max431.000-430.000=1.000
mean0.302-0.304=-0.002
σ0.009-0.009=-0.000
min0.288-0.290=-0.001
max0.357-0.347=0.011
name=record_render_syntax_err.art
mean2.166
σ0.002
min2.165
max2.169
mean431.000-430.000=1.000
σ0.000
min431.000-430.000=1.000
max431.000-430.000=1.000
mean0.293-0.297=-0.004
σ0.016-0.014=0.002
min0.272-0.277=-0.005
max0.415-0.369=0.046

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.681-11.543=0.138
σ11.380-11.669=-0.289
min0.714-0.574=0.140
max90.366-106.199=-15.833
mean468.351-458.430=9.921
σ19.410-17.277=2.133
min409.000-408.000=1.000
max481.000-475.000=6.000
mean0.027-0.027=0.000
σ0.028-0.028=-0.001
min0.001-0.001=0.000
max0.216-0.253=-0.037

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean293.781-279.198=14.582
σ1.066-1.026=0.040
min293.281-278.812=14.469
max296.788-282.118=14.670
mean441.500-440.800=0.700
σ20.266-12.865=7.401
min414.000-426.000=-12.000
max471.000-463.000=8.000
mean0.594-0.580=0.014
σ0.019-0.020=-0.001
min0.557-0.542=0.015
max0.625-0.621=0.004

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean106.546-169.257=-62.711
σ0.000-0.000=0.000
min106.546-169.257=-62.711
max106.546-169.257=-62.711
mean848.000-851.000=-3.000
σ0.000
min848.000-851.000=-3.000
max848.000-851.000=-3.000
mean143.160-201.723=-58.563
σ23.350-16.639=6.711
min92.854-188.992=-96.138
max163.217-241.070=-77.853
phase=macro expansion
mean5.894-5.896=-0.002
σ0.000
min5.894-5.896=-0.002
max5.894-5.896=-0.002
mean848.000-851.000=-3.000
σ0.000
min848.000-851.000=-3.000
max848.000-851.000=-3.000
mean6.634-9.082=-2.448
σ2.106-0.124=1.982
min5.837-8.960=-3.123
max11.845-9.357=2.488
phase=parser
mean1042.134-1042.131=0.003
σ0.001-0.000=0.001
min1042.131-1042.131=0.000
max1042.134-1042.131=0.003
mean848.000-851.000=-3.000
σ0.000
min848.000-851.000=-3.000
max848.000-851.000=-3.000
mean421.035-598.729=-177.694
σ124.952-31.417=93.536
min371.970-521.649=-149.679
max730.166-617.962=112.204
phase=processProdDef
mean2241.193-2243.718=-2.525
σ0.000-0.000=0.000
min2241.193-2243.718=-2.525
max2241.194-2243.718=-2.524
mean848.000-851.000=-3.000
σ0.000
min848.000-851.000=-3.000
max848.000-851.000=-3.000
mean2940.314-2795.574=144.740
σ296.189-321.968=-25.779
min2522.933-2272.752=250.181
max3324.794-3205.126=119.668
phase=renaming
mean21.430-21.440=-0.010
σ0.001-0.000=0.001
min21.430-21.440=-0.010
max21.433-21.440=-0.006
mean848.000-851.000=-3.000
σ0.000
min848.000-851.000=-3.000
max848.000-851.000=-3.000
mean40.566-47.876=-7.311
σ3.680-0.934=2.745
min36.304-46.730=-10.426
max47.732-49.589=-1.857
phase=type checker
mean958.466-958.596=-0.131
σ0.047-0.047=0.000
min958.439-958.573=-0.134
max958.568-958.698=-0.130
mean848.000-851.000=-3.000
σ0.000
min848.000-851.000=-3.000
max848.000-851.000=-3.000
mean686.828-675.036=11.792
σ40.541-9.472=31.069
min657.674-662.684=-5.009
max781.799-686.693=95.107

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean327.853-282.856=44.998
σ0.036-0.036=0.000
min327.803-282.804=44.999
max327.890-282.891=44.998
mean442.000-440.000=2.000
σ4.416-2.739=1.677
min435.000-438.000=-3.000
max447.000-443.000=4.000
mean0.435-0.381=0.054
σ0.006-0.010=-0.005
min0.427-0.374=0.053
max0.439-0.399=0.041
iters=5
new-facts=1
prior-facts=6402
mean1313.056-1268.061=44.995
σ0.142-0.105=0.037
min1312.968-1267.964=45.003
max1313.303-1268.230=45.073
mean1107.600-1092.600=15.000
σ182.815-169.042=13.773
min881.000-893.000=-12.000
max1327.000-1326.000=1.000
mean1.966-2.035=-0.069
σ0.151-0.086=0.065
min1.879-1.897=-0.018
max2.234-2.118=0.116
iters=5
new-facts=6402
prior-facts=1
mean1552.072-1507.088=44.984
σ3.038-2.954=0.084
min1550.563-1505.569=44.995
max1557.500-1512.363=45.137
mean631.000-624.400=6.600
σ90.907-88.124=2.783
min504.000-493.000=11.000
max740.000-727.000=13.000
mean2.313-2.332=-0.019
σ0.171-0.231=-0.060
min2.131-2.119=0.012
max2.558-2.607=-0.048
iters=5
new-facts=6402
prior-facts=6402
mean1875.063-1830.063=45.001
σ0.055-0.048=0.007
min1874.997-1829.994=45.003
max1875.134-1830.111=45.023
mean1750.000-1724.600=25.400
σ154.834-186.330=-31.496
min1511.000-1464.000=47.000
max1940.000-1894.000=46.000
mean2.926-3.033=-0.107
σ0.084-0.335=-0.251
min2.848-2.828=0.019
max3.064-3.630=-0.566

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.601-1.769=-0.168
σ0.056-0.120=-0.064
min1.512-1.648=-0.136
max1.652-1.922=-0.269

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean419.405-406.200=13.204
σ508.891-509.797=-0.907
min164.958-151.298=13.660
max1182.741-1170.897=11.844
mean101.250-101.750=-0.500
σ6.500-7.500=-1.000
min98.000
max111.000-113.000=-2.000
mean433.000-436.000=-3.000
σ0.000
min433.000-436.000=-3.000
max433.000-436.000=-3.000
mean0.525-0.503=0.021
σ0.502-0.498=0.004
min0.271-0.249=0.022
max1.278-1.250=0.028
policies=20
mean1649.815-1513.152=136.663
σ0.380-0.275=0.105
min1649.598-1512.969=136.629
max1650.384-1513.559=136.825
mean980.000
σ0.000
min980.000
max980.000
mean451.500-442.000=9.500
σ10.755-6.928=3.827
min440.000-436.000=4.000
max466.000-448.000=18.000
mean2.660-2.511=0.149
σ0.019-0.010=0.009
min2.635-2.503=0.131
max2.676-2.525=0.150
policies=200
mean16506.015-15139.943=1366.072
σ0.082-0.105=-0.023
min16505.939-15139.820=1366.119
max16506.088-15140.035=1366.053
mean9800.000
σ0.000
min9800.000
max9800.000
mean503.000-497.500=5.500
σ4.397-5.508=-1.111
min498.000-492.000=6.000
max508.000-504.000=4.000
mean26.785-25.239=1.546
σ0.172-0.088=0.084
min26.602-25.124=1.478
max26.935-25.338=1.597