Compare commits

Home / Compare d8eaab48 .. 0e76f4e6

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.677-54.580=0.097
σ0.357-0.341=0.016
min54.548-54.451=0.096
max55.692-55.550=0.142
mean421.000-418.000=3.000
σ0.000
min421.000-418.000=3.000
max421.000-418.000=3.000
mean0.142-0.141=0.001
σ0.005-0.006=-0.001
min0.137-0.136=0.001
max0.150-0.152=-0.002
cache=y
dps=1
iters=10
policies=100
mean43.704-43.704=0.001
σ0.005-0.006=-0.000
min43.694-43.694=0.000
max43.712-43.710=0.002
mean421.000-418.000=3.000
σ0.000
min421.000-418.000=3.000
max421.000-418.000=3.000
mean0.122-0.120=0.002
σ0.004-0.006=-0.002
min0.118-0.115=0.003
max0.129-0.131=-0.001

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.618-36.618=0.000
σ0.002-0.002=0.000
min36.616-36.615=0.000
max36.620-36.620=0.000
mean4156.125-4147.500=8.625
σ1903.350-1890.836=12.514
min1555.000-1554.000=1.000
max7320.000-7273.000=47.000
mean12.681-12.876=-0.196
σ0.180-0.458=-0.278
min12.423-12.332=0.091
max12.950-13.628=-0.679
op=column_set() on indexed, repeat
mean36.617-36.617=0.001
σ0.002-0.002=0.000
min36.616-36.616=0.000
max36.620-36.619=0.001
mean4156.125-4147.500=8.625
σ1903.350-1890.836=12.514
min1555.000-1554.000=1.000
max7320.000-7273.000=47.000
mean11.872-12.871=-0.999
σ0.209-0.556=-0.346
min11.494-11.904=-0.411
max12.188-13.419=-1.231
op=table_records(), no indexes
mean41.932-41.932=0.001
σ0.001-0.001=0.000
min41.929-41.928=0.000
max41.933-41.933=0.000
mean7320.000-7273.000=47.000
σ0.000
min7320.000-7273.000=47.000
max7320.000-7273.000=47.000
mean15.120-15.407=-0.287
σ0.571-0.312=0.259
min13.793-14.745=-0.952
max15.636-15.746=-0.110
op=table_records(), repeat
mean41.929-41.928=0.001
σ0.002-0.002=-0.000
min41.928-41.927=0.001
max41.932-41.932=0.001
mean4156.125-4147.500=8.625
σ1903.350-1890.836=12.514
min1555.000-1554.000=1.000
max7320.000-7273.000=47.000
mean13.235-13.463=-0.229
σ1.753-1.581=0.172
min11.533-11.510=0.023
max15.720-15.397=0.322
op=vertical_lookup_list() on indexed, after table_records()
mean27.654-27.653=0.001
σ0.136-0.136=0.000
min27.604-27.603=0.001
max27.991-27.990=0.001
mean4156.125-4147.500=8.625
σ1903.350-1890.836=12.514
min1555.000-1554.000=1.000
max7320.000-7273.000=47.000
mean8.894-8.929=-0.036
σ0.430-0.372=0.058
min8.434-8.297=0.137
max9.431-9.479=-0.048
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.401-26.400=0.001
σ0.002-0.002=-0.000
min26.400-26.399=0.001
max26.404-26.403=0.001
mean4156.125-4147.500=8.625
σ1903.350-1890.836=12.514
min1555.000-1554.000=1.000
max7320.000-7273.000=47.000
mean6.578-6.765=-0.187
σ0.382-0.232=0.150
min6.169-6.419=-0.250
max7.301-7.134=0.167
op=vertical_lookup_row_list() on 2 indexed
mean25.668-25.666=0.002
σ0.003-0.002=0.001
min25.666-25.665=0.001
max25.674-25.668=0.005
mean4156.125-4147.500=8.625
σ1903.350-1890.836=12.514
min1555.000-1554.000=1.000
max7320.000-7273.000=47.000
mean5.801-6.150=-0.350
σ0.098-0.205=-0.107
min5.662-5.735=-0.073
max5.958-6.362=-0.404
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.970-25.968=0.001
σ0.002-0.002=-0.000
min25.968-25.967=0.001
max25.974-25.972=0.001
mean4156.125-4147.500=8.625
σ1903.350-1890.836=12.514
min1555.000-1554.000=1.000
max7320.000-7273.000=47.000
mean5.767-5.835=-0.068
σ0.081-0.067=0.014
min5.645-5.726=-0.081
max5.904-5.946=-0.042

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean2524.606-2524.606=-0.000
σ0.000-0.000=-0.000
min2524.606-2524.606=0.000
max2524.606-2524.606=-0.000
mean7320.000-7273.000=47.000
σ0.000
min7320.000-7273.000=47.000
max7320.000-7273.000=47.000
mean1065.872-1524.507=-458.635
σ69.328-771.362=-702.034
min1033.839-1040.096=-6.257
max1236.620-3176.521=-1939.901
op=load_product(), with indexes
mean8744.722-8744.721=0.000
σ0.469-0.469=0.000
min8744.553-8744.553=-0.000
max8745.883-8745.882=0.001
mean3576.625-3569.375=7.250
σ1798.905-1789.167=9.738
min1045.000-1046.000=-1.000
max5806.000-5775.000=31.000
mean4801.362-4738.056=63.306
σ1443.904-1552.005=-108.101
min3511.760-3533.519=-21.759
max6926.206-7160.475=-234.269

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense
mean62.078-62.078=0.000
σ0.002-0.002=-0.000
min62.076-62.076=0.000
max62.081-62.081=0.000
mean4156.125-4147.500=8.625
σ1903.350-1890.836=12.514
min1555.000-1554.000=1.000
max7320.000-7273.000=47.000
mean78.068-83.975=-5.907
σ5.778-13.726=-7.947
min69.559-70.323=-0.764
max88.569-101.704=-13.135
op=vertical_lookup_list() on unindexed missing Int
mean40.945-40.944=0.001
σ0.001-0.001=-0.000
min40.945-40.944=0.001
max40.946-40.945=0.001
mean7320.000-7273.000=47.000
σ0.000
min7320.000-7273.000=47.000
max7320.000-7273.000=47.000
mean106.051-107.404=-1.353
σ5.036-3.559=1.477
min96.509-103.026=-6.517
max112.619-112.670=-0.051
op=vertical_lookup_row_list() on unindexed small unique Int
mean66.686-66.686=0.001
σ0.000-0.000=-0.000
min66.686-66.685=0.001
max66.687-66.687=0.001
mean7320.000-7273.000=47.000
σ0.000
min7320.000-7273.000=47.000
max7320.000-7273.000=47.000
mean163.848-164.069=-0.221
σ7.011-4.836=2.175
min148.659-153.335=-4.676
max168.480-167.779=0.701

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean11656.900-11656.836=0.064
σ4.304-4.324=-0.020
min11655.306-11655.305=0.001
max11667.552-11667.538=0.013
mean4156.125-4147.500=8.625
σ1903.350-1890.836=12.514
min1555.000-1554.000=1.000
max7320.000-7273.000=47.000
mean6548.188-6263.055=285.134
σ1271.639-894.872=376.767
min4901.698-4941.006=-39.308
max7829.998-7487.161=342.837
op=vertical_lookup_list() on indexed
mean11560.066-11560.065=0.001
σ0.001-0.001=0.000
min11560.065-11560.064=0.001
max11560.067-11560.066=0.001
mean7320.000-7273.000=47.000
σ0.000
min7320.000-7273.000=47.000
max7320.000-7273.000=47.000
mean5329.552-5965.969=-636.417
σ1261.939-1613.816=-351.877
min4359.863-4416.973=-57.110
max7627.877-7811.108=-183.231

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean172.559
σ0.001
min172.558
max172.562
mean477.000
σ0.000
min477.000
max477.000
mean388.845-378.375=10.469
σ14.956-28.517=-13.562
min354.354-308.664=45.690
max399.367-396.137=3.229
op=pointedIndexFromKvs
rows=400000
mean357.322
σ0.001
min357.321
max357.325
mean560.000-561.000=-1.000
σ0.000
min560.000-561.000=-1.000
max560.000-561.000=-1.000
mean763.948-735.875=28.073
σ69.250-60.700=8.551
min595.962-635.814=-39.851
max802.342-799.530=2.811

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean1024.328-1008.499=15.829
σ0.759-0.759=-0.000
min1023.851-1008.028=15.823
max1027.464-1011.637=15.826
mean1053.000-1047.250=5.750
σ299.574-313.304=-13.730
min581.000-573.000=8.000
max1460.000-1510.000=-50.000
mean1.561-1.545=0.017
σ0.071-0.088=-0.017
min1.505-1.418=0.086
max1.787-1.760=0.027

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913
σ0.020
min0.903
max0.949
mean402.000
σ0.000
min402.000
max402.000
mean0.548-0.321=0.227
σ0.561-0.288=0.273
min0.248-0.138=0.110
max1.546-0.800=0.745
name=error_case.art
mean2.339
σ0.012
min2.333
max2.361
mean402.000
σ0.000
min402.000
max402.000
mean0.837-0.465=0.371
σ0.522-0.305=0.216
min0.577-0.318=0.259
max1.770-1.012=0.758
name=heavy_item.art
mean1.522-1.522=0.000
σ0.002
min1.521-1.521=0.000
max1.525-1.525=0.000
mean402.000
σ0.000
min402.000
max402.000
mean0.389-0.220=0.169
σ0.046-0.025=0.021
min0.358-0.197=0.161
max0.470-0.262=0.208
name=nested_use.art
mean2.357
σ0.002
min2.356
max2.360
mean402.000
σ0.000
min402.000
max402.000
mean0.584-0.335=0.249
σ0.031-0.018=0.013
min0.557-0.324=0.232
max0.634-0.368=0.266

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.840-964.741=0.099
σ0.011
min964.835-964.736=0.099
max964.859-964.760=0.099
mean431.000-435.600=-4.600
σ0.000-5.079=-5.079
min431.000
max431.000-441.000=-10.000
mean0.587-0.569=0.018
σ0.116-0.028=0.087
min0.467-0.521=-0.054
max0.754-0.592=0.162
name=cdr_enhanced_4k_loc.art
mean885.820-885.814=0.007
σ0.002
min885.819-885.812=0.007
max885.823-885.816=0.007
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean0.467-0.408=0.059
σ0.074-0.027=0.047
min0.383-0.372=0.011
max0.524-0.444=0.080

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.709-7.708=0.001
σ0.002
min7.708-7.707=0.001
max7.712-7.711=0.001
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean2.638-1.952=0.686
σ0.099-0.075=0.025
min2.467-1.844=0.624
max3.037-2.323=0.714
name=record_render_large_syntax_err.art
mean7.657-7.656=0.001
σ0.002
min7.656-7.655=0.001
max7.660-7.659=0.001
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean2.609-2.058=0.551
σ0.102-0.278=-0.176
min2.461-1.826=0.636
max3.074-3.296=-0.222
name=record_render_success.art
mean2.289-2.289=0.000
σ0.002
min2.289-2.288=0.000
max2.293-2.292=0.000
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean0.307-0.320=-0.012
σ0.014-0.028=-0.015
min0.291-0.293=-0.002
max0.419-0.514=-0.095
name=record_render_syntax_err.art
mean2.165-2.165=0.000
σ0.002
min2.164-2.164=0.000
max2.168-2.168=0.000
mean431.000-441.000=-10.000
σ0.000
min431.000-441.000=-10.000
max431.000-441.000=-10.000
mean0.289-0.310=-0.021
σ0.007-0.066=-0.059
min0.278-0.282=-0.004
max0.335-0.721=-0.386

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean298.763-298.078=0.684
σ0.985-1.047=-0.063
min298.373-297.698=0.675
max301.563-301.058=0.505
mean444.700
σ13.284-17.820=-4.537
min428.000-417.000=11.000
max466.000-471.000=-5.000
mean0.594-0.613=-0.019
σ0.020-0.011=0.009
min0.561-0.597=-0.036
max0.618-0.629=-0.011

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean239.532-222.008=17.524
σ0.000
min239.532-222.008=17.524
max239.532-222.008=17.524
mean1743.000-1754.000=-11.000
σ0.000
min1743.000-1754.000=-11.000
max1743.000-1754.000=-11.000
mean220.343-207.063=13.280
σ3.884-1.512=2.372
min211.920-204.935=6.985
max223.826-208.954=14.872
phase=macro expansion
mean6.130
σ0.000
min6.130
max6.130
mean1743.000-1754.000=-11.000
σ0.000
min1743.000-1754.000=-11.000
max1743.000-1754.000=-11.000
mean12.036-11.310=0.726
σ1.363-1.427=-0.064
min9.435-9.507=-0.072
max12.948-12.653=0.295
phase=parser
mean1042.365-1042.101=0.265
σ0.000-0.000=-0.000
min1042.365-1042.100=0.264
max1042.366-1042.101=0.264
mean1743.000-1754.000=-11.000
σ0.000
min1743.000-1754.000=-11.000
max1743.000-1754.000=-11.000
mean694.234-722.521=-28.287
σ61.727-63.277=-1.550
min591.419-597.374=-5.955
max734.854-832.038=-97.184
phase=processProdDef
mean6169.659-6169.660=-0.000
σ0.001
min6169.659
max6169.660
mean1743.000-1754.000=-11.000
σ0.000
min1743.000-1754.000=-11.000
max1743.000-1754.000=-11.000
mean5672.733-5561.810=110.923
σ1186.503-1268.978=-82.475
min3516.660-3883.278=-366.618
max7686.089-7043.030=643.058
phase=renaming
mean21.671-21.667=0.004
σ0.000
min21.671-21.667=0.004
max21.671-21.667=0.004
mean1743.000-1754.000=-11.000
σ0.000
min1743.000-1754.000=-11.000
max1743.000-1754.000=-11.000
mean46.485-40.347=6.138
σ2.880-8.062=-5.182
min40.195-29.649=10.545
max48.436-48.035=0.401
phase=type checker
mean990.446-958.461=31.985
σ0.066
min990.348-958.363=31.985
max990.536-958.551=31.985
mean1743.000-1754.000=-11.000
σ0.000
min1743.000-1754.000=-11.000
max1743.000-1754.000=-11.000
mean824.766-737.256=87.510
σ77.645-79.763=-2.118
min691.004-636.660=54.344
max900.191-869.637=30.555

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean293.918-293.686=0.233
σ0.038-0.040=-0.002
min293.864-293.632=0.233
max293.958-293.726=0.231
mean441.000-445.000=-4.000
σ2.739-0.000=2.739
min439.000-445.000=-6.000
max444.000-445.000=-1.000
mean0.393-0.400=-0.007
σ0.001-0.002=-0.001
min0.392-0.399=-0.006
max0.395-0.403=-0.009
iters=5
new-facts=1
prior-facts=6402
mean1320.192-1303.788=16.404
σ0.099-0.042=0.058
min1320.089-1303.744=16.346
max1320.318-1303.844=16.474
mean1111.200-1063.200=48.000
σ187.716-168.433=19.283
min891.000-841.000=50.000
max1373.000-1274.000=99.000
mean1.986-2.023=-0.038
σ0.104-0.186=-0.083
min1.911-1.819=0.092
max2.167-2.203=-0.035
iters=5
new-facts=6402
prior-facts=1
mean1559.732-1546.817=12.916
σ4.061-4.033=0.028
min1557.700-1544.795=12.905
max1566.990-1554.025=12.965
mean624.400-612.600=11.800
σ87.905-73.545=14.361
min494.000-497.000=-3.000
max729.000-691.000=38.000
mean2.265-2.287=-0.022
σ0.157-0.228=-0.071
min2.150-2.135=0.015
max2.542-2.687=-0.145
iters=5
new-facts=6402
prior-facts=6402
mean1882.282-1865.807=16.475
σ0.059-0.059=0.001
min1882.201-1865.747=16.454
max1882.349-1865.887=16.462
mean1726.400-1711.800=14.600
σ179.517-249.198=-69.681
min1504.000-1419.000=85.000
max1970.000-2064.000=-94.000
mean3.059-2.968=0.091
σ0.323-0.147=0.176
min2.736-2.856=-0.120
max3.483-3.216=0.267

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.783-1.820=-0.037
σ0.178-0.112=0.066
min1.498-1.647=-0.148
max1.929-1.908=0.021

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean453.537-450.090=3.446
σ585.531-579.265=6.266
min160.767-160.454=0.314
max1331.833-1318.987=12.846
mean102.250
σ8.500
min98.000
max115.000
mean440.000-443.000=-3.000
σ0.000
min440.000-443.000=-3.000
max440.000-443.000=-3.000
mean0.522-0.527=-0.005
σ0.495-0.509=-0.014
min0.273-0.270=0.003
max1.265-1.291=-0.026
policies=20
mean1608.035-1604.901=3.133
σ0.425-0.398=0.027
min1607.759-1604.628=3.131
max1608.668-1605.490=3.178
mean980.000
σ0.000
min980.000
max980.000
mean447.500-446.750=0.750
σ8.660-2.500=6.160
min440.000-443.000=-3.000
max455.000-448.000=7.000
mean2.703-2.662=0.041
σ0.012-0.011=0.001
min2.688-2.651=0.036
max2.716-2.674=0.042
policies=200
mean16088.589-16057.332=31.257
σ0.134-0.085=0.049
min16088.496-16057.281=31.214
max16088.787-16057.458=31.329
mean9800.000
σ0.000
min9800.000
max9800.000
mean496.250-495.250=1.000
σ5.377-4.856=0.521
min490.000-489.000=1.000
max503.000-500.000=3.000
mean27.274-27.008=0.266
σ0.185-0.070=0.116
min27.106-26.919=0.187
max27.536-27.065=0.471