Compare commits

Home / Compare 3c4a0f30 .. 8ecbe155

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.677-54.617=0.061
σ0.357-0.350=0.006
min54.548-54.487=0.061
max55.692-55.613=0.079
mean421.000-420.000=1.000
σ0.000
min421.000-420.000=1.000
max421.000-420.000=1.000
mean0.142-0.141=0.002
σ0.005-0.005=-0.000
min0.137-0.136=0.001
max0.150-0.152=-0.002
cache=y
dps=1
iters=10
policies=100
mean43.704-43.785=-0.080
σ0.005-0.006=-0.001
min43.694-43.779=-0.085
max43.712-43.799=-0.087
mean421.000-420.000=1.000
σ0.000
min421.000-420.000=1.000
max421.000-420.000=1.000
mean0.122-0.124=-0.001
σ0.004-0.005=-0.001
min0.118-0.119=-0.002
max0.129-0.130=-0.001

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.618-36.618=0.000
σ0.002-0.002=-0.000
min36.616-36.616=0.000
max36.620-36.620=-0.000
mean4156.125-4112.000=44.125
σ1903.350-1577.972=325.378
min1555.000-1915.000=-360.000
max7320.000-6032.000=1288.000
mean12.681-13.191=-0.510
σ0.180-1.788=-1.608
min12.423-12.228=0.194
max12.950-17.592=-4.642
op=column_set() on indexed, repeat
mean36.617-36.617=0.000
σ0.002-0.002=-0.000
min36.616-36.616=0.001
max36.620-36.620=0.001
mean4156.125-4112.000=44.125
σ1903.350-1577.972=325.378
min1555.000-1915.000=-360.000
max7320.000-6032.000=1288.000
mean11.872-12.910=-1.038
σ0.209-1.723=-1.514
min11.494-11.433=0.061
max12.188-16.799=-4.611
op=table_records(), no indexes
mean41.932-41.928=0.004
σ0.001-0.000=0.001
min41.929-41.928=0.001
max41.933-41.929=0.004
mean7320.000-6032.000=1288.000
σ0.000
min7320.000-6032.000=1288.000
max7320.000-6032.000=1288.000
mean15.120-13.722=1.399
σ0.571-1.063=-0.492
min13.793-11.606=2.187
max15.636-14.925=0.712
op=table_records(), repeat
mean41.929-41.928=0.001
σ0.002-0.002=0.000
min41.928-41.927=0.001
max41.932-41.931=0.001
mean4156.125-4046.000=110.125
σ1903.350-1628.502=274.849
min1555.000-1915.000=-360.000
max7320.000-6032.000=1288.000
mean13.235-12.955=0.280
σ1.753-2.149=-0.396
min11.533-10.399=1.134
max15.720-14.931=0.789
op=vertical_lookup_list() on indexed, after table_records()
mean27.654-27.605=0.049
σ0.136-0.002=0.135
min27.604-27.603=0.001
max27.991-27.607=0.384
mean4156.125-4046.000=110.125
σ1903.350-1628.502=274.849
min1555.000-1915.000=-360.000
max7320.000-6032.000=1288.000
mean8.894-8.530=0.364
σ0.430-0.391=0.039
min8.434-7.995=0.440
max9.431-9.070=0.361
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.401-26.400=0.001
σ0.002-0.002=-0.000
min26.400-26.399=0.001
max26.404-26.403=0.001
mean4156.125-4046.000=110.125
σ1903.350-1628.502=274.849
min1555.000-1915.000=-360.000
max7320.000-6032.000=1288.000
mean6.578-6.416=0.162
σ0.382-0.171=0.211
min6.169-6.185=-0.016
max7.301-6.651=0.650
op=vertical_lookup_row_list() on 2 indexed
mean25.668-25.666=0.002
σ0.003-0.002=0.001
min25.666-25.665=0.001
max25.674-25.669=0.004
mean4156.125-4112.000=44.125
σ1903.350-1577.972=325.378
min1555.000-1915.000=-360.000
max7320.000-6032.000=1288.000
mean5.801-5.737=0.064
σ0.098-0.555=-0.457
min5.662-5.385=0.277
max5.958-7.090=-1.132
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.970-25.968=0.001
σ0.002-0.002=0.000
min25.968-25.967=0.001
max25.974-25.971=0.003
mean4156.125-4112.000=44.125
σ1903.350-1577.972=325.378
min1555.000-1915.000=-360.000
max7320.000-6032.000=1288.000
mean5.767-5.599=0.168
σ0.081-0.528=-0.447
min5.645-5.358=0.287
max5.904-6.902=-0.998

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean2524.606-2524.593=0.012
σ0.000-0.000=-0.000
min2524.606-2524.593=0.012
max2524.606-2524.594=0.012
mean7320.000-6032.000=1288.000
σ0.000
min7320.000-6032.000=1288.000
max7320.000-6032.000=1288.000
mean1065.872-1437.378=-371.505
σ69.328-625.108=-555.779
min1033.839-1027.584=6.255
max1236.620-2651.473=-1414.853
op=load_product(), with indexes
mean8744.722-8744.559=0.163
σ0.469-0.033=0.436
min8744.553-8744.544=0.009
max8745.883-8744.640=1.243
mean3576.625-3816.125=-239.500
σ1798.905-1666.942=131.963
min1045.000-1915.000=-870.000
max5806.000-6032.000=-226.000
mean4801.362-4478.887=322.475
σ1443.904-1363.206=80.698
min3511.760-3550.577=-38.817
max6926.206-7502.152=-575.946

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense
mean62.078-62.078=0.000
σ0.002-0.002=0.000
min62.076-62.076=0.000
max62.081-62.081=0.000
mean4156.125-4112.000=44.125
σ1903.350-1577.972=325.378
min1555.000-1915.000=-360.000
max7320.000-6032.000=1288.000
mean78.068-148.251=-70.183
σ5.778-190.410=-184.632
min69.559-71.361=-1.802
max88.569-618.733=-530.164
op=vertical_lookup_list() on unindexed missing Int
mean40.945-40.944=0.001
σ0.001-0.001=-0.000
min40.945-40.944=0.001
max40.946-40.945=0.001
mean7320.000-6032.000=1288.000
σ0.000
min7320.000-6032.000=1288.000
max7320.000-6032.000=1288.000
mean106.051-96.547=9.504
σ5.036-10.002=-4.966
min96.509-75.687=20.822
max112.619-107.244=5.375
op=vertical_lookup_row_list() on unindexed small unique Int
mean66.686-66.686=0.001
σ0.000-0.000=-0.000
min66.686-66.685=0.001
max66.687-66.686=0.001
mean7320.000-6032.000=1288.000
σ0.000
min7320.000-6032.000=1288.000
max7320.000-6032.000=1288.000
mean163.848-231.903=-68.055
σ7.011-247.667=-240.655
min148.659-112.011=36.648
max168.480-843.605=-675.125

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean11656.900-11656.833=0.068
σ4.304-4.318=-0.014
min11655.306-11655.305=0.001
max11667.552-11667.521=0.031
mean4156.125-4046.000=110.125
σ1903.350-1628.502=274.849
min1555.000-1915.000=-360.000
max7320.000-6032.000=1288.000
mean6548.188-6248.247=299.942
σ1271.639-1245.688=25.951
min4901.698-4909.141=-7.443
max7829.998-8282.019=-452.021
op=vertical_lookup_list() on indexed
mean11560.066-11560.068=-0.003
σ0.001-0.001=0.000
min11560.065-11560.067=-0.002
max11560.067-11560.069=-0.002
mean7320.000-6032.000=1288.000
σ0.000
min7320.000-6032.000=1288.000
max7320.000-6032.000=1288.000
mean5329.552-6710.566=-1381.013
σ1261.939-1216.705=45.234
min4359.863-4403.511=-43.647
max7627.877-8101.423=-473.546

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean172.559-172.583=-0.025
σ0.001-0.026=-0.025
min172.558
max172.562-172.630=-0.068
mean477.000-1915.000=-1438.000
σ0.000
min477.000-1915.000=-1438.000
max477.000-1915.000=-1438.000
mean388.845-357.309=31.535
σ14.956-42.655=-27.699
min354.354-269.836=84.517
max399.367-407.123=-7.757
op=pointedIndexFromKvs
rows=400000
mean357.322-357.362=-0.041
σ0.001-0.040=-0.039
min357.321
max357.325-357.412=-0.086
mean560.000-1915.000=-1355.000
σ0.000
min560.000-1915.000=-1355.000
max560.000-1915.000=-1355.000
mean763.948-771.261=-7.313
σ69.250-54.326=14.925
min595.962-642.851=-46.888
max802.342-808.220=-5.879

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean1024.328-1008.185=16.143
σ0.759-0.874=-0.115
min1023.851-1007.685=16.166
max1027.464-1011.803=15.661
mean1053.000-1048.900=4.100
σ299.574-307.178=-7.604
min581.000-562.000=19.000
max1460.000-1481.000=-21.000
mean1.561-1.550=0.011
σ0.071-0.092=-0.022
min1.505-1.453=0.052
max1.787-1.823=-0.036

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913-0.883=0.030
σ0.020-0.002=0.018
min0.903-0.881=0.021
max0.949-0.886=0.063
mean402.000-1481.000=-1079.000
σ0.000
min402.000-1481.000=-1079.000
max402.000-1481.000=-1079.000
mean0.548-0.408=0.140
σ0.561-0.133=0.427
min0.248-0.330=-0.082
max1.546-0.644=0.902
name=error_case.art
mean2.339-2.333=0.006
σ0.012-0.003=0.010
min2.333-2.332=0.001
max2.361-2.337=0.024
mean402.000-1481.000=-1079.000
σ0.000
min402.000-1481.000=-1079.000
max402.000-1481.000=-1079.000
mean0.837-0.883=-0.046
σ0.522-0.085=0.437
min0.577-0.827=-0.249
max1.770-1.030=0.739
name=heavy_item.art
mean1.522-1.521=0.001
σ0.002-0.002=0.000
min1.521-1.520=0.001
max1.525-1.524=0.001
mean402.000-1481.000=-1079.000
σ0.000
min402.000-1481.000=-1079.000
max402.000-1481.000=-1079.000
mean0.389-0.532=-0.143
σ0.046-0.053=-0.007
min0.358-0.491=-0.133
max0.470-0.620=-0.151
name=nested_use.art
mean2.357-2.355=0.001
σ0.002
min2.356-2.355=0.001
max2.360-2.358=0.001
mean402.000-1481.000=-1079.000
σ0.000
min402.000-1481.000=-1079.000
max402.000-1481.000=-1079.000
mean0.584-0.828=-0.244
σ0.031-0.043=-0.012
min0.557-0.791=-0.235
max0.634-0.900=-0.266

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.840-963.541=1.299
σ0.011-0.028=-0.017
min964.835-963.523=1.312
max964.859-963.589=1.270
mean431.000-1481.000=-1050.000
σ0.000
min431.000-1481.000=-1050.000
max431.000-1481.000=-1050.000
mean0.587-0.825=-0.238
σ0.116-0.054=0.062
min0.467-0.750=-0.283
max0.754-0.871=-0.117
name=cdr_enhanced_4k_loc.art
mean885.820-884.854=0.967
σ0.002-0.006=-0.004
min885.819-884.847=0.972
max885.823-884.861=0.962
mean431.000-1481.000=-1050.000
σ0.000
min431.000-1481.000=-1050.000
max431.000-1481.000=-1050.000
mean0.467-0.739=-0.273
σ0.074-0.017=0.057
min0.383-0.709=-0.326
max0.524-0.750=-0.226

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.709-7.706=0.002
σ0.002-0.002=-0.000
min7.708-7.705=0.002
max7.712-7.712=-0.000
mean431.000-1481.000=-1050.000
σ0.000
min431.000-1481.000=-1050.000
max431.000-1481.000=-1050.000
mean2.638-4.246=-1.608
σ0.099-0.098=0.001
min2.467-4.087=-1.619
max3.037-4.652=-1.615
name=record_render_large_syntax_err.art
mean7.657-7.655=0.002
σ0.002
min7.656-7.654=0.002
max7.660-7.658=0.002
mean431.000-1481.000=-1050.000
σ0.000
min431.000-1481.000=-1050.000
max431.000-1481.000=-1050.000
mean2.609-4.244=-1.634
σ0.102-0.077=0.025
min2.461-4.089=-1.628
max3.074-4.570=-1.496
name=record_render_success.art
mean2.289-2.288=0.002
σ0.002
min2.289-2.287=0.002
max2.293-2.291=0.002
mean431.000-1481.000=-1050.000
σ0.000
min431.000-1481.000=-1050.000
max431.000-1481.000=-1050.000
mean0.307-0.743=-0.435
σ0.014-0.019=-0.005
min0.291-0.706=-0.415
max0.419-0.859=-0.441
name=record_render_syntax_err.art
mean2.165-2.163=0.002
σ0.002
min2.164-2.163=0.002
max2.168-2.166=0.002
mean431.000-1481.000=-1050.000
σ0.000
min431.000-1481.000=-1050.000
max431.000-1481.000=-1050.000
mean0.289-0.700=-0.411
σ0.007-0.015=-0.007
min0.278-0.674=-0.396
max0.335-0.786=-0.451

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean298.763-297.413=1.350
σ0.985-0.287=0.697
min298.373-297.173=1.200
max301.563-298.197=3.366
mean444.700-1915.000=-1470.300
σ13.284-0.000=13.284
min428.000-1915.000=-1487.000
max466.000-1915.000=-1449.000
mean0.594-0.607=-0.014
σ0.020-0.036=-0.016
min0.561-0.569=-0.009
max0.618-0.694=-0.075

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean239.532-221.996=17.536
σ0.000-0.000=-0.000
min239.532-221.996=17.536
max239.532-221.996=17.536
mean1743.000-1817.000=-74.000
σ0.000
min1743.000-1817.000=-74.000
max1743.000-1817.000=-74.000
mean220.343-206.686=13.657
σ3.884-1.766=2.118
min211.920-204.391=7.529
max223.826-208.674=15.152
phase=macro expansion
mean6.130-6.127=0.002
σ0.000
min6.130-6.127=0.002
max6.130-6.127=0.002
mean1743.000-1817.000=-74.000
σ0.000
min1743.000-1817.000=-74.000
max1743.000-1817.000=-74.000
mean12.036-10.758=1.278
σ1.363-2.099=-0.737
min9.435-7.534=1.901
max12.948-13.800=-0.851
phase=parser
mean1042.365-1041.086=1.280
σ0.000-0.000=0.000
min1042.365-1041.086=1.279
max1042.366-1041.086=1.280
mean1743.000-1817.000=-74.000
σ0.000
min1743.000-1817.000=-74.000
max1743.000-1817.000=-74.000
mean694.234-636.292=57.942
σ61.727-105.698=-43.970
min591.419-421.881=169.538
max734.854-728.774=6.080
phase=processProdDef
mean6169.659-6169.660=-0.001
σ0.001-0.000=0.000
min6169.659-6169.660=-0.002
max6169.660-6169.661=-0.001
mean1743.000-1817.000=-74.000
σ0.000
min1743.000-1817.000=-74.000
max1743.000-1817.000=-74.000
mean5672.733-6563.675=-890.942
σ1186.503-977.444=209.058
min3516.660-4343.988=-827.328
max7686.089-7272.435=413.654
phase=renaming
mean21.671-21.666=0.005
σ0.000
min21.671-21.666=0.005
max21.671-21.666=0.005
mean1743.000-1817.000=-74.000
σ0.000
min1743.000-1817.000=-74.000
max1743.000-1817.000=-74.000
mean46.485-41.386=5.099
σ2.880-8.321=-5.441
min40.195-26.374=13.821
max48.436-51.814=-3.377
phase=type checker
mean990.446-958.665=31.780
σ0.066-0.055=0.011
min990.348-958.579=31.769
max990.536-958.736=31.800
mean1743.000-1817.000=-74.000
σ0.000
min1743.000-1817.000=-74.000
max1743.000-1817.000=-74.000
mean824.766-792.979=31.787
σ77.645-111.448=-33.803
min691.004-632.669=58.335
max900.191-970.706=-70.515

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean293.918-289.764=4.154
σ0.038-0.036=0.002
min293.864-289.716=4.148
max293.958-289.802=4.156
mean441.000-1817.000=-1376.000
σ2.739-0.000=2.739
min439.000-1817.000=-1378.000
max444.000-1817.000=-1373.000
mean0.393-0.390=0.004
σ0.001-0.001=-0.000
min0.392-0.389=0.003
max0.395-0.392=0.002
iters=5
new-facts=1
prior-facts=6402
mean1320.192-1299.601=20.591
σ0.099-0.122=-0.023
min1320.089-1299.501=20.588
max1320.318-1299.807=20.510
mean1111.200-1817.000=-705.800
σ187.716-0.000=187.716
min891.000-1817.000=-926.000
max1373.000-1817.000=-444.000
mean1.986-1.935=0.051
σ0.104-0.169=-0.065
min1.911-1.815=0.096
max2.167-2.196=-0.029
iters=5
new-facts=6402
prior-facts=1
mean1559.732-1542.630=17.102
σ4.061-3.998=0.063
min1557.700-1540.660=17.040
max1566.990-1549.776=17.214
mean624.400-1817.000=-1192.600
σ87.905-0.000=87.905
min494.000-1817.000=-1323.000
max729.000-1817.000=-1088.000
mean2.265-2.239=0.027
σ0.157-0.094=0.063
min2.150-2.131=0.019
max2.542-2.372=0.170
iters=5
new-facts=6402
prior-facts=6402
mean1882.282-1861.629=20.653
σ0.059-0.047=0.012
min1882.201-1861.579=20.622
max1882.349-1861.690=20.659
mean1726.400-1856.200=-129.800
σ179.517-53.677=125.840
min1504.000-1817.000=-313.000
max1970.000-1915.000=55.000
mean3.059-3.114=-0.055
σ0.323-0.332=-0.009
min2.736-2.862=-0.126
max3.483-3.693=-0.210

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.783-1.806=-0.023
σ0.178-0.109=0.069
min1.498-1.628=-0.130
max1.929-1.898=0.030

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean453.537-446.183=7.354
σ585.531-577.141=8.390
min160.767-157.610=3.157
max1331.833-1311.895=19.939
mean102.250-96.250=6.000
σ8.500
min98.000-92.000=6.000
max115.000-109.000=6.000
mean440.000-1817.000=-1377.000
σ0.000
min440.000-1817.000=-1377.000
max440.000-1817.000=-1377.000
mean0.522-0.553=-0.031
σ0.495-0.577=-0.082
min0.273-0.258=0.016
max1.265-1.419=-0.154
policies=20
mean1608.035-1576.410=31.625
σ0.425-0.346=0.079
min1607.759-1576.174=31.585
max1608.668-1576.924=31.744
mean980.000-920.000=60.000
σ0.000
min980.000-920.000=60.000
max980.000-920.000=60.000
mean447.500-1817.000=-1369.500
σ8.660-0.000=8.660
min440.000-1817.000=-1377.000
max455.000-1817.000=-1362.000
mean2.703-2.619=0.084
σ0.012-0.020=-0.008
min2.688-2.590=0.098
max2.716-2.636=0.080
policies=200
mean16088.589-15772.234=316.355
σ0.134-0.141=-0.007
min16088.496-15772.109=316.387
max16088.787-15772.428=316.360
mean9800.000-9200.000=600.000
σ0.000
min9800.000-9200.000=600.000
max9800.000-9200.000=600.000
mean496.250-1817.000=-1320.750
σ5.377-0.000=5.377
min490.000-1817.000=-1327.000
max503.000-1817.000=-1314.000
mean27.274-26.634=0.640
σ0.185-0.217=-0.032
min27.106-26.432=0.674
max27.536-26.855=0.681