Compare commits

Home / Compare ca4dd1eb .. c4cf1180

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.677-54.715=-0.037
σ0.357-0.364=-0.007
min54.548-54.581=-0.033
max55.692-55.750=-0.058
mean421.000-418.000=3.000
σ0.000
min421.000-418.000=3.000
max421.000-418.000=3.000
mean0.142-0.141=0.001
σ0.005-0.003=0.002
min0.137-0.138=-0.001
max0.150-0.147=0.003
cache=y
dps=1
iters=10
policies=100
mean43.704-43.788=-0.083
σ0.005-0.007=-0.001
min43.694-43.778=-0.084
max43.712-43.800=-0.088
mean421.000-418.000=3.000
σ0.000
min421.000-418.000=3.000
max421.000-418.000=3.000
mean0.122-0.123=-0.001
σ0.004-0.004=-0.000
min0.118-0.119=-0.002
max0.129-0.130=-0.001

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.618-36.618=0.000
σ0.002-0.002=-0.000
min36.616-36.616=0.000
max36.620-36.620=-0.000
mean4156.125-4099.875=56.250
σ1903.350-1586.364=316.986
min1555.000-1898.000=-343.000
max7320.000-6030.000=1290.000
mean12.681-12.234=0.447
σ0.180-0.210=-0.030
min12.423-11.989=0.434
max12.950-12.573=0.376
op=column_set() on indexed, repeat
mean36.617-36.617=0.000
σ0.002-0.002=0.000
min36.616-36.616=-0.000
max36.620-36.620=0.000
mean4156.125-4099.875=56.250
σ1903.350-1586.364=316.986
min1555.000-1898.000=-343.000
max7320.000-6030.000=1290.000
mean11.872-12.546=-0.674
σ0.209-0.432=-0.222
min11.494-11.532=-0.038
max12.188-12.892=-0.704
op=table_records(), no indexes
mean41.932-41.929=0.003
σ0.001-0.000=0.001
min41.929-41.928=0.000
max41.933-41.929=0.004
mean7320.000-6030.000=1290.000
σ0.000
min7320.000-6030.000=1290.000
max7320.000-6030.000=1290.000
mean15.120-13.323=1.797
σ0.571-1.254=-0.683
min13.793-10.892=2.901
max15.636-14.876=0.760
op=table_records(), repeat
mean41.929-41.929=-0.000
σ0.002-0.002=-0.000
min41.928-41.928=0.000
max41.932-41.931=0.001
mean4156.125-4026.000=130.125
σ1903.350-1643.940=259.410
min1555.000-1898.000=-343.000
max7320.000-6030.000=1290.000
mean13.235-12.811=0.423
σ1.753-1.766=-0.012
min11.533-10.352=1.182
max15.720-14.662=1.057
op=vertical_lookup_list() on indexed, after table_records()
mean27.654-27.605=0.048
σ0.136-0.002=0.135
min27.604-27.604=-0.000
max27.991-27.608=0.383
mean4156.125-4026.000=130.125
σ1903.350-1643.940=259.410
min1555.000-1898.000=-343.000
max7320.000-6030.000=1290.000
mean8.894-8.387=0.507
σ0.430-0.355=0.075
min8.434-7.858=0.577
max9.431-8.785=0.646
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.401-26.401=0.000
σ0.002-0.002=0.000
min26.400-26.400=-0.000
max26.404-26.404=0.000
mean4156.125-4026.000=130.125
σ1903.350-1643.940=259.410
min1555.000-1898.000=-343.000
max7320.000-6030.000=1290.000
mean6.578-6.479=0.098
σ0.382-0.133=0.249
min6.169-6.273=-0.104
max7.301-6.689=0.612
op=vertical_lookup_row_list() on 2 indexed
mean25.668-25.668=-0.000
σ0.003-0.004=-0.001
min25.666-25.666=-0.000
max25.674-25.677=-0.003
mean4156.125-4099.875=56.250
σ1903.350-1586.364=316.986
min1555.000-1898.000=-343.000
max7320.000-6030.000=1290.000
mean5.801-5.590=0.211
σ0.098-0.072=0.026
min5.662-5.483=0.179
max5.958-5.689=0.269
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.970-25.970=0.000
σ0.002-0.002=0.000
min25.968-25.968=0.000
max25.974-25.972=0.001
mean4156.125-4099.875=56.250
σ1903.350-1586.364=316.986
min1555.000-1898.000=-343.000
max7320.000-6030.000=1290.000
mean5.767-5.408=0.359
σ0.081-0.082=-0.001
min5.645-5.300=0.345
max5.904-5.530=0.374

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean2524.606-2524.603=0.003
σ0.000-0.000=0.000
min2524.606-2524.603=0.003
max2524.606-2524.603=0.003
mean7320.000-6030.000=1290.000
σ0.000
min7320.000-6030.000=1290.000
max7320.000-6030.000=1290.000
mean1065.872-1497.312=-431.439
σ69.328-713.651=-644.323
min1033.839-1032.543=1.296
max1236.620-2822.370=-1585.750
op=load_product(), with indexes
mean8744.722-8744.568=0.154
σ0.469-0.033=0.436
min8744.553-8744.553=-0.000
max8745.883-8744.649=1.234
mean3576.625-3798.250=-221.625
σ1798.905-1678.397=120.508
min1045.000-1898.000=-853.000
max5806.000-6030.000=-224.000
mean4801.362-4337.222=464.140
σ1443.904-1123.196=320.708
min3511.760-3517.717=-5.957
max6926.206-6301.003=625.202

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense
mean62.078-62.078=-0.000
σ0.002-0.002=0.000
min62.076-62.076=-0.000
max62.081-62.080=0.001
mean4156.125-4099.875=56.250
σ1903.350-1586.364=316.986
min1555.000-1898.000=-343.000
max7320.000-6030.000=1290.000
mean78.068-108.298=-30.230
σ5.778-86.192=-80.413
min69.559-69.886=-0.327
max88.569-320.575=-232.006
op=vertical_lookup_list() on unindexed missing Int
mean40.945-40.945=0.000
σ0.001-0.001=-0.000
min40.945-40.945=0.000
max40.946-40.946=-0.000
mean7320.000-6030.000=1290.000
σ0.000
min7320.000-6030.000=1290.000
max7320.000-6030.000=1290.000
mean106.051-95.069=10.982
σ5.036-8.464=-3.428
min96.509-83.380=13.130
max112.619-105.582=7.037
op=vertical_lookup_row_list() on unindexed small unique Int
mean66.686-66.687=-0.000
σ0.000-0.000=-0.000
min66.686-66.686=0.000
max66.687-66.687=-0.000
mean7320.000-6030.000=1290.000
σ0.000
min7320.000-6030.000=1290.000
max7320.000-6030.000=1290.000
mean163.848-224.391=-60.543
σ7.011-229.304=-222.293
min148.659-123.809=24.849
max168.480-790.939=-622.459

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean11656.900-11656.959=-0.059
σ4.304-4.321=-0.017
min11655.306-11655.431=-0.124
max11667.552-11667.652=-0.101
mean4156.125-4026.000=130.125
σ1903.350-1643.940=259.410
min1555.000-1898.000=-343.000
max7320.000-6030.000=1290.000
mean6548.188-6260.898=287.291
σ1271.639-1327.068=-55.429
min4901.698-4906.618=-4.920
max7829.998-8138.143=-308.146
op=vertical_lookup_list() on indexed
mean11560.066-11560.070=-0.004
σ0.001-0.002=-0.001
min11560.065-11560.067=-0.002
max11560.067-11560.074=-0.007
mean7320.000-6030.000=1290.000
σ0.000
min7320.000-6030.000=1290.000
max7320.000-6030.000=1290.000
mean5329.552-6447.574=-1118.021
σ1261.939-1446.836=-184.897
min4359.863-4414.287=-54.423
max7627.877-8262.481=-634.604

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean172.559-172.580=-0.021
σ0.001-0.022=-0.021
min172.558
max172.562-172.616=-0.054
mean477.000-1898.000=-1421.000
σ0.000
min477.000-1898.000=-1421.000
max477.000-1898.000=-1421.000
mean388.845-331.566=57.278
σ14.956-38.254=-23.298
min354.354-265.125=89.229
max399.367-395.025=4.342
op=pointedIndexFromKvs
rows=400000
mean357.322-357.366=-0.045
σ0.001-0.043=-0.041
min357.321
max357.325-357.412=-0.087
mean560.000-1898.000=-1338.000
σ0.000
min560.000-1898.000=-1338.000
max560.000-1898.000=-1338.000
mean763.948-759.253=4.696
σ69.250-58.430=10.820
min595.962-627.467=-31.505
max802.342-800.183=2.159

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean1024.328-1024.306=0.022
σ0.759-0.881=-0.122
min1023.851-1023.807=0.044
max1027.464-1027.951=-0.488
mean1053.000-1043.800=9.200
σ299.574-293.580=5.994
min581.000-550.000=31.000
max1460.000-1438.000=22.000
mean1.561-1.553=0.009
σ0.071-0.075=-0.004
min1.505-1.450=0.054
max1.787-1.781=0.006

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913-0.883=0.030
σ0.020-0.002=0.018
min0.903-0.881=0.021
max0.949-0.886=0.063
mean402.000-1438.000=-1036.000
σ0.000
min402.000-1438.000=-1036.000
max402.000-1438.000=-1036.000
mean0.548-0.428=0.120
σ0.561-0.137=0.423
min0.248-0.358=-0.110
max1.546-0.673=0.872
name=error_case.art
mean2.339-2.333=0.006
σ0.012-0.003=0.010
min2.333-2.332=0.001
max2.361-2.337=0.024
mean402.000-1438.000=-1036.000
σ0.000
min402.000-1438.000=-1036.000
max402.000-1438.000=-1036.000
mean0.837-0.900=-0.063
σ0.522-0.084=0.438
min0.577-0.844=-0.267
max1.770-1.044=0.726
name=heavy_item.art
mean1.522-1.521=0.001
σ0.002-0.002=0.000
min1.521-1.520=0.001
max1.525-1.524=0.001
mean402.000-1438.000=-1036.000
σ0.000
min402.000-1438.000=-1036.000
max402.000-1438.000=-1036.000
mean0.389-0.572=-0.183
σ0.046-0.075=-0.029
min0.358-0.520=-0.162
max0.470-0.699=-0.229
name=nested_use.art
mean2.357-2.355=0.001
σ0.002
min2.356-2.355=0.001
max2.360-2.358=0.001
mean402.000-1438.000=-1036.000
σ0.000
min402.000-1438.000=-1036.000
max402.000-1438.000=-1036.000
mean0.584-0.858=-0.274
σ0.031-0.046=-0.015
min0.557-0.815=-0.259
max0.634-0.929=-0.295

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.840-964.021=0.820
σ0.011-0.027=-0.017
min964.835-964.001=0.834
max964.859-964.068=0.791
mean431.000-1438.000=-1007.000
σ0.000
min431.000-1438.000=-1007.000
max431.000-1438.000=-1007.000
mean0.587-0.861=-0.274
σ0.116-0.032=0.084
min0.467-0.805=-0.338
max0.754-0.885=-0.131
name=cdr_enhanced_4k_loc.art
mean885.820-884.895=0.925
σ0.002-0.006=-0.004
min885.819-884.889=0.930
max885.823-884.901=0.921
mean431.000-1438.000=-1007.000
σ0.000
min431.000-1438.000=-1007.000
max431.000-1438.000=-1007.000
mean0.467-0.704=-0.237
σ0.074-0.054=0.021
min0.383-0.621=-0.238
max0.524-0.753=-0.229

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.709-7.707=0.001
σ0.002-0.002=-0.000
min7.708-7.707=0.001
max7.712-7.711=0.001
mean431.000-1438.000=-1007.000
σ0.000
min431.000-1438.000=-1007.000
max431.000-1438.000=-1007.000
mean2.638-4.315=-1.677
σ0.099-0.072=0.027
min2.467-4.182=-1.715
max3.037-4.553=-1.516
name=record_render_large_syntax_err.art
mean7.657-7.656=0.001
σ0.002
min7.656-7.655=0.001
max7.660-7.659=0.001
mean431.000-1438.000=-1007.000
σ0.000
min431.000-1438.000=-1007.000
max431.000-1438.000=-1007.000
mean2.609-4.286=-1.677
σ0.102-0.079=0.023
min2.461-4.131=-1.670
max3.074-4.541=-1.467
name=record_render_success.art
mean2.289-2.288=0.001
σ0.002
min2.289-2.287=0.001
max2.293-2.291=0.001
mean431.000-1438.000=-1007.000
σ0.000
min431.000-1438.000=-1007.000
max431.000-1438.000=-1007.000
mean0.307-0.754=-0.447
σ0.014-0.038=-0.024
min0.291-0.582=-0.291
max0.419-0.897=-0.478
name=record_render_syntax_err.art
mean2.165-2.164=0.001
σ0.002
min2.164-2.163=0.001
max2.168-2.167=0.001
mean431.000-1438.000=-1007.000
σ0.000
min431.000-1438.000=-1007.000
max431.000-1438.000=-1007.000
mean0.289-0.730=-0.441
σ0.007-0.015=-0.008
min0.278-0.695=-0.417
max0.335-0.799=-0.464

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean298.763-298.407=0.356
σ0.985-0.285=0.700
min298.373-298.119=0.254
max301.563-299.132=2.431
mean444.700-1898.000=-1453.300
σ13.284-0.000=13.284
min428.000-1898.000=-1470.000
max466.000-1898.000=-1432.000
mean0.594-0.599=-0.005
σ0.020-0.047=-0.027
min0.561-0.509=0.051
max0.618-0.704=-0.086

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean239.532-239.536=-0.004
σ0.000
min239.532-239.536=-0.004
max239.532-239.536=-0.004
mean1743.000-1456.000=287.000
σ0.000-50.912=-50.912
min1743.000-1438.000=305.000
max1743.000-1582.000=161.000
mean220.343-223.725=-3.381
σ3.884-2.077=1.808
min211.920-220.583=-8.664
max223.826-226.297=-2.471
phase=macro expansion
mean6.130-6.133=-0.004
σ0.000
min6.130-6.133=-0.004
max6.130-6.133=-0.004
mean1743.000-1456.000=287.000
σ0.000-50.912=-50.912
min1743.000-1438.000=305.000
max1743.000-1582.000=161.000
mean12.036-9.268=2.768
σ1.363-1.796=-0.434
min9.435-6.344=3.091
max12.948-10.682=2.267
phase=parser
mean1042.365-1041.423=0.943
σ0.000-0.011=-0.011
min1042.365-1041.396=0.969
max1042.366-1041.427=0.939
mean1743.000-1456.000=287.000
σ0.000-50.912=-50.912
min1743.000-1438.000=305.000
max1743.000-1582.000=161.000
mean694.234-655.880=38.354
σ61.727-171.515=-109.788
min591.419-383.563=207.855
max734.854-835.482=-100.629
phase=processProdDef
mean6169.659-6169.656=0.003
σ0.001-0.000=0.000
min6169.659-6169.656=0.003
max6169.660-6169.657=0.003
mean1743.000-1496.250=246.750
σ0.000-117.883=-117.883
min1743.000-1438.000=305.000
max1743.000-1760.000=-17.000
mean5672.733-6436.722=-763.988
σ1186.503-1061.302=125.201
min3516.660-3898.628=-381.968
max7686.089-7223.938=462.151
phase=renaming
mean21.671
σ0.000
min21.671
max21.671
mean1743.000-1456.000=287.000
σ0.000-50.912=-50.912
min1743.000-1438.000=305.000
max1743.000-1582.000=161.000
mean46.485-32.069=14.416
σ2.880-1.960=0.920
min40.195-29.426=10.768
max48.436-35.879=12.557
phase=type checker
mean990.446-990.675=-0.229
σ0.066-0.055=0.011
min990.348-990.589=-0.241
max990.536-990.745=-0.209
mean1743.000-1456.000=287.000
σ0.000-50.912=-50.912
min1743.000-1438.000=305.000
max1743.000-1582.000=161.000
mean824.766-721.177=103.589
σ77.645-52.237=25.408
min691.004-664.705=26.299
max900.191-821.131=79.060

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean293.918-293.229=0.689
σ0.038-0.039=-0.000
min293.864-293.176=0.688
max293.958-293.270=0.688
mean441.000-1760.000=-1319.000
σ2.739-0.000=2.739
min439.000-1760.000=-1321.000
max444.000-1760.000=-1316.000
mean0.393-0.415=-0.021
σ0.001-0.015=-0.014
min0.392-0.402=-0.010
max0.395-0.439=-0.044
iters=5
new-facts=1
prior-facts=6402
mean1320.192-1319.269=0.923
σ0.099-0.060=0.040
min1320.089-1319.195=0.894
max1320.318-1319.356=0.961
mean1111.200-1760.000=-648.800
σ187.716-0.000=187.716
min891.000-1760.000=-869.000
max1373.000-1760.000=-387.000
mean1.986-2.033=-0.048
σ0.104-0.167=-0.063
min1.911-1.852=0.059
max2.167-2.253=-0.086
iters=5
new-facts=6402
prior-facts=1
mean1559.732-1558.840=0.892
σ4.061-4.025=0.036
min1557.700-1556.938=0.762
max1566.990-1566.038=0.953
mean624.400-1760.000=-1135.600
σ87.905-0.000=87.905
min494.000-1760.000=-1266.000
max729.000-1760.000=-1031.000
mean2.265-2.485=-0.219
σ0.157-0.263=-0.105
min2.150-2.141=0.009
max2.542-2.690=-0.148
iters=5
new-facts=6402
prior-facts=6402
mean1882.282-1881.335=0.948
σ0.059-0.053=0.006
min1882.201-1881.269=0.932
max1882.349-1881.397=0.952
mean1726.400-1815.200=-88.800
σ179.517-75.586=103.931
min1504.000-1760.000=-256.000
max1970.000-1898.000=72.000
mean3.059-3.080=-0.021
σ0.323-0.135=0.188
min2.736-2.917=-0.181
max3.483-3.274=0.209

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.783-1.763=0.020
σ0.178-0.110=0.068
min1.498-1.575=-0.076
max1.929-1.849=0.080

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean453.537-450.185=3.351
σ585.531-583.337=2.194
min160.767-158.514=2.253
max1331.833-1325.191=6.643
mean102.250-96.250=6.000
σ8.500
min98.000-92.000=6.000
max115.000-109.000=6.000
mean440.000-1760.000=-1320.000
σ0.000
min440.000-1760.000=-1320.000
max440.000-1760.000=-1320.000
mean0.522-0.440=0.082
σ0.495-0.320=0.176
min0.273-0.279=-0.005
max1.265-0.919=0.346
policies=20
mean1608.035-1585.452=22.582
σ0.425-0.420=0.005
min1607.759-1585.179=22.580
max1608.668-1586.079=22.589
mean980.000-920.000=60.000
σ0.000
min980.000-920.000=60.000
max980.000-920.000=60.000
mean447.500-1760.000=-1312.500
σ8.660-0.000=8.660
min440.000-1760.000=-1320.000
max455.000-1760.000=-1305.000
mean2.703-2.753=-0.050
σ0.012-0.011=0.001
min2.688-2.743=-0.055
max2.716-2.766=-0.050
policies=200
mean16088.589-15862.293=226.297
σ0.134-0.146=-0.013
min16088.496-15862.163=226.333
max16088.787-15862.444=226.343
mean9800.000-9200.000=600.000
σ0.000
min9800.000-9200.000=600.000
max9800.000-9200.000=600.000
mean496.250-1760.000=-1263.750
σ5.377-0.000=5.377
min490.000-1760.000=-1270.000
max503.000-1760.000=-1257.000
mean27.274-27.026=0.248
σ0.185-0.576=-0.391
min27.106-26.671=0.434
max27.536-27.888=-0.351