Compare commits

Home / Compare bfc65261 .. 0c33d39b

grey: no significant change wrt stddev
red: change exceeding max stddev
purple: some change, not seemingly significant

AST compact size (buildDeclTree)

Factorcompact size MiB
phase=buildDeclTree
mean0.203
σ0.000
min0.203
max0.203

AST compact size (macro expansion)

Factorcompact size MiB
phase=macro expansion
mean5.082
σ0.000
min5.082
max5.082

AST compact size (parser)

Factorcompact size MiB
phase=parser
mean4.988
σ0.000
min4.988
max4.988

AST compact size (processProdDef Decls)

Factorcompact size MiB
phase=processProdDef Decls
mean4.863-5.395=-0.531
σ0.000
min4.863-5.395=-0.531
max4.863-5.395=-0.531

AST compact size (processProdDef)

Factorcompact size MiB
phase=processProdDef
mean446.102-446.633=-0.531
σ0.000
min446.102-446.633=-0.531
max446.102-446.633=-0.531

AST compact size (renaming)

Factorcompact size MiB
phase=renaming
mean5.238
σ0.000
min5.238
max5.238

AST compact size (type checker)

Factorcompact size MiB
phase=type checker
mean6.395
σ0.000
min6.395
max6.395

Aggregation: measureAggregations

Factorallocated MiBpeak allocated MiBtime_s
cache=n
dps=1
iters=10
policies=100
mean54.697-54.718=-0.021
σ0.362-0.363=-0.000
min54.558-54.574=-0.017
max55.728-55.750=-0.022
mean416.900-414.000=2.900
σ0.316-0.000=0.316
min416.000-414.000=2.000
max417.000-414.000=3.000
mean0.139-0.140=-0.001
σ0.005-0.004=0.000
min0.133-0.135=-0.002
max0.147-0.147=-0.000
cache=y
dps=1
iters=10
policies=100
mean43.782-43.777=0.005
σ0.006-0.008=-0.002
min43.774-43.764=0.010
max43.795-43.788=0.007
mean417.000-414.000=3.000
σ0.000
min417.000-414.000=3.000
max417.000-414.000=3.000
mean0.142-0.122=0.021
σ0.005-0.003=0.002
min0.131-0.119=0.012
max0.153-0.129=0.024

CSV - indexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on indexed
mean36.605-36.604=0.001
σ0.000-0.000=0.000
min36.605-36.604=0.001
max36.606-36.605=0.001
mean4224.125-4433.625=-209.500
σ1971.011-2227.963=-256.951
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean12.296-13.471=-1.175
σ0.401-3.713=-3.312
min12.033-11.869=0.164
max13.235-22.582=-9.347
op=column_set() on indexed, repeat
mean36.605-36.605=0.001
σ0.000-0.000=0.000
min36.605-36.604=0.001
max36.606-36.605=0.001
mean4450.250-4612.500=-162.250
σ2240.781-2259.436=-18.655
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean140.186-170.741=-30.555
σ360.364-443.250=-82.885
min12.622-11.457=1.165
max1032.041-1267.700=-235.658
op=table_records(), no indexes
mean41.940-41.939=0.000
σ0.002-0.002=0.000
min41.938-41.938=0.000
max41.943-41.942=0.001
mean7440.000-7411.000=29.000
σ0.000
min7440.000-7411.000=29.000
max7440.000-7411.000=29.000
mean11.056-10.978=0.078
σ0.134-0.161=-0.027
min10.864-10.715=0.149
max11.285-11.252=0.033
op=table_records(), repeat
mean41.925-41.925=0.001
σ0.001-0.000=0.001
min41.924-41.924=0.000
max41.929-41.925=0.003
mean4224.125-4205.250=18.875
σ1971.011-1956.238=14.773
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean11.644-11.557=0.087
σ1.317-1.348=-0.030
min10.789-10.781=0.008
max14.828-14.853=-0.025
op=vertical_lookup_list() on indexed, after table_records()
mean27.634-27.633=0.001
σ0.137-0.136=0.000
min27.585-27.584=0.001
max27.972-27.971=0.001
mean4224.125-4205.250=18.875
σ1971.011-1956.238=14.773
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean9.015-8.672=0.343
σ1.294-0.595=0.699
min8.056-8.247=-0.191
max12.076-10.084=1.992
op=vertical_lookup_list() on indexed, after table_records(), repeat
mean26.385-26.384=0.001
σ0.000-0.000=-0.000
min26.385-26.384=0.001
max26.385-26.384=0.001
mean4224.125-4205.250=18.875
σ1971.011-1956.238=14.773
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean6.935-6.811=0.124
σ1.069-0.438=0.631
min6.028-6.526=-0.498
max9.500-7.854=1.646
op=vertical_lookup_row_list() on 2 indexed
mean25.663-25.662=0.001
σ0.004-0.004=0.000
min25.662-25.661=0.001
max25.674-25.673=0.001
mean4450.250-4612.500=-162.250
σ2240.781-2259.436=-18.655
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean6.298-6.562=-0.264
σ0.636-1.038=-0.402
min5.781-5.837=-0.056
max7.698-8.742=-1.045
op=vertical_lookup_row_list() on 3 (1 unindexed)
mean25.962-25.961=0.002
σ0.001-0.001=-0.000
min25.962-25.961=0.002
max25.964-25.962=0.001
mean4450.250-4612.500=-162.250
σ2240.781-2259.436=-18.655
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean5.987-6.476=-0.488
σ0.510-1.033=-0.523
min5.593-5.768=-0.175
max7.204-8.648=-1.444

CSV - loading a lazy table

Factorallocated MiBpeak allocated MiBtime (ms)
op=load_product(), no indexes
mean1021.599-1021.608=-0.009
σ0.002-0.002=-0.000
min1021.597-1021.606=-0.009
max1021.602-1021.611=-0.009
mean7440.000-7411.000=29.000
σ0.000
min7440.000-7411.000=29.000
max7440.000-7411.000=29.000
mean350.182-344.629=5.554
σ2.832-2.737=0.095
min346.961-341.147=5.815
max355.885-349.284=6.601
op=load_product(), with indexes
mean3495.846-3495.854=-0.008
σ0.455-0.454=0.001
min3495.685-3495.694=-0.009
max3496.971-3496.977=-0.006
mean3794.250-3776.125=18.125
σ2329.428-2313.289=16.139
min721.000-720.000=1.000
max7440.000-7411.000=29.000
mean2074.253-1711.212=363.041
σ983.512-532.300=451.212
min1390.213-1392.306=-2.093
max3887.437-2735.992=1151.445

CSV - unindexed operations

Factorallocated MiBpeak allocated MiBtime (ms)
op=column_set() on unindexed, dense, minor warmup
mean107.841-107.841=0.001
σ0.000-0.000=0.000
min107.841-107.840=0.001
max107.842-107.841=0.001
mean4224.125-4205.250=18.875
σ1971.011-1956.238=14.773
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean133.361-131.435=1.925
σ12.969-6.371=6.599
min124.166-126.238=-2.071
max163.772-141.087=22.685
op=column_set() on unindexed, dense, repeat
mean62.069-62.068=0.000
σ0.001-0.001=-0.000
min62.065-62.065=0.001
max62.069-62.069=-0.000
mean4224.125-4433.625=-209.500
σ1971.011-2227.963=-256.951
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean42.694-143.906=-101.211
σ1.230-286.993=-285.763
min41.012-40.819=0.193
max44.248-854.171=-809.924
op=vertical_lookup_list() repeated on unindexed missing Int
mean40.927-40.926=0.001
σ0.002-0.002=-0.000
min40.926-40.925=0.001
max40.930-40.929=0.000
mean7440.000-7411.000=29.000
σ0.000
min7440.000-7411.000=29.000
max7440.000-7411.000=29.000
mean22.281-119.577=-97.297
σ1.225-275.996=-274.770
min21.452-21.760=-0.308
max25.079-802.631=-777.552
op=vertical_lookup_row_list() on unindexed small unique Int
mean71.294-71.294=0.001
σ0.002-0.002=-0.000
min71.293-71.292=0.001
max71.297-71.296=0.001
mean7440.000-7411.000=29.000
σ0.000
min7440.000-7411.000=29.000
max7440.000-7411.000=29.000
mean105.392-58.326=47.066
σ140.721-8.367=132.354
min54.895-54.787=0.108
max453.655-79.002=374.653

CSV - warmup

Factorallocated MiBpeak allocated MiBtime (ms)
op=table_records()
mean4646.842-4646.842=0.000
σ0.013-0.013=-0.000
min4646.837-4646.837=0.000
max4646.875-4646.875=0.000
mean4224.125-4205.250=18.875
σ1971.011-1956.238=14.773
min1482.000-1481.000=1.000
max7440.000-7411.000=29.000
mean3433.684-3349.914=83.770
σ615.672-815.206=-199.534
min2513.022-2485.074=27.948
max4190.805-4720.669=-529.864
op=vertical_lookup_list() on indexed
mean1973.830-1973.829=0.001
σ0.002-0.002=-0.000
min1973.829-1973.827=0.001
max1973.833-1973.831=0.001
mean7440.000-7411.000=29.000
σ0.000
min7440.000-7411.000=29.000
max7440.000-7411.000=29.000
mean1243.099-947.686=295.413
σ855.182-73.599=781.583
min905.994-902.274=3.720
max3355.814-1092.746=2263.068
op=vertical_lookup_list() on unindexed missing Int, minor warmup
mean905.084-905.083=0.001
σ0.002
min905.082-905.080=0.001
max905.087-905.086=0.001
mean7440.000-7411.000=29.000
σ0.000
min7440.000-7411.000=29.000
max7440.000-7411.000=29.000
mean409.418-388.543=20.875
σ42.381-6.075=36.306
min391.235-374.203=17.032
max514.156-393.561=120.595

CSV internal benchmark: pointedIndexFromKvs

Factorallocated MiBpeak allocated MiBtime (ms)
op=pointedIndexFromKvs
rows=200000
mean148.144
σ0.001
min148.144
max148.148
mean476.000-477.000=-1.000
σ0.000-0.926=-0.926
min476.000
max476.000-478.000=-2.000
mean354.991-389.258=-34.268
σ18.661-13.533=5.129
min328.502-371.977=-43.475
max382.274-407.686=-25.411
op=pointedIndexFromKvs
rows=400000
mean308.493
σ0.001
min308.493
max308.497
mean562.875-565.000=-2.125
σ0.354-0.000=0.354
min562.000-565.000=-3.000
max563.000-565.000=-2.000
mean730.687-774.213=-43.526
σ53.828-48.137=5.691
min618.838-679.649=-60.811
max802.723-848.256=-45.534

Marine Hull: transactPolicy

Factorallocated MiBpeak allocated MiBtime
datapoints=1
iterations=20
mean964.558-964.531=0.027
σ1.880-1.820=0.059
min963.837-963.826=0.011
max972.509-972.228=0.281
mean1056.550-1038.400=18.150
σ308.669-307.991=0.678
min558.000-542.000=16.000
max1512.000-1535.000=-23.000
mean1.288-1.285=0.002
σ0.153-0.167=-0.014
min1.184-1.176=0.007
max1.797-1.899=-0.102

Parser: Comma Sep

Factorallocated MiBpeak allocated MiBtime (ms)
name=ambiguous_item.art
mean0.913
σ0.020
min0.903
max0.949
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.392-0.376=0.016
σ0.503-0.439=0.064
min0.136-0.147=-0.011
max1.289-1.158=0.131
name=error_case.art
mean2.346-2.346=-0.000
σ0.012
min2.340-2.340=-0.000
max2.368-2.368=-0.000
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.477-0.430=0.047
σ0.322-0.197=0.126
min0.323-0.330=-0.007
max1.054-0.782=0.272
name=heavy_item.art
mean1.523
σ0.002
min1.522
max1.526
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.220-0.221=-0.001
σ0.020-0.027=-0.007
min0.193-0.201=-0.007
max0.250-0.268=-0.018
name=nested_use.art
mean2.364-2.364=-0.000
σ0.002
min2.363-2.363=-0.000
max2.367-2.367=-0.000
mean402.000-401.000=1.000
σ0.000
min402.000-401.000=1.000
max402.000-401.000=1.000
mean0.352-0.354=-0.002
σ0.013-0.071=-0.057
min0.338-0.312=0.025
max0.368-0.480=-0.111

Parser: Real World

Factorallocated MiBpeak allocated MiBtime (sec)
name=aon_gpf25_7k_loc.art
mean964.978-964.979=-0.001
σ0.009
min964.973-964.974=-0.001
max964.995-964.996=-0.001
mean435.800-430.400=5.400
σ6.017-0.894=5.122
min430.000
max442.000-432.000=10.000
mean0.646-0.587=0.059
σ0.054-0.033=0.021
min0.592-0.550=0.042
max0.721-0.615=0.107
name=cdr_enhanced_4k_loc.art
mean886.000-886.001=-0.001
σ0.002
min885.999-886.000=-0.001
max886.003-886.004=-0.001
mean442.000-439.000=3.000
σ0.000
min442.000-439.000=3.000
max442.000-439.000=3.000
mean0.622-0.437=0.184
σ0.019-0.054=-0.035
min0.598-0.381=0.217
max0.642-0.521=0.121

Parser: Record Parsing

Factorallocated MiBpeak allocated MiBtime (ms)
name=record_render_large_success.art
mean7.719
σ0.002
min7.718
max7.722
mean442.000-439.000=3.000
σ0.000
min442.000-439.000=3.000
max442.000-439.000=3.000
mean3.601-2.723=0.878
σ0.095-0.133=-0.038
min3.434-2.044=1.390
max3.982-3.069=0.914
name=record_render_large_syntax_err.art
mean7.667
σ0.002
min7.666
max7.670
mean442.000-439.000=3.000
σ0.000
min442.000-439.000=3.000
max442.000-439.000=3.000
mean3.573-2.630=0.943
σ0.082-0.114=-0.032
min3.430-2.472=0.959
max3.813-3.250=0.563
name=record_render_success.art
mean2.290
σ0.002
min2.290
max2.294
mean442.000-439.000=3.000
σ0.000
min442.000-439.000=3.000
max442.000-439.000=3.000
mean0.542-0.307=0.235
σ0.016-0.025=-0.009
min0.516-0.287=0.229
max0.641-0.533=0.108
name=record_render_syntax_err.art
mean2.166
σ0.002
min2.165
max2.169
mean442.000-439.000=3.000
σ0.000
min442.000-439.000=3.000
max442.000-439.000=3.000
mean0.520-0.294=0.226
σ0.012-0.012=0.000
min0.499-0.278=0.221
max0.593-0.345=0.248

Revision Creation

Factorallocated MiBpeak allocated MiBtime_s
number of threads=8
revision count=1000
mean11.122-12.022=-0.900
σ11.054-11.863=-0.809
min0.573-0.574=-0.000
max93.458-116.997=-23.538
mean462.096-460.722=1.374
σ18.068-20.209=-2.141
min407.000
max474.000-479.000=-5.000
mean0.026-0.027=-0.001
σ0.027-0.029=-0.002
min0.001-0.001=-0.000
max0.228-0.286=-0.057

SoV import

Factorallocated MiBpeak allocated MiBtime_s
iters=10
rows=1000
mean279.115-278.924=0.191
σ1.061-0.979=0.081
min278.668-278.414=0.254
max282.126-281.690=0.436
mean441.000-446.900=-5.900
σ15.839-20.469=-4.630
min417.000
max469.000-481.000=-12.000
mean0.574-0.588=-0.014
σ0.017-0.013=0.004
min0.550-0.577=-0.027
max0.609-0.615=-0.006

Timed compile per phase

Factorallocated MiBpeak allocated MiBtime (ms)
phase=buildDeclTree
mean106.546-169.257=-62.711
σ0.000-0.000=0.000
min106.546-169.257=-62.711
max106.546-169.257=-62.711
mean822.625-872.000=-49.375
σ49.627-0.000=49.627
min727.000-872.000=-145.000
max849.000-872.000=-23.000
mean147.454-203.481=-56.027
σ8.855-14.752=-5.898
min136.383-191.898=-55.515
max159.536-238.348=-78.812
phase=macro expansion
mean5.894-5.899=-0.005
σ0.001-0.000=0.001
min5.894-5.899=-0.006
max5.897-5.899=-0.002
mean809.250-872.000=-62.750
σ58.717-0.000=58.717
min702.000-872.000=-170.000
max849.000-872.000=-23.000
mean8.114-7.504=0.610
σ1.272-1.748=-0.476
min6.007-5.794=0.213
max9.189-9.381=-0.193
phase=parser
mean1042.132-1042.131=0.001
σ0.000-0.000=0.000
min1042.132-1042.131=0.001
max1042.133-1042.131=0.001
mean809.250-872.000=-62.750
σ58.717-0.000=58.717
min702.000-872.000=-170.000
max849.000-872.000=-23.000
mean543.920-480.352=63.568
σ85.614-113.134=-27.520
min438.604-372.916=65.689
max619.291-594.489=24.802
phase=processProdDef
mean2241.192-2243.719=-2.527
σ0.001-0.000=0.001
min2241.192-2243.719=-2.527
max2241.194-2243.719=-2.525
mean822.625-872.000=-49.375
σ49.627-0.000=49.627
min727.000-872.000=-145.000
max849.000-872.000=-23.000
mean2832.647-2675.743=156.903
σ271.527-458.047=-186.520
min2484.471-1966.708=517.763
max3215.030-3317.970=-102.941
phase=renaming
mean21.430-21.436=-0.007
σ0.000
min21.430-21.436=-0.007
max21.430-21.436=-0.007
mean809.250-872.000=-62.750
σ58.717-0.000=58.717
min702.000-872.000=-170.000
max849.000-872.000=-23.000
mean39.555-34.807=4.748
σ8.904-6.501=2.404
min19.814-24.490=-4.676
max47.772-42.145=5.626
phase=type checker
mean958.466-958.596=-0.130
σ0.047-0.047=0.000
min958.439-958.573=-0.134
max958.568-958.698=-0.130
mean809.250-872.000=-62.750
σ58.717-0.000=58.717
min702.000-872.000=-170.000
max849.000-872.000=-23.000
mean658.988-663.722=-4.733
σ15.184-39.957=-24.773
min633.058-624.801=8.257
max673.435-730.320=-56.885

Workflow: transactPolicy

Factorallocated MiBpeak allocated MiBtime
iters=5
new-facts=1
prior-facts=1
mean282.856-282.360=0.496
σ0.035-0.035=0.000
min282.804-282.309=0.495
max282.890-282.397=0.493
mean444.800-439.000=5.800
σ1.095-2.739=-1.643
min444.000-437.000=7.000
max446.000-442.000=4.000
mean0.383-0.380=0.003
σ0.009-0.006=0.004
min0.373-0.375=-0.001
max0.397-0.388=0.010
iters=5
new-facts=1
prior-facts=6402
mean1268.047-1267.610=0.437
σ0.113-0.108=0.005
min1267.948-1267.462=0.486
max1268.235-1267.766=0.469
mean1094.000-1086.800=7.200
σ169.951-164.544=5.408
min892.000-889.000=3.000
max1326.000-1309.000=17.000
mean2.004-2.149=-0.145
σ0.096-0.234=-0.138
min1.886-1.875=0.011
max2.108-2.523=-0.415
iters=5
new-facts=6402
prior-facts=1
mean1507.022-1506.521=0.502
σ2.957-2.928=0.029
min1505.478-1505.016=0.462
max1512.299-1511.746=0.554
mean626.000-623.200=2.800
σ87.164-87.311=-0.147
min496.000-493.000=3.000
max728.000-725.000=3.000
mean2.198-2.164=0.034
σ0.110-0.083=0.028
min2.118-2.100=0.018
max2.392-2.304=0.087
iters=5
new-facts=6402
prior-facts=6402
mean1830.056-1829.574=0.482
σ0.048-0.040=0.008
min1829.984-1829.515=0.468
max1830.109-1829.613=0.495
mean1722.200-1726.800=-4.600
σ191.061-173.092=17.969
min1458.000-1478.000=-20.000
max1896.000-1887.000=9.000
mean3.025-3.023=0.002
σ0.344-0.258=0.086
min2.814-2.867=-0.054
max3.638-3.479=0.158

newProductH/cdr_auton/broker

Factortime_s
iterations=5
mean1.773-1.682=0.090
σ0.091-0.215=-0.124
min1.623-1.325=0.298
max1.862-1.866=-0.003

policy addition benchmark

Factorallocated MiBdb queriespeak allocated MiBtime
policies=2
mean405.942-393.867=12.076
σ509.285-509.607=-0.322
min151.295-139.060=12.235
max1169.870-1158.277=11.593
mean101.750-99.750=2.000
σ7.500
min98.000-96.000=2.000
max113.000-111.000=2.000
mean444.000-440.000=4.000
σ0.000
min444.000-440.000=4.000
max444.000-440.000=4.000
mean0.498-0.490=0.008
σ0.482-0.498=-0.016
min0.254-0.231=0.023
max1.221-1.238=-0.017
policies=20
mean1513.143-1390.723=122.420
σ0.274-0.165=0.109
min1512.962-1390.601=122.361
max1513.548-1390.965=122.583
mean980.000-960.000=20.000
σ0.000
min980.000-960.000=20.000
max980.000-960.000=20.000
mean450.500-445.750=4.750
σ10.376-11.500=-1.124
min444.000-440.000=4.000
max466.000-463.000=3.000
mean2.498-2.309=0.189
σ0.010-0.020=-0.010
min2.485-2.290=0.195
max2.510-2.329=0.181
policies=200
mean15140.022-13915.906=1224.116
σ0.122-0.105=0.017
min15139.867-13915.806=1224.060
max15140.161-13916.049=1224.113
mean9800.000-9600.000=200.000
σ0.000
min9800.000-9600.000=200.000
max9800.000-9600.000=200.000
mean495.250-494.750=0.500
σ4.272-4.113=0.159
min490.000-491.000=-1.000
max500.000
mean25.160-23.380=1.780
σ0.093-0.150=-0.057
min25.091-23.251=1.840
max25.289-23.578=1.711