Param init delay #344

mb706 · 2021-03-22T21:31:47Z

Delay initing / cloning of Params to when it is necessary by:

making Params immutable, except for the id. If you want a different Param you have to create a new one now. This should not be a problem with to_tune() etc. I have never had the need to modify a Param.
using the $params_unid slot, which returns Params with the $id set to the wrong value (the names of the list identify the actual id), which avoids cloning. With this is it possible to have deep hierarchies of ParamSetCollections that just return their leaf Params without having to clone at all.
Cloning Param on-demand whenever the user accesses $params. This basically never happens, except when the user himself actually accesses $params, since $params_unid carries the same information.

This works with:

mb706 · 2021-03-22T21:38:07Z

Benchmark

In Brief

creating a Param is 300 us 4x faster in param6 (400 us vs. 100 us).
--> comparison: the assert_id() call is ~ 40 us
--> ... which param6::prm() doesn't do. This is to some degree because
there are fewer asserts in param6?
creating a ParamSet is faster in param6 (940 us vs. 400 us)
creating a ParamSet with values: 1300 us vs. 580 us
getting values: 5.7 us vs. 4.7 us (almost the same)
setting values: 600 us vs. 3000 us (paradox is faster!)
- in an earlier version this was apparently ~ 300 us in param6
ParamSetCollection: 740 vs. 830 us (faster in paradox)
- but: with many 100 params, it is 3600 us vs. 900 us
- with 10,000 params it is 210 ms vs. 20 ms (milliseconds)
  --> comparison: map(<10,000 params>, "id") (for duplicate checks)
  takes ~50 ms, so param6 is obviously not counting some checks, again.
ParamSetCollection created inside
- Creating 100 params: ~ 100ms --> 1 ms / param
- 10,000 params using paradox "rep" is fastest: 17ms, followed by
  param6 (21 ms) followed by paradox without "rep" (60 ms)

Take Home Message: Most of this is because of assert()s when creating
objects, whenever we avoid creating objects (e.g. in rep()) we are pretty fast.

Benchmark

# remotes::install_github("xoopR/set6")
# remotes::install_github("xoopR/param6@patch_clone_bottleneck")
# remotes::install_github("alan-turing-institute/distr6")
# remotes::install_github("mlr-org/paradox")

library("microbenchmark")

Param Construction Times

microbenchmark(
  paradox = paradox::ParamDbl$new("a"),
  param6 = param6::prm("a", "reals"),
  assert = assert_id("test")
)

######### RaphaelS1's Result:
## Unit: microseconds
##     expr     min      lq      mean   median       uq       max neval cld
##  paradox 530.340 581.893  2856.005 685.5035 956.8250  188503.6   100   a
##   param6 106.951 120.406 17325.349 146.1760 183.9735 1647753.6   100   a
######### mb706's Result:
#> Unit: microseconds
#>     expr      min        lq     mean    median        uq       max neval cld
#>  paradox  438.192  490.6795 1444.977  557.7975  666.4945  66786.82   100   a
#>   param6 1908.516 2066.6475 9026.828 2282.0690 2819.9355 527334.77   100   a
######### mb706's Result (param6 cloning bottleneck fixed):
#> Unit: microseconds
#>     expr     min       lq     mean   median      uq     max neval cld
#>  paradox 340.791 353.7955 399.5291 375.3185 424.028 691.004   100   b
#>   param6  81.777  88.8675 108.1652  99.6890 110.153 268.745   100  a
######### mb706's Result (paradox improvements):
#> Unit: microseconds
#>     expr     min      lq     mean  median       uq     max neval cld
#>  paradox 355.448 370.946 443.6327 389.491 451.4765 983.736   100   b
#>   param6  81.625  92.174 116.5401 103.954 119.9960 257.124   100  a

ParamSet Construction Times (without values)

microbenchmark(
  paradox = {
    paradox::ParamSet$new(list(
      paradox::ParamDbl$new("a")
    ))
  },
  distr6 = {
    distr6::ParameterSet$new(
      id = "a",
      support = set6::Reals$new(),
      value = 1
    )
  },
  param6 = {
    param6::ParameterSet$new(list(
      param6::prm("a", "reals")
    ))
  }
)

######### RaphaelS1's Result:
## Unit: microseconds
##     expr      min        lq      mean    median        uq       max neval cld
##  paradox 1609.038 2174.3760 3513.4166 2781.0720 3747.9320 15659.522   100  b
##   distr6 2472.554 3672.8845 6497.1024 5364.0035 7760.7680 49922.386   100   c
##   param6  363.630  522.1065  909.8822  605.4135  836.1085  6982.207   100 a
######### mb706's Result:
#> Unit: milliseconds
#>     expr      min       lq     mean   median       uq       max neval cld
#>  paradox 1.360965 1.525759 1.884099 1.692618 2.132996  3.401718   100 a
#>   distr6 1.906816 2.175472 2.614912 2.337153 2.773957 11.790750   100  b
#>   param6 2.200960 2.413296 3.004322 2.775427 3.265760  5.683351   100   c
######### mb706's Result (param6 cloning bottleneck fixed):
#> Unit: microseconds
#>     expr      min        lq      mean   median       uq      max neval cld
#>  paradox 1124.088 1768.3155 2239.4555 2367.077 2530.478 3890.839   100  b
#>   distr6 1675.121 2868.2395 3272.4636 3388.221 3550.744 9796.993   100   c
#>   param6  275.365  542.4905  637.5095  642.470  702.424 2973.078   100 a
######### mb706's Result (paradox improvements):
#> Unit: microseconds
#>     expr      min        lq      mean    median        uq      max neval cld
#>  paradox  658.664  755.9615 1079.2900  939.8250 1286.6300 4003.110   100  b
#>   distr6 1681.721 1877.7820 2546.9160 2282.5720 3135.4515 4440.512   100   c
#>   param6  269.525  351.2735  465.2623  408.7925  590.2955  905.533   100 a

ParamSet Construction Times (with values)

microbenchmark(
  paradox = {
    ps <- paradox::ParamSet$new(list(
      paradox::ParamDbl$new("a")
    ))
    ps$values$a <- 1
  },
  distr6 = {
    distr6::ParameterSet$new(
      id = "a",
      support = set6::Reals$new(),
      value = 12
    )
  },
  param6 = {
    param6::ParameterSet$new(list(
      param6::prm("a", "reals", 1)
    ))
  }
)

######### RaphaelS1's Result:
## Unit: microseconds
##     expr      min       lq      mean   median        uq       max neval cld
##  paradox 6105.521 7105.520 9939.8702 8267.198 10744.329 38730.537   100   c
##   distr6 2248.061 2618.910 4086.1559 2935.913  3888.729 21699.133   100  b
##   param6  454.811  544.949  888.8888  633.952   863.704  5595.308   100 a
######### mb706's Result:
#> Unit: milliseconds
#>     expr      min       lq     mean   median       uq       max neval cld
#>  paradox 1.756259 1.982387 2.609187 2.163434 2.617840 10.470030   100  a
#>   distr6 2.023483 2.247876 2.982715 2.573572 3.046401  7.769178   100  a
#>   param6 2.388264 2.624278 3.593221 2.940272 3.587762 11.291065   100   b
######### mb706's Result (param6 cloning bottleneck fixed):
#> Unit: microseconds
#>     expr      min       lq      mean    median       uq       max neval cld
#>  paradox 1486.486 1691.700 2397.5187 2092.3210 2892.519  5290.279   100   b
#>   distr6 1590.974 1900.193 2719.2516 2438.5905 3445.730  4847.604   100   b
#>   param6  400.325  530.114  999.9538  767.4315  990.963 23347.918   100  a
######### mb706's Result (paradox improvements):
#> Unit: microseconds
#>     expr      min        lq      mean    median        uq      max neval cld
#>  paradox  913.177 1104.6155 1420.5150 1313.9325 1490.4535 6054.456   100  b
#>   distr6 1534.665 1925.6355 2406.4413 2274.7340 2767.4710 4200.716   100   c
#>   param6  392.560  494.7755  618.1847  576.1005  707.8275 1086.220   100 a

Getting Values

paradox <- paradox::ParamSet$new(list(
  paradox::ParamDbl$new("a")
))
paradox$values$a <- 1

param6 <- param6::ParameterSet$new(list(
  param6::prm("a", "reals", 1)
))

distr6 <- distr6::ParameterSet$new(
  id = "a",
  support = set6::Reals$new(),
  value = 1
)

microbenchmark(
  paradox = paradox$values$a,
  distr6 = distr6$getParameterValue("a"),
  param6 = param6$values$a
)


######### RaphaelS1's Result:
## Unit: microseconds
##     expr     min       lq      mean   median      uq      max neval cld
##  paradox   2.259   2.7495   5.81384   4.9830   7.493   22.859   100  a
##   distr6 224.898 228.6225 313.32112 255.9815 300.272 1346.424   100   b
##   param6   2.167   2.5280   4.80121   3.5695   5.660   25.868   100  a
######### mb706's Result:
#> Unit: microseconds
#>     expr     min       lq      mean   median       uq       max neval cld
#>  paradox   5.166   5.9215   9.82681   8.9180  12.8975    20.962   100  a
#>   distr6 356.694 425.6670 618.31735 468.5095 518.0945 13602.424   100   b
#>   param6   4.795   5.8870  10.73301   7.6820  11.8990   132.937   100  a
######### mb706's Result (param6 cloning bottleneck fixed):
#> Unit: microseconds
#>     expr     min       lq      mean   median       uq      max neval cld
#>  paradox   1.901   2.1290   3.27443   2.6350   4.2250   11.791   100  a
#>   distr6 148.496 151.4365 226.65299 157.5925 173.4185 5941.647   100   b
#>   param6   1.785   2.0095   4.29681   2.6600   3.8740  106.610   100  a
######### mb706's Result (paradox improvements):
#> Unit: microseconds
#>     expr     min       lq      mean   median       uq      max neval cld
#>  paradox   2.085   4.5475   7.77506   5.6810  10.5000   44.280   100  a
#>   distr6 166.726 257.5370 411.78218 331.8775 383.1775 9405.069   100   b
#>   param6   1.958   3.6720   7.41901   4.7295   9.0050  120.004   100  a

Setting Values

vals = list(a = 2)
microbenchmark(
  paradox = {paradox$values = vals},
  distr6 = distr6$setParameterValue(lst = vals),
  param6 = {param6$values = vals}
)

######### RaphaelS1's Result:
## Unit: microseconds
##     expr      min        lq      mean   median        uq       max neval cld
##  paradox 4443.723 6170.4430 8538.9734 7403.095 9851.6190 24445.143   100   c
##   distr6 1748.316 2178.0500 4642.6509 2783.577 3716.7730 89812.954   100  b
##   param6  219.380  316.5885  628.3729  385.912  544.5435  6749.543   100 a
######### mb706's Result:
#> Unit: microseconds
#>     expr      min       lq      mean    median       uq        max neval cld
#>  paradox  424.410  555.329  828.7148  828.2355 1005.120   1566.704   100  a
#>   distr6 1571.383 1904.153 4008.1785 2545.6550 3280.782 101720.834   100   b
#>   param6 2074.857 2358.704 3635.8081 3028.2580 4642.464   9557.661   100   b
######### mb706's Result (param6 cloning bottleneck fixed):
#> Unit: microseconds
#>     expr      min       lq      mean    median        uq        max neval cld
#>  paradox  348.559  440.998  534.2739  475.7205  590.2865   1053.639   100  a
#>   distr6 1211.964 1403.457 2463.0274 1547.0895 1998.2290  68193.272   100  ab
#>   param6 1707.462 1861.377 3691.7001 1989.9970 2373.3100 106668.590   100   b
######### mb706's Result (paradox improvements):
#> Unit: microseconds
#>     expr      min       lq      mean   median        uq        max neval cld
#>  paradox  336.213  461.967  628.6033  633.278  796.0075   1154.121   100  a
#>   distr6 1215.674 1690.755 2868.7096 2394.423 2763.4710  56608.896   100  ab
#>   param6 1783.361 2168.640 4773.8808 3330.361 4040.9595 119841.829   100   b

ParamSetCollection

paradox1 <- paradox::ParamSet$new(list(
  paradox::ParamDbl$new("a")
))
paradox1$values$a <- 1
paradox2 <- paradox::ParamSet$new(list(
  paradox::ParamDbl$new("b")
))
paradox2$values$b <- 2

param61 <- param6::ParameterSet$new(list(
  param6::prm("a", "reals", 1)
))
param62 <- param6::ParameterSet$new(list(
  param6::prm("b", "reals", 2)
))

distr61 <- distr6::ParameterSet$new(
  id = "a",
  support = set6::Reals$new(),
  value = 1
)
distr62 <- distr6::ParameterSet$new(
  id = "a",
  support = set6::Reals$new(),
  value = 2
)

microbenchmark(
  paradox = paradox::ParamSetCollection$new(list(paradox1, paradox2)),
  distr6 = distr6::ParameterSetCollection$new(D1 = distr61, D2 = distr62),
  param6 = param6:::c.ParameterSet(param61, param62)
)

######### RaphaelS1's Result:
## Unit: microseconds
##     expr      min        lq      mean   median        uq        max neval cld
##  paradox 3434.875 4233.1565 7824.6972 5396.075 7550.0195 156833.891   100   b
##   distr6  327.489  417.6195  651.8163  462.862  584.7045   4962.294   100  a
##   param6  775.039  987.4275 1458.6529 1246.534 1584.7415   6533.939   100  a
######### mb706's Result:
#> Unit: microseconds
#>     expr      min        lq      mean   median        uq       max neval cld
#>  paradox  712.726  848.8335 1139.0358  965.889 1506.3260  2329.830   100  b
#>   distr6  302.141  354.6790  501.7752  395.453  637.0355  3105.607   100 a
#>   param6 4286.703 5406.6935 6987.7337 6099.883 8033.9120 14304.924   100   c
######### mb706's Result (param6 cloning bottleneck fixed):
#> Unit: microseconds
#>     expr     min       lq     mean   median      uq      max neval cld
#>  paradox 547.649 590.4850 788.5231 680.7035 956.807 1949.165   100   b
#>   distr6 224.875 245.9060 369.4724 284.9050 445.688 3654.080   100  a
#>   param6 563.480 618.8115 824.1178 701.3475 970.837 1851.873   100   b
######### mb706's Result (paradox improvements):
#> Unit: microseconds
#>     expr     min       lq     mean   median        uq      max neval cld
#>  paradox 489.719 598.3375 776.6420 739.9660  936.7075 1585.274   100  b
#>   distr6 231.262 279.9295 373.2453 328.8425  461.7670  682.622   100 a
#>   param6 556.984 652.1585 897.1888 824.7545 1067.8620 1628.438   100   c

ParamSetCollection (Many Params)

paradox <- lapply(lapply(1:100, function(i) paradox::ParamDbl$new(paste0("Pre", i, "__a"))),
  function(.x) paradox::ParamSet$new(list(.x)))

distr6 <- rep(list(distr6::ParameterSet$new(
  id = "a",
  support = set6::Reals$new(),
  value = 1
)), 100)
names(distr6) <- paste0("Pre__", 1:100)

microbenchmark(
  paradox = paradox::ParamSetCollection$new(paradox),
  distr6 = distr6::ParameterSetCollection$new(lst = distr6),
  param6 = param6:::rep.ParameterSet(param6, 100, "Pre")
)

######### RaphaelS1's Result:
## Unit: microseconds
##     expr        min          lq        mean      median         uq         max
##  paradox 158895.858 200484.9905 300432.9309 234279.4435 328672.810 1695932.106
##   distr6    449.279    548.6530    859.9568    661.9135    943.319    3711.874
##   param6    684.358    807.7755   1415.5362    904.0350   1108.981   10954.299
##  neval cld
##    100   b
##    100  a
##    100  a
######### mb706's Result:
#> Unit: microseconds
#>     expr      min        lq      mean   median       uq       max neval cld
#>  paradox 3658.643 5170.7875 6826.5675 7295.435 8165.241 11837.219   100   c
#>   distr6  403.721  588.6340  804.9574  861.568  962.162  1311.893   100 a
#>   param6  614.247  845.8365 1189.6018 1267.081 1449.198  2135.038   100  b
######### mb706's Result (param6 cloning bottleneck fixed):
#> Unit: microseconds
#>     expr      min        lq      mean    median        uq      max neval cld
#>  paradox 2616.674 2925.4065 3845.0919 3490.7155 4883.4020 6351.811   100   b
#>   distr6  294.492  339.7640  499.5086  404.9260  541.2005 3913.798   100  a
#>   param6  458.148  550.7495  689.6945  599.8735  861.3705 1159.847   100  a
######### mb706's Result (paradox improvements):
#> Unit: microseconds
#>     expr      min       lq      mean    median        uq      max neval cld
#>  paradox 1952.610 3417.688 3613.1927 3627.9405 3900.6280 4920.488   100   c
#>   distr6  373.202  662.689  752.0552  716.4255  793.7845 3493.492   100 a
#>   param6  480.603  822.152  899.4905  888.6040  982.2020 1325.568   100  b

ParamSetCollection (Lots and Lots of Params)

paradox <- lapply(lapply(1:10000, function(i) paradox::ParamDbl$new(paste0("Pre", i, "__a"))),
  function(.x) paradox::ParamSet$new(list(.x)))

distr6 <- rep(list(distr6::ParameterSet$new(
  id = "a",
  support = set6::Reals$new(),
  value = 1
)), 10000)
names(distr6) <- paste0("Pre__", 1:10000)

microbenchmark(
  paradox = paradox::ParamSetCollection$new(paradox),
  distr6 = distr6::ParameterSetCollection$new(lst = distr6),
  param6 = param6:::rep.ParameterSet(param6, 10000, "Pre")
)

######### mb706's Result (paradox improvements):
#> Unit: milliseconds
#>     expr        min         lq       mean     median        uq       max neval
#>  paradox 192.623746 202.220676 215.592013 211.776631 224.19729 287.57988   100
#>   distr6   6.694531   7.983857   9.678699   8.696074  11.28805  19.69504   100
#>   param6  14.758645  17.316368  20.764572  19.665052  22.78877  41.82120   100
#>  cld
#>    c
#>  a
#>   b

ParamSetCollection (Many Params, created inside benchmark)

microbenchmark(
  paradox = {
    paradox <- lapply(lapply(1:100, function(i)
      paradox::ParamDbl$new(paste0("Pre", i, "__a"))), function(.x) paradox::ParamSet$new(list(.x)))
    paradox::ParamSetCollection$new(paradox)
  },
  paradox_new = {  # this is how you'd actually run it if you need to "rep" a ParamSet, for some reason
    psi = paradox::ParamSet$new(list(paradox::ParamDbl$new("a")))
    paradox::ParamSetCollection$new(sapply(paste0("Pre", 1:100), function(n) psi, simplify = FALSE), ignore_ids = TRUE)
  },
  paradox_rep = {  # This is the paradox way of rep'ing Params
    paradox::ParamDbl$new("a")$rep(100)
  },
  distr6 = {
    distr6 <- rep(list(distr6::ParameterSet$new(
      id = "a",
      support = set6::Reals$new(),
      value = 1
    )), 100)
    names(distr6) <- paste0("Pre__", 1:100)
    distr6::ParameterSetCollection$new(lst = distr6)
  },
  param6 = {
    param6 <- param6::ParameterSet$new(list(
      param6::prm("a", "reals", 1)))
    param6:::rep.ParameterSet(param6, 100, "Pre")
  }
)

######### RaphaelS1's Result:
## Unit: milliseconds
##     expr        min         lq       mean     median         uq        max
##  paradox 377.244803 522.665487 667.640479 586.522011 652.450550 2337.03302
##   distr6   2.922534   3.710481   5.965636   4.785811   6.843199   22.44207
##   param6   1.308180   1.528151   2.682907   1.974937   2.846436   11.05868
##  neval cld
##    100   b
##    100  a
##    100  a
######### mb706's Result:
#> Unit: milliseconds
#>     expr        min         lq       mean     median         uq        max
#>  paradox 151.177938 184.799960 230.768453 227.470179 252.742789 446.254719
#>   distr6   2.361071   2.804373   4.436003   4.033262   5.379300   9.415628
#>   param6   3.007534   3.441879   5.349196   5.030025   6.685412  12.083396
#>  neval cld
#>    100   b
#>    100  a
#>    100  a
######### mb706's Result (param6 cloning bottleneck fixed):
#> Unit: microseconds
#>     expr        min         lq       mean     median         uq        max
#>  paradox 121004.112 149252.471 203277.314 200651.725 237464.408 401316.830
#>   distr6   1977.049   2422.980   3604.289   3067.932   4274.595  18821.434
#>   param6    929.819   1135.461   1590.799   1344.419   1863.353   7592.413
#>  neval cld
#>    100   b
#>    100  a
#>    100  a
######### mb706's Result (paradox improvements):
#> Unit: microseconds
#>         expr       min        lq       mean     median         uq        max
#>      paradox 66095.640 84756.717 108109.297 103201.156 126516.355 233046.325
#>  paradox_new  1682.539  1934.644   2767.193   2418.868   3464.860   5160.380
#>  paradox_rep  1253.802  1630.262   2289.636   2372.430   2683.882   6198.630
#>       distr6  2010.300  2465.477   3400.764   3094.561   4087.820   8169.714
#>       param6   936.091  1239.214   1764.877   1722.437   2087.738   8256.486
#>  neval cld
#>    100   b
#>    100  a
#>    100  a
#>    100  a
#>    100  a
#>

ParamSetCollection (Lots and Lots of Params, created inside benchmark)

microbenchmark(
  paradox_new = {  # this is how you'd actually run it if you need to "rep" a ParamSet, for some reason
    psi = paradox::ParamSet$new(list(paradox::ParamDbl$new("a")))
    paradox::ParamSetCollection$new(sapply(paste0("Pre", 1:10000), function(n) psi, simplify = FALSE), ignore_ids = TRUE)
  },
  paradox_rep = {  # This is the paradox way of rep'ing Params
    paradox::ParamDbl$new("a")$rep(10000)
  },
  distr6 = {
    distr6 <- rep(list(distr6::ParameterSet$new(
      id = "a",
      support = set6::Reals$new(),
      value = 1
    )), 10000)
    names(distr6) <- paste0("Pre__", 1:10000)
    distr6::ParameterSetCollection$new(lst = distr6)
  },
  param6 = {
    param6 <- param6::ParameterSet$new(list(
      param6::prm("a", "reals", 1)))
    param6:::rep.ParameterSet(param6, 10000, "Pre")
  }
)

######### mb706's Result (paradox improvements):
#> Unit: milliseconds
#>         expr      min       lq     mean   median       uq       max neval cld
#>  paradox_new 52.69427 56.78475 65.27795 59.98950 70.63508 107.45465   100   c
#>  paradox_rep 12.50344 13.83371 16.19216 14.78004 17.10901  28.49433   100 a
#>       distr6 12.39720 13.98090 16.41835 15.37463 17.16129  34.34660   100 a
#>       param6 17.07428 19.31627 23.22281 21.38862 25.51976  50.47521   100  b

mb706 · 2021-03-23T00:08:50Z

Apparently $tags stops mlr3pipelines from working -.-

mb706 · 2021-03-23T13:21:11Z

######### RaphaelS1's Result:
## Unit: milliseconds
##     expr        min         lq       mean     median         uq        max
##  paradox 377.244803 522.665487 667.640479 586.522011 652.450550 2337.03302
##   distr6   2.922534   3.710481   5.965636   4.785811   6.843199   22.44207
##   param6   1.308180   1.528151   2.682907   1.974937   2.846436   11.05868
##  neval cld
##    100   b
##    100  a
##    100  a
######### mb706's Result:
#> Unit: milliseconds
#>     expr        min         lq       mean     median         uq        max
#>  paradox 151.177938 184.799960 230.768453 227.470179 252.742789 446.254719
#>   distr6   2.361071   2.804373   4.436003   4.033262   5.379300   9.415628
#>   param6   3.007534   3.441879   5.349196   5.030025   6.685412  12.083396
#>  neval cld
#>    100   b
#>    100  a
#>    100  a
######### mb706's Result (param6 cloning bottleneck fixed):
#> Unit: microseconds
#>     expr        min         lq       mean     median         uq        max
#>  paradox 121004.112 149252.471 203277.314 200651.725 237464.408 401316.830
#>   distr6   1977.049   2422.980   3604.289   3067.932   4274.595  18821.434
#>   param6    929.819   1135.461   1590.799   1344.419   1863.353   7592.413
#>  neval cld
#>    100   b
#>    100  a
#>    100  a
######### mb706's Result (paradox improvements):
#> Unit: microseconds
#>         expr       min        lq       mean     median         uq        max
#>      paradox 66095.640 84756.717 108109.297 103201.156 126516.355 233046.325
#>  paradox_new  1682.539  1934.644   2767.193   2418.868   3464.860   5160.380
#>  paradox_rep  1253.802  1630.262   2289.636   2372.430   2683.882   6198.630
#>       distr6  2010.300  2465.477   3400.764   3094.561   4087.820   8169.714
#>       param6   936.091  1239.214   1764.877   1722.437   2087.738   8256.486
#>  neval cld
#>    100   b
#>    100  a
#>    100  a
#>    100  a
#>    100  a
#>


##################################################
# ParamSetCollection (Lots and Lots of Params, created inside benchmark)

microbenchmark(
  paradox_new = {  # this is how you'd actually run it if you need to "rep" a ParamSet, for some reason
    psi = paradox::ParamSet$new(list(paradox::ParamDbl$new("a")))
    paradox::ParamSetCollection$new(sapply(paste0("Pre", 1:10000), function(n) psi, simplify = FALSE), ignore_ids = TRUE)
  },
  paradox_new_repaired = {  # this is how you'd actually run it if you need to "rep" a ParamSet, for some reason
    psi = paradox::ParamSet$new(list(paradox::ParamDbl$new("a")))
    paradox:::ps_union(sapply(paste0("Pre", 1:10000), function(n) psi, simplify = FALSE), ignore_ids = TRUE)
  },
  paradox_rep = {  # This is the paradox way of rep'ing Params
    paradox::ParamDbl$new("a")$rep(10000)
  },
  distr6 = {
    distr6 <- rep(list(distr6::ParameterSet$new(
      id = "a",
      support = set6::Reals$new(),
      value = 1
    )), 10000)
    names(distr6) <- paste0("Pre__", 1:10000)
    distr6::ParameterSetCollection$new(lst = distr6)
  },
  param6 = {
    param6 <- param6::ParameterSet$new(list(
      param6::prm("a", "reals", 1)))
    param6:::rep.ParameterSet(param6, 10000, "Pre")
  }
)

######### mb706's Result (paradox improvements):
#> Unit: milliseconds
#>                  expr       min        lq      mean    median        uq
#>           paradox_new  56.28408  61.05218  69.37902  65.91111  71.71352
#>  paradox_new_repaired 219.31653 242.45706 274.42037 257.69544 284.44070
#>           paradox_rep  13.35333  14.62634  18.89729  15.87236  20.10294
#>                distr6  13.28087  14.60666  19.41066  16.28139  19.72731
#>                param6  13.27241  15.17947  20.03385  17.56478  21.00630
#>       max neval cld
#>  114.0315   100  b
#>  423.6844   100   c
#>  113.8388   100 a
#>  162.8358   100 a
#>  134.9014   100 a

mb706 · 2021-03-23T13:25:00Z

                                                                                                  
> microbenchmark::microbenchmark(pipeline_greplicate(lrn_po, n = 60), times = 3)
Unit: milliseconds                 
                                expr      min       lq     mean  median                           
 pipeline_greplicate(lrn_po, n = 60) 301.0116 304.2843 327.9852 307.557
       uq      max neval          
 341.4719 375.3868     3  
> microbenchmark::microbenchmark(pipeline_greplicate(lrn_po, n = 100), times = 3)                  
Unit: milliseconds                                                                                
                                 expr     min       lq     mean   median                          
 pipeline_greplicate(lrn_po, n = 100) 494.757 499.7468 511.4505 504.7366
       uq     max neval                                                                           
 519.7973 534.858     3                                                                           
> microbenchmark::microbenchmark(pipeline_greplicate(lrn_po, n = 1000), times = 3)                 
Unit: seconds                                                                                     
                                  expr      min       lq     mean   median                        
 pipeline_greplicate(lrn_po, n = 1000) 5.796972 5.826553 5.991252 5.856133                        
       uq     max neval                                                                           
 6.088392 6.32065     3

mb706 added 4 commits March 22, 2021 20:24

faster

7ea0d92

extend tests a little

8e59aab

minor improvements

ae1e32d

document

1fa2d73

more efficient deps handling in PSC

ca6c1c3

mb706 added 9 commits March 23, 2021 01:09

fix ps_union

3592457

transparent tags

edc1044

dynamic tags shadow

fa26af8

avoid some instantiation of tags

922d2c3

renaming param to prevent misuse

9f13b8b

trying to make R 4 tests work

21baf90

testing shadow tags

7cf9ac9

tag shadows work with adds / removes

7ee166e

tag shadows work with adds / removes

38ac8c8

pat-s mentioned this pull request Aug 26, 2021

test with paradox performance improvement mlr-org/mlr3book#228

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Param init delay #344

Param init delay #344

mb706 commented Mar 22, 2021 •

edited

Loading

mb706 commented Mar 22, 2021

mb706 commented Mar 23, 2021

mb706 commented Mar 23, 2021

mb706 commented Mar 23, 2021

Param init delay #344

Are you sure you want to change the base?

Param init delay #344

Conversation

mb706 commented Mar 22, 2021 • edited Loading

mb706 commented Mar 22, 2021

Benchmark

In Brief

Benchmark

Param Construction Times

ParamSet Construction Times (without values)

ParamSet Construction Times (with values)

Getting Values

Setting Values

ParamSetCollection

ParamSetCollection (Many Params)

ParamSetCollection (Lots and Lots of Params)

ParamSetCollection (Many Params, created inside benchmark)

ParamSetCollection (Lots and Lots of Params, created inside benchmark)

mb706 commented Mar 23, 2021

mb706 commented Mar 23, 2021

mb706 commented Mar 23, 2021

mb706 commented Mar 22, 2021 •

edited

Loading