Home
Uni-Logo
 

Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars

Simon Schrodi, D. Stoll, B. Ru, R. Sukthanker, Thomas Brox, F. Hutter
Advances in Neural Information Processing Systems (NeurIPS), 2023
Abstract: The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards this goal but lack a unifying search space design framework and typically only search over some limited aspect of architectures. In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature. By enhancing and using their properties, we effectively enable search over the complete architecture and can foster regularity. Further, we propose an efficient hierarchical kernel design for a Bayesian Optimization search strategy to efficiently search over such huge spaces. We demonstrate the versatility of our search space design framework and show that our search strategy can be superior to existing NAS approaches. Code is available at https://github.com/automl/hierarchical_nas_construction.
Paper Downloads

Images and movies

 

BibTex reference

@InProceedings{SB23,
  author       = "S. Schrodi and D. Stoll and B. Ru and R. Sukthanker and T. Brox and F. Hutter",
  title        = "Construction of Hierarchical Neural Architecture Search Spaces based on Context-free Grammars",
  booktitle    = "Advances in Neural Information Processing Systems (NeurIPS)",
  month        = " ",
  year         = "2023",
  url          = "http://lmbweb.informatik.uni-freiburg.de/Publications/2023/SB23"
}

Other publications in the database