Research software and paper
Convex Two-Layer ReLU Fitting
A paper and reference implementation for fitting sufficiently wide, scalar-output, fully connected two-layer ReLU networks with weight decay through an equivalent convex problem.
Current state
Fitting a sufficiently wide scalar-output two-layer ReLU network with weight decay can be represented as a convex problem, but writing that problem down directly requires considering a number of activation regions that can grow combinatorially with the data. Rather than enumerate all of those regions, the solver keeps a small working set of ReLU features, refits their coefficients, and uses the current residual to search for useful features to add. The paper’s guarantees apply to an idealized version that finds the best new feature exactly at every iteration. The implementation makes this choice heuristically, so its reported gap values are numerical estimates rather than rigorous certificates of global optimality.
Future plans
In the short term, we plan to measure how this method performs at scale with multiple CPUs or GPUs. Later, we plan to investigate extensions to deeper neural networks, other architectures, and vector outputs.
Details
- Phase
- Paper and code public
- Domain
- Convex optimization · neural network fitting
- My role
- Author and developer
Artifacts
- Repository repository
- Paper paper