You may try setting JVM properties. Java allows you to configure the maximum heap size and various other settings with command line parameters when the JVM is started. BeakerX allows you to add parameters to the JVM with a control panel added to Jupyter's tree view.
Related issues: #7666.