Consolidate interface code between python bindings and pyprima

[nbelakovski]

Right now both the python bindings and pyprima have their own separate __init__.py files with a lot of code to interpret the problem and translate it into a form that that corresponding interface can accept.

Initially I thought that having separate files would make sense since these are two separate packages, but now I think it makes more sense to consolidate them, as there is a fair bit of common code, and the result is that we have a lot of duplication not only of the code, but also of the tests.

I think if we have a folder at the root of the project named pycommon and we symlink it into python and pyprima that should make the relationships clear and also maintainable.

[zaikunzhang]

Hello @nbelakovski ,

Thank you for thinking about this.

I propose the following, which is what I do with MATLAB.

1. Keep only one Python interface for calling PRIMA solvers. This interface should provide a function named prima, which is the only function that the users are supposed to call.
2. The prima function should accept an option named fortran; if it is true, then call the Fortran backend via the Python binding; otherwise, call pyprima.
3. Everything concerning Python should be located in a directory named python directly under the PRIMA root directory. The python directory then contains the code(s) for the Python function prima and (at least) two subdirectories, one for the Python binding and the other for pyprima.
4. The Python PRIMA package published on PyPI provides the prima function, which provides both the Python bindings and pyprima.

Thanks.

Zaikun

[nbelakovski]

Thanks for your response @zaikunzhang. I have some follow ups.

1. Keep only one Python interface for calling PRIMA solvers. This interface should provide a function named prima, which is the only function that the users are supposed to call.

I thought that for Python code we are trying to maintain API compatibility with both scipy and COBYQA? In that case the function that users call should be called minimize and should have the same interface as the one from those two packages. The idea here is to make it easy for Python users to switch packages - they should only have to change the name of the package from which they import minimize and nothing else.

2. The prima function should accept an option named fortran; if it is true, then call the Fortran backend via the Python binding; otherwise, call pyprima.

Does this mean that we will only be publishing one package under two names (prima and pyprima)? That could make distribution tricky. As we've discussed, there are some issues with musl which make it difficult for us to publish the bindings to that platform, but obviously we can publish pure Python code to any platform. Having separate packages can make it easier to support more platforms for the pure Python code. Having separate packages also means that any issues with the bindings will not delay releases of pyprima for new Python versions or new platforms down the road. Additionally the fortran option would break interoperability with the minimize interface discussed above (this problem could be solved if we make fortran one of the keywords in the options argument).

3. Everything concerning Python should be located in a directory named python directly under the PRIMA root directory. The python directory then contains the code(s) for the Python function prima and (at least) two subdirectories, one for the Python binding and the other for pyprima.

I agree that we should try to keep the Python code consolidated. I think this can be done but the structure might look quite different. Python build systems is a strange ecosystem, but I will do my best to come up with as tidy of a solution as I can.

4. The Python PRIMA package published on PyPI provides the prima function, which provides both the Python bindings and pyprima.

See my questions for 1. and 2.

[zaikunzhang]

I thought that for Python code we are trying to maintain API compatibility with both scipy and COBYQA?

The signature of COBYQA is different from that of SciPy, as far as I remember.

We should keep consistent with PDFO instead of COBYQA. PDFO was designed to be compatible with SciPy.

In that case the function that users call should be called minimize and should have the same interface as the one from those two packages.

Sure.

Then I don't know whether we should provide the prima function. Should we offer both a prima function and a minimize function? Let us do the same as PDFO (replacing pdfo with prima wherever applicable).

Does this mean that we will only be publishing one package under two names (prima and pyprima)?

No. We publish only one package under only one name prima. This package can call both the Fortran version and the Python version of the PRIMA solvers

As we've discussed, there are some issues with musl which make it difficult for us to publish the bindings to that platform, but obviously we can publish pure Python code to any platform.

I don't think this is an issue if we do some checking in the code. See, for example,

prima/matlab/interfaces/private/preprima.m

Line 2035 in 2f09e46

wmsg = sprintf('%s: fortran = true but the Fortran MEX function is not available for the %s variant of %s with precision %s; fortran is reset to false.', invoker, variant, options.solver, options.precision);

Basically, if the Fortran version is requested but not available, then fall back to the Python version.

Additionally the fortran option would break interoperability with the minimize interface discussed above (this problem could be solved if we make fortran one of the keywords in the options argument).

PRIMA should support everything that SciPy supports, but it can support more than SciPy. This is also how PDFO was designed.

I think this can be done but the structure might look quite different. Python build systems is a strange ecosystem, but I will do my best to come up with as tidy of a solution as I can.

We have to open our minds here :)

Don't regard https://github.com/libprima/prima as the Python package you want to build. Obviously, it is not. Instead, it is a "mother package" that contains the code needed by the Python package. In the worst case, the building system of the Python package can just copy the necessary code to a temporary directory and build it there. In other words, the Python package can be generated "on the fly" (in the worst case) instead of being stored statically at https://github.com/libprima/prima.

[nbelakovski]

For the sake of completeness here is a look at all the signatures:

COBYQA signature:

def minimize(fun, x0, args=(), bounds=None, constraints=(), callback=None, options=None, **kwargs):

PDFO signature:

def pdfo(fun, x0, args=(), method=None, bounds=None, constraints=(), options=None):

scipy signature:

def minimize(fun, x0, args=(), method=None, jac=None, hess=None,
             hessp=None, bounds=None, constraints=(), tol=None,
             callback=None, options=None):

	name	fun	x0	args	method	jac	hess	hessp	bounds	constraints	tol	callback	options	**kwargs
COBYQA	minimize	✅	✅	✅	❌	❌	❌	❌	✅	✅	❌	✅	✅	✅
PDFO	pdfo	✅	✅	✅	✅	❌	❌	❌	✅	✅	❌	❌	✅	❌
scipy	minimize	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	✅	❌
PRIMA?	minimize?	✅	✅	✅	✅	❌	❌	❌	✅	✅	✅	✅	✅	❌

Notably scipy has a tol option for tolerance of the final solution which was not included in either PDFO or COBYQA, and COBYQA has a **kwargs option which allows for setting of various constants like DECREASE_RADIUS_FACTOR and INCREASE_RADIUS_THRESHOLD.

In the last line I've suggested what the PRIMA interface should look like. Effectively I copied PDFO and added callback and tol (which would map to rhoend) for the sake of keeping it compatible with scipy (but of course we won't add jac/hess/hessp as that would be silly). I really do think we should keep the name minimize which is what we've started out with and what we've included in scipy. Having function names be verbs that reflect what the function itself does makes the code clearer and easier to remember. I'd even argue that a minimize function should be added to PDFO and to the MATLAB code as well, while keeping the old interfaces for backwards compatibility of course. Are you OK with adding tol or would you prefer to leave it out and continue using options to set rhoend (which would still happen even if we have tol as an argument).

We publish only one package under only one name prima. This package can call both the Fortran version and the Python version of the PRIMA solvers.

The prima function should accept an option named fortran; if it is true, then call the Fortran backend via the Python binding; otherwise, call pyprima.

I think we could do this if we whitelist the platforms we support in the CMake build, but I don't really want to work on merging the two packages right now. My main goal is to consolidate the duplicate code and ultimately to get the python bindings published in order to address scipy/scipy#23509