Bug(?) when adding constraints to Links

[andrevlisboa]

Checklist

• I am using the current master branch or the latest

Describe the Bug

I think there might me an error when post-processing the network model, when multiple Links have additional constraints. When it is just one, it works fine.

As an example:

Looking at a modelling like:

|main_bus|-----|bus|--link--|store|

And adding the following constraint:

link_power = network_model.variables["Link-p"].sel({"Link": link["name"]})

lhs = link_power.sum()
rhs = 100 (for example)
network_model.add_constraints(lhs == rhs, name="constraint1")


link_power = network_model.variables["Link-p"].sel({"Link": link["name"]})

lhs = link_power.sum()
rhs = 100 (for example)
network_model.add_constraints(lhs == rhs, name="constraint1")

Error Message

Xarray will raise an error in post-processing on merge data

Note: The optimization is done correctly, however the post processing raises an error

[fneum]

What is the error message? Do you have an example how we can reproduce the behaviour?

[andrevlisboa]

Hi!

Example below:

import pypsa
import pandas as pd

SNAPSHOTS = pd.date_range("2023-01-01 09:00", "2023-01-01 19:00", freq="H")

LOAD = {"name": "load_1",
    	"profile": pd.Series([40, 70, 60, 50, 10, 20, 40, 50, 50, 40, 40], index=SNAPSHOTS)}

GENERATORS = [{"name": "gas_1",
    			    "p_nom": 100,
                            "p_min" : 0,
       			     "marginal_cost": 10},
              {"name": "gas_2",
    			"p_nom": 100,
			"p_min" : 50,
       			"marginal_cost": 5}]


############
network = pypsa.Network()

network.set_snapshots(SNAPSHOTS)

main_bus = "bus_grid"
network.add("Bus", name=main_bus)

for gen in GENERATORS:
	network.add("Generator",
				name=gen["name"],
				bus=main_bus,
				committable=True,
				p_nom=gen["p_nom"],
				marginal_cost=gen["marginal_cost"],
			)

network.add("Load", 
			name=LOAD["name"],  
			bus=main_bus,  
			p_set=LOAD["profile"]) 

network_model = network.optimize.create_model()

# CONSTRAINT: Generator has min.power
for gen in GENERATORS:
        generator_power = network_model.variables['Generator-p'].sel(Generator=gen["name"])
        generator_status = network_model.variables['Generator-status'].sel({"Generator-com": gen["name"]})
        lhs = generator_power - generator_status * gen["p_min"]
        rhs = 0
        network_model.add_constraints(lhs >= rhs, name="Generator-p-min"+ "_" + gen["name"])


network.optimize.solve_model(solver_name="highs")   

[FabianHofmann]

Hey, the code worked for me. Perhaps you have to update Linopy? Here are some improvements: import pypsa import xarray as xr import pandas as pd SNAPSHOTS = pd.date_range("2023-01-01 09:00", "2023-01-01 19:00", freq="H") LOAD = {     "name": "load_1",     "profile": pd.Series(         [40, 70, 60, 50, 10, 20, 40, 50, 50, 40, 40], index=SNAPSHOTS     ), } GENERATORS = [     {"name": "gas_1", "p_nom": 100, "p_min": 0, "marginal_cost": 10},     {"name": "gas_2", "p_nom": 100, "p_min": 50, "marginal_cost": 5}, ] ############ network = pypsa.Network() network.set_snapshots(SNAPSHOTS) main_bus = "bus_grid" network.add("Bus", name=main_bus) for gen in GENERATORS:     network.add(         "Generator",         name=gen["name"],         bus=main_bus,         committable=True,         p_nom=gen["p_nom"],         marginal_cost=gen["marginal_cost"],     ) network.generators["p_min"] = [gen["p_min"] for gen in GENERATORS] network.add("Load", name=LOAD["name"], bus=main_bus, p_set=LOAD["profile"]) network_model = network.optimize.create_model() generator_power = network_model.variables["Generator-p"] generator_status = network_model.variables["Generator-status"] p_min = xr.DataArray(network.generators.p_min, dims=["Generator-com"]) lhs = generator_power - generator_status * p_min rhs = 0 network_model.add_constraints(lhs >= rhs, name="Generator-p-min") network.optimize.solve_model(solver_name="highs")

[andrevlisboa]

I have the following version:
pypsa==0.25.2
linopy==0.3.1
xarray==2023.10.1

Do you have anything different?

Regarding the improvement -> since there will be two dims "Generator" and "Generator-com", it will add wrongly some constraints, right?

And thank you for the help!

[rizky-r-lab]

Same with me,
i have error in my PopOS, while in my windows 11 running fine.

The error is
File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/pypsa/optimization/optimize.py:618, in OptimizationAccessor.solve_model(self, solver_name, solver_options, assign_all_duals, **kwargs)
616 if status == "ok":
617 assign_solution(n)
--> 618 assign_duals(n, assign_all_duals)
619 post_processing(n)
621 return status, condition

File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/pypsa/optimization/optimize.py:384, in assign_duals(n, assign_all_duals)
382 m = n.model
383 unassigned = []
--> 384 for name, dual in m.dual.items():
385 try:
386 c, attr = name.split("-", 1)

File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/linopy/model.py:189, in Model.dual(self)
184 @Property
185 def dual(self):
186 """
187 Dual values calculated by the optimization.
188 """
--> 189 return self.constraints.dual

File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/linopy/constraints.py:626, in Constraints.dual(self)
622 """
623 Get the dual values of all constraints.
624 """
625 try:
--> 626 return save_join(*[v.dual.rename(k) for k, v in self.items()])
627 except AttributeError:
628 return Dataset()

File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/linopy/common.py:131, in save_join(integer_dtype, *dataarrays)
129 if integer_dtype:
130 arrs = [ds.fillna(-1).astype(int) for ds in arrs]
--> 131 return Dataset({ds.name: ds for ds in arrs})

File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/xarray/core/dataset.py:696, in Dataset.init(self, data_vars, coords, attrs)
693 if isinstance(coords, Dataset):
694 coords = coords._variables
--> 696 variables, coord_names, dims, indexes, _ = merge_data_and_coords(
697 data_vars, coords
698 )
700 self._attrs = dict(attrs) if attrs is not None else None
701 self._close = None

File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/xarray/core/dataset.py:425, in merge_data_and_coords(data_vars, coords)
421 coords = create_coords_with_default_indexes(coords, data_vars)
423 # exclude coords from alignment (all variables in a Coordinates object should
424 # already be aligned together) and use coordinates' indexes to align data_vars
--> 425 return merge_core(
426 [data_vars, coords],
427 compat="broadcast_equals",
428 join="outer",
429 explicit_coords=tuple(coords),
430 indexes=coords.xindexes,
431 priority_arg=1,
432 skip_align_args=[1],
433 )

File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/xarray/core/merge.py:720, in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value, skip_align_args)
718 collected = collect_variables_and_indexes(aligned, indexes=indexes)
719 prioritized = _get_priority_vars_and_indexes(aligned, priority_arg, compat=compat)
--> 720 variables, out_indexes = merge_collected(
721 collected, prioritized, compat=compat, combine_attrs=combine_attrs
722 )
724 dims = calculate_dimensions(variables)
726 coord_names, noncoord_names = determine_coords(coerced)

File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/xarray/core/merge.py:290, in merge_collected(grouped, prioritized, compat, combine_attrs, equals)
288 variables = [variable for variable, _ in elements_list]
289 try:
--> 290 merged_vars[name] = unique_variable(
291 name, variables, compat, equals.get(name, None)
292 )
293 except MergeError:
294 if compat != "minimal":
295 # we need more than "minimal" compatibility (for which
296 # we drop conflicting coordinates)

File ~/miniconda3/envs/pypsa/lib/python3.10/site-packages/xarray/core/merge.py:144, in unique_variable(name, variables, compat, equals)
141 break
143 if not equals:
--> 144 raise MergeError(
145 f"conflicting values for variable {name!r} on objects to be combined. "
146 "You can skip this check by specifying compat='override'."
147 )
149 if combine_method:
150 for var in variables[1:]:

MergeError: conflicting values for variable 'Link' on objects to be combined. You can skip this check by specifying compat='override'.

[FabianHofmann]

hey @rizky-r-lab, could you confirm you are one the later master branch in pypsa?

[rizky-r-lab]

hey @FabianHofmann , i install using conda. However if i install using latest master branch, there are several package installed and the error disappear.
FYI, the problems is in highs solver. Because if i using gurobi, there are no problem

[Tooblippe]

Hi, We are having a similar issue when we add these variables and constraints. The solver solves, but it also seems in post processing to trough the error below.

` #Obtain variables and attributes for link1 & store1
Link1_MWh = m.variables["Link-p"].sel(Link="Lk95_(B2_HCB-B_Songo)") # This is the electricity link from water link
Link1_p_nom = network.links.at["Lk95_(B2_HCB-B_Songo)", "p_nom"] # Capacity of electricity link from turbine
Store1_level = m.variables["Store-e"].sel(Store="HCB-Res") # This is store
Store1_e_nom = network.stores.at["HCB-Res", "e_nom"] # Total storage
min_turbine1_eff = network.stores.at["HCB-Res", "min_turbine_eff"] # This is a new attribute to be added to csv file
delta_turbine1_eff = 1 - min_turbine1_eff # Define Efficiency Range

    #Obtain variables and attributes for link2 & store2
    Link2_MWh = m.variables["Link-p"].sel(Link="Lk114_(B3_MNK-B_MNK)") # This is the electricity link from water link
    Link2_p_nom = network.links.at["Lk114_(B3_MNK-B_MNK)", "p_nom"]  # Capacity of electricity link from turbine
    Store2_level = m.variables["Store-e"].sel(Store="MNK-Res") # This is store
    Store2_e_nom = network.stores.at["MNK-Res", "e_nom"]  # Total storage
    min_turbine2_eff = network.stores.at["MNK-Res", "min_turbine_eff"] # This is a new attribute to be added to csv file
    delta_turbine2_eff = 1 - min_turbine2_eff # Define Efficiency Range

    # Add the constraint 
    m.add_constraints(
        Link1_MWh
        <= ((Store1_level / Store1_e_nom * delta_turbine1_eff) + min_turbine1_eff) * Link1_p_nom,
        name="Turbine1_eff",
        )
    m.add_constraints(
        Link2_MWh
        <= ((Store2_level / Store2_e_nom * delta_turbine2_eff) + min_turbine2_eff) * Link2_p_nom,
        name="Turbine2_eff",
        )`

Error

`C:\Users\User\AppData\Roaming\Python\Python39\site-packages\linopy\common.py:124: UserWarning: Coordinates across variables not equal. Perform outer join.
warn(
Traceback (most recent call last):

File "C:\Users\User\OneDrive\S(Pypsa)\2401_GMNK\Single-Yr_GMNK_v3.2.py", line 186, in
network=expansion('A(RSA-DC_M_P50)')

File "C:\Users\User\OneDrive\S(Pypsa)\2401_GMNK\Single-Yr_GMNK_v3.2.py", line 165, in expansion
network.optimize.solve_model(solver_name="cplex",

File "C:\Users\User.conda\envs\pypsa-root\lib\site-packages\pypsa\optimization\optimize.py", line 616, in solve_model
assign_duals(n, assign_all_duals)

File "C:\Users\User.conda\envs\pypsa-root\lib\site-packages\pypsa\optimization\optimize.py", line 384, in assign_duals
for name, dual in m.dual.items():

File "C:\Users\User\AppData\Roaming\Python\Python39\site-packages\linopy\model.py", line 189, in dual
return self.constraints.dual

File "C:\Users\User\AppData\Roaming\Python\Python39\site-packages\linopy\constraints.py", line 626, in dual
return save_join(*[v.dual.rename(k) for k, v in self.items()])

File "C:\Users\User\AppData\Roaming\Python\Python39\site-packages\linopy\common.py", line 131, in save_join
return Dataset({ds.name: ds for ds in arrs})

File "C:\Users\User.conda\envs\pypsa-root\lib\site-packages\xarray\core\dataset.py", line 696, in init
variables, coord_names, dims, indexes, _ = merge_data_and_coords(

File "C:\Users\User.conda\envs\pypsa-root\lib\site-packages\xarray\core\dataset.py", line 425, in merge_data_and_coords
return merge_core(

File "C:\Users\User.conda\envs\pypsa-root\lib\site-packages\xarray\core\merge.py", line 720, in merge_core
variables, out_indexes = merge_collected(

File "C:\Users\User.conda\envs\pypsa-root\lib\site-packages\xarray\core\merge.py", line 290, in merge_collected
merged_vars[name] = unique_variable(

File "C:\Users\User.conda\envs\pypsa-root\lib\site-packages\xarray\core\merge.py", line 144, in unique_variable
raise MergeError(

MergeError: conflicting values for variable 'Link' on objects to be combined. You can skip this check by specifying compat='override'.

None`