ptdf calculation #604

Cellophil · 2023-04-14T14:33:19Z

No description provided.

for more information, see https://pre-commit.ci

Cellophil · 2023-04-14T14:37:44Z

I have the suspicion that small impedances cause numerical problems in the PTDF calculation - can someone confirm or check? Possibly a warning could be put in consistency check if the impedances are too far apart.
Apart from that I converted PTDF and BODF into DataFrames, and replaced the sparse solve by sparse inverse. I don't know why sparse solve was used, maybe there's a good reason, but inverse seems to be the dedicated function to do it.

fneum · 2023-04-15T09:08:11Z

Do you have a small example where you see numerical problems in the PTDF calculation, that could help us reproduce the behaviour?

Regarding the PR: I think using the scipy inversion function makes a lot of sense; also turning PTDF and BODF into DataFrames. But the tests fail because the code is not yet fully adapted to take DataFrames rather than numpy arrays for these.

Cellophil · 2023-04-15T09:31:26Z

Hi, yes I realized, I'll debug. There's a sum in the bodf calculation that adds a numpy matrix and a scipy sparse, that fails at the moment. Maybe the possibility of keeping Ptdf and bodf sparse was a reason for returning pure numpy in the first place... Am 15. Apr. 2023, 11:08, um 11:08, Fabian Neumann ***@***.***> schrieb:

…

Do you have a small example where you see numerical problems in the PTDF calculation, that could help us reproduce the behaviour? Regarding the PR: I think using the scipy inversion function makes a lot of sense; also turning PTDF and BODF into DataFrames. But the tests fail because the code is not yet fully adapted to take DataFrames rather than numpy arrays for these. -- Reply to this email directly or view it on GitHub: #604 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

Cellophil · 2023-04-15T09:46:19Z

I had a case where the B matrix had entries of 1e9 - the solver reported a singular matrix. Scaling down actually "solved" it but adapting the impedance spread is maybe better. I'll extract an example. Maybe an optional sparse return format for ptdf and bodf could be an unstacked version of a Dataframe ie multi index pandas Series with cutoff? Am 15. Apr. 2023, 11:08, um 11:08, Fabian Neumann ***@***.***> schrieb:

…

Do you have a small example where you see numerical problems in the PTDF calculation, that could help us reproduce the behaviour? Regarding the PR: I think using the scipy inversion function makes a lot of sense; also turning PTDF and BODF into DataFrames. But the tests fail because the code is not yet fully adapted to take DataFrames rather than numpy arrays for these. -- Reply to this email directly or view it on GitHub: #604 (comment) You are receiving this because you authored the thread. Message ID: ***@***.***>

… ptdf_update

for more information, see https://pre-commit.ci

Cellophil · 2023-04-18T16:43:21Z

Updates on my side:

tbh, I'm not so sure about the numerical issue anymore, possibly I had a hickup there
the various matrices calculated in sub_network rely on using the same bus and branch ordering in a number of places - probably in order to use scipy sparse matrices. Some functions use for loops a lot, maybe there's a speedup to be gained (incidence matrix for instance)
An equivalent of sparse scipy matrices could be set up in the form of Pandas Series with multiindexes. It's an extra step in the middle and probably a bit slower than the blank handling of numpy and scipy, but it would keep track of the index.
I'll try to set up a working version that accepts the result at the end and attaches the correct indexes. For now.
til pandas supports sparse matrices

… ptdf_update

for more information, see https://pre-commit.ci

… ptdf_update

for more information, see https://pre-commit.ci

Cellophil · 2023-05-12T10:06:51Z

This is probably on me but I have problems with pytest - pandapower and it's numba dependency I think. Sorry for the failed checks, hope it starting to take form.

… ptdf_update

for more information, see https://pre-commit.ci

Cellophil · 2023-05-16T11:58:36Z

Hi @fneum ,

I replaced the usage of BODF and PTDF is a dozen places; and I feel it really makes sense to continue with pandas instead of the annonymous numpy. Some further things should be cleaned up, I patched it in some places...
What I couldn't really check is p_branch_shift and p_bus_shift, I don't know whether that's covered by the test cases. And what it does.

Best,
Andreas

Cellophil · 2023-05-17T07:26:57Z

@fneum One test fails concerning tkinter and some plotting library - it passes on my machine.

codecov · 2023-06-22T06:52:25Z

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (e3a38e7) 78.15% compared to head (b586369) 78.14%.
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #604      +/-   ##
==========================================
- Coverage   78.15%   78.14%   -0.01%     
==========================================
  Files          26       26              
  Lines        6756     6773      +17     
  Branches     1470     1473       +3     
==========================================
+ Hits         5280     5293      +13     
- Misses       1162     1164       +2     
- Partials      314      316       +2

Files	Coverage Δ
pypsa/components.py	`79.43% <100.00%> (+0.03%)`	⬆️
pypsa/opf.py	`89.92% <100.00%> (+0.01%)`	⬆️
pypsa/optimization/abstract.py	`46.34% <100.00%> (ø)`
pypsa/pf.py	`83.90% <90.47%> (+0.12%)`	⬆️
pypsa/contingency.py	`78.34% <76.92%> (-1.40%)`	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fneum

Sorry for taking a long time to get to this! This looks very good!

I had a few small questions. Maybe you can answer them.

Have you tested that the code changes yield same PTDF as before?

The one test failure is fine.

Final thing: Would you add a release note outlining the changes? I.e. BODF and PTDF are now dataframes and new utility function to build PTDF for whole network.

fneum · 2023-07-10T20:05:45Z

pypsa/pf.py

+def get_network_ptdf(network):
+    # convenience function
+


Maybe add a small docstring that this gives PTDF for whole network (not per subnetwork).

fneum · 2023-07-10T20:08:02Z

pypsa/pf.py

+    sub_network.p_branch_shift = (
+        -b * phase_shift
+    )  # np.multiply(-b, phase_shift, where=b != np.inf)


Could you briefly explain why where=b != np.inf is no longer needed?

And remove old code.

fneum · 2023-07-10T20:08:48Z

pypsa/pf.py

        logger.warning(
            "Warning! Some series impedances are zero - this will cause a singularity in LPF!"
        )
-    b_diag = csr_matrix((b, (r_[: len(b)], r_[: len(b)])))
+
+    # old: b_diag = csr_matrix((b, (np.r_[: len(b)], np.r_[: len(b)])))


Suggested change

# old: b_diag = csr_matrix((b, (np.r_[: len(b)], np.r_[: len(b)])))

fneum · 2023-07-10T20:09:14Z

pypsa/pf.py

    # susceptances
-    b = np.divide(1.0, z, out=np.full_like(z, np.inf), where=z != 0)
+    # old: b = np.divide(1.0, z, out=np.full_like(z, np.inf), where=z != 0)


Suggested change

# old: b = np.divide(1.0, z, out=np.full_like(z, np.inf), where=z != 0)

fneum · 2023-07-10T20:09:40Z

pypsa/pf.py

+    # I = eye(n_pvpq)
+    # B_inverse = spsolve(csc_matrix(sub_network.B[1:, 1:]), I)


Suggested change

# I = eye(n_pvpq)

# B_inverse = spsolve(csc_matrix(sub_network.B[1:, 1:]), I)

fneum · 2023-07-10T20:09:49Z

pypsa/pf.py

@@ -24,10 +24,11 @@
 from numpy import ones, r_
 from numpy.linalg import norm
 from pandas.api.types import is_list_like
-from scipy.sparse import csc_matrix, csr_matrix, dok_matrix
+from scipy.sparse import csc_matrix, csr_matrix, diags, dok_matrix, eye


Suggested change

from scipy.sparse import csc_matrix, csr_matrix, diags, dok_matrix, eye

from scipy.sparse import csc_matrix, csr_matrix, diags, dok_matrix

fneum · 2023-07-10T20:10:35Z

pypsa/pf.py

-    z = np.concatenate(
-        [
-            (c.df.loc[c.ind, attribute]).values
-            for c in sub_network.iterate_components(network.passive_branch_components)
-        ]
-    )
+    z = sub_network.branches()[attribute]


Oh, this looked very complicated beforehand.

Cellophil · 2023-07-13T14:07:16Z

Hi @fneum , great I'll check it out next week - agreed code needs a little polishing. With ptdf index, I'll check it once again that it's correct.

FabianHofmann · 2023-10-09T16:02:15Z

Hey @Cellophil, gently pinging you. Do you think it is feasible for you to resolve the remaining issues?

Cellophil · 2023-10-10T07:33:50Z

Hi, sry for the delay, I took parental leave. Time is unfortunately a bit limited atm, but I'll check things this week. Best, Andreas Am 9. Okt. 2023, 18:02, um 18:02, Fabian Hofmann ***@***.***> schrieb:

…

Hey @Cellophil, gently pinging you. Do you think it is feasible for you to resolve the remaining issues? -- Reply to this email directly or view it on GitHub: #604 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

FabianHofmann · 2023-10-10T09:07:46Z

Hey Andreas, no worries, thanks for letting us know. The young ones have full priority :) We can also try to take it from here.

Cellophil and others added 2 commits April 14, 2023 16:32

couple improvements

ed888bd

[pre-commit.ci] auto fixes from pre-commit.com hooks

65b0286

for more information, see https://pre-commit.ci

Cellophil changed the title ~~couple improvements~~ ptdf calculation Apr 14, 2023

bug fix

aa14b1d

Cellophil and others added 4 commits April 17, 2023 18:14

updates

da3d6f2

bug fixes, use buses_o instead of buses_i()

befb3ad

Merge branch 'ptdf_update' of https://github.com/Cellophil/PyPSA into…

48b3a22

… ptdf_update

[pre-commit.ci] auto fixes from pre-commit.com hooks

ea1c304

for more information, see https://pre-commit.ci

Cellophil and others added 17 commits April 19, 2023 08:32

Merge branch 'master' into ptdf_update

c6e38c5

PTDF network formulation adapted to pandas

9628d1a

bug fix

fe3b2b1

Merge branch 'ptdf_update' of https://github.com/Cellophil/PyPSA into…

d170f03

… ptdf_update

[pre-commit.ci] auto fixes from pre-commit.com hooks

6fd41a5

for more information, see https://pre-commit.ci

Merge branch 'master' into ptdf_update

59d9445

couple improvements

b628cff

bug fix

0319426

updates

b184ad6

bug fixes, use buses_o instead of buses_i()

e58dc85

[pre-commit.ci] auto fixes from pre-commit.com hooks

942389c

for more information, see https://pre-commit.ci

PTDF network formulation adapted to pandas

59e9714

bug fix

c2f8977

[pre-commit.ci] auto fixes from pre-commit.com hooks

15e9aea

for more information, see https://pre-commit.ci

bodf column names

ad2bbe3

bug fix

0cb5cec

bug fixes and network ptdf convenience fct

8425959

Cellophil and others added 3 commits May 12, 2023 12:04

Merge branch 'ptdf_update' of https://github.com/Cellophil/PyPSA into…

3b7c2b5

… ptdf_update

-

fe7229f

[pre-commit.ci] auto fixes from pre-commit.com hooks

0b830fa

for more information, see https://pre-commit.ci

Cellophil and others added 7 commits May 12, 2023 13:15

multiindex for B

c77c19a

lpf_cont adapted to pd df

c34464d

sorting through, checking diff in lpf

e56ec5a

bug fix, B setup

7b9631a

Merge branch 'ptdf_update' of https://github.com/Cellophil/PyPSA into…

e4d1230

… ptdf_update

[pre-commit.ci] auto fixes from pre-commit.com hooks

3096201

for more information, see https://pre-commit.ci

Merge branch 'master' into ptdf_update

2aa05d4

Merge branch 'master' into ptdf_update

70fc2d9

fneum reviewed Jul 10, 2023

View reviewed changes

Merge branch 'master' into ptdf_update

756fa7e

Merge branch 'master' into ptdf_update

b586369

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ptdf calculation #604

ptdf calculation #604

Cellophil commented Apr 14, 2023

Cellophil commented Apr 14, 2023

fneum commented Apr 15, 2023

Cellophil commented Apr 15, 2023 via email

Cellophil commented Apr 15, 2023 via email

Cellophil commented Apr 18, 2023 •

edited

Cellophil commented May 12, 2023

Cellophil commented May 16, 2023

Cellophil commented May 17, 2023

codecov bot commented Jun 22, 2023 •

edited

fneum left a comment

fneum Jul 10, 2023

fneum Jul 10, 2023

fneum Jul 10, 2023

fneum Jul 10, 2023

fneum Jul 10, 2023

fneum Jul 10, 2023

fneum Jul 10, 2023

Cellophil commented Jul 13, 2023

FabianHofmann commented Oct 9, 2023

Cellophil commented Oct 10, 2023 via email

FabianHofmann commented Oct 10, 2023

		# I = eye(n_pvpq)
		# B_inverse = spsolve(csc_matrix(sub_network.B[1:, 1:]), I)

	from scipy.sparse import csc_matrix, csr_matrix, diags, dok_matrix, eye
	from scipy.sparse import csc_matrix, csr_matrix, diags, dok_matrix

ptdf calculation #604

Are you sure you want to change the base?

ptdf calculation #604

Conversation

Cellophil commented Apr 14, 2023

Cellophil commented Apr 14, 2023

fneum commented Apr 15, 2023

Cellophil commented Apr 15, 2023 via email

Cellophil commented Apr 15, 2023 via email

Cellophil commented Apr 18, 2023 • edited

Cellophil commented May 12, 2023

Cellophil commented May 16, 2023

Cellophil commented May 17, 2023

codecov bot commented Jun 22, 2023 • edited

Codecov Report

fneum left a comment

Choose a reason for hiding this comment

fneum Jul 10, 2023

Choose a reason for hiding this comment

fneum Jul 10, 2023

Choose a reason for hiding this comment

fneum Jul 10, 2023

Choose a reason for hiding this comment

fneum Jul 10, 2023

Choose a reason for hiding this comment

fneum Jul 10, 2023

Choose a reason for hiding this comment

fneum Jul 10, 2023

Choose a reason for hiding this comment

fneum Jul 10, 2023

Choose a reason for hiding this comment

Cellophil commented Jul 13, 2023

FabianHofmann commented Oct 9, 2023

Cellophil commented Oct 10, 2023 via email

FabianHofmann commented Oct 10, 2023

Cellophil commented Apr 18, 2023 •

edited

codecov bot commented Jun 22, 2023 •

edited