Crash in GlobalRouter repairAntennas #4878

MichaelBell · 2024-03-30T23:39:44Z

Describe the bug

While running the Global Routing step, on certain designs a crash occurs in repair antennas. This only seems to happen if more than one iteration is required. This seems very similar to #4471, but I have confirmed it still happens with a very recent version of OpenROAD.

Expected Behavior

The global router should not crash

OpenROAD Environment

I'm using OpenROAD through an OpenLane docker image efabless/openlane-tools:openroad_app-9741bc497a3d5f434b0dacdc3a05af4ebe78846c-centos-7-amd64 using OpenROAD 

But this happens with various versions of OpenROAD, including 6ff71c8822a72dc0c33dd5a8a10076d8691b5a4f and 0889970d1790a2617e69f253221b8bd7626e51dc (used in OpenLane tag 2024.03.07 and 2024.03.12).

It does not happen with 6f9b2bb8b808b1bb5831d4525d868212ae50517a (used in OpenLane tag 2023.11.23).

OpenLane Environment

open_pdks cd1748bb197f9b7af62a54507de6624e30363943
Kernel: Linux v5.15.146.1-microsoft-standard-WSL2
Distribution: ubuntu 22.04
Python: v3.10.12 (OK)
Container Engine: docker v20.10.25 (OK)
OpenLane Git Version: fix-antenna-crash-dev
python-venv: INSTALLED
---
PDK Version Verification Status: OK
---
Git Log (Last 3 Commits)

1648ffd 2024-03-30T19:27:25+00:00 Update OpenROAD again - Mike Bell -  (HEAD -> fix-antenna-crash)
60f6256 2024-03-29T23:08:42+00:00 Update OpenROAD - Mike Bell -  ()
a663df2 2024-03-06T15:47:26+02:00 Update OpenROAD (#2093) - Mohamed Gaber -  (grafted)
---
Git Remotes

mike    git@github.com:MichaelBell/OpenLane.git (fetch)
mike    git@github.com:MichaelBell/OpenLane.git (push)
origin  https://github.com/The-OpenROAD-Project/OpenLane.git (fetch)
origin  https://github.com/The-OpenROAD-Project/OpenLane.git (push)

To Reproduce

Clone the openroad-antenna-bug branch of https://github.com/MichaelBell/tt06-tinyQV recursively
Follow https://docs.google.com/document/d/1aUUZ1jthRpg4QURIIyzlOaPWlmQzr-jBn3wZipVUPt4/edit#heading=h.wwc5ldl01nl5 to set up the OpenLane flow for TinyTapeout, with the following changes:

Use the above project instead of the factory test project.
Use the 2024.03.07 tag of OpenLane

Run:

tt/tt_tool.py --create-user-config
tt/tt_tool.py --harden

I've also attached a zip of the issue reproducible folder.
issue_reproducible.zip

Relevant log output

From 19-global.log:

[INFO GRT-0018] Total wirelength: 268085 um
[INFO GRT-0014] Routed nets: 5770
[INFO GRT-0006] Repairing antennas, iteration 1.
[INFO GRT-0043] No OR_DEFAULT vias defined.
[INFO GRT-0012] Found 32 antenna violations.
[INFO GRT-0015] Inserted 36 diodes.
[INFO GRT-0009] rerouting 4422 nets.
[INFO GRT-0001] Minimum degree: 2
[INFO GRT-0002] Maximum degree: 26
[INFO GRT-0006] Repairing antennas, iteration 2.
[INFO GRT-0043] No OR_DEFAULT vias defined.
[INFO GRT-0012] Found 17 antenna violations.
[INFO GRT-0015] Inserted 21 diodes.
[INFO GRT-0009] rerouting 550 nets.
[INFO GRT-0001] Minimum degree: 2
[INFO GRT-0002] Maximum degree: 16
[INFO GRT-0006] Repairing antennas, iteration 3.
[INFO GRT-0043] No OR_DEFAULT vias defined.
Signal 11 received
Stack trace:
 0# 0x0000000000D8E2B7 in openroad
 1# 0x00007F62E3B66400 in /lib64/libc.so.6
 2# odb::_dbBox::isOct() const in openroad
 3# odb::dbBox::getBox() in openroad
 4# odb::dbWirePathItr::getNextShape(odb::dbWirePathShape&) in openroad
 5# odb::tmg_conn::loadWire(odb::dbWire*) in openroad
 6# odb::tmg_conn::analyzeNet(odb::dbNet*) in openroad
 7# odb::orderWires(utl::Logger*, odb::dbNet*) in openroad
 8# grt::RepairAntennas::makeNetWire(odb::dbNet*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >&, std::map<int, odb::dbTechVia*, std::less<int>, std::allocator<std::pair<int const, odb::dbTechVia*> > >&) in openroad
 9# grt::RepairAntennas::makeNetWires(std::map<odb::dbNet*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >, grt::cmpById, std::allocator<std::pair<odb::dbNet* const, std::vector<grt::GSegment, std::allocator<grt::GSegment> > > > >&, int) in openroad
10# grt::RepairAntennas::checkAntennaViolations(std::map<odb::dbNet*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >, grt::cmpById, std::allocator<std::pair<odb::dbNet* const, std::vector<grt::GSegment, std::allocator<grt::GSegment> > > > >&, int, odb::dbMTerm*, float) in openroad
11# grt::GlobalRouter::repairAntennas(odb::dbMTerm*, int, float) in openroad
12# 0x00000000012E4276 in openroad

OpenLane console log:
(venv) ~/tt/tt06-tinyQV$ tt/tt_tool.py --harden
OpenLane 1648ffd17a9430240eb753e4bb85b67ff99dc568
All rights reserved. (c) 2020-2023 Efabless Corporation and contributors.
Available under the Apache License, version 2.0. See the LICENSE file for more details.

[INFO]: Using configuration in '../work/src/config.tcl'...
[INFO]: PDK Root: /home/mdb36/tt/pdk
[INFO]: Process Design Kit: sky130A
[INFO]: Standard Cell Library: sky130_fd_sc_hd
[INFO]: Optimization Standard Cell Library: sky130_fd_sc_hd
[INFO]: Run Directory: /work/runs/wokwi
[INFO]: Removing existing /work/runs/wokwi...
[WARNING]: The variable name DESIGN_IS_CORE was renamed to FP_PDN_MULTILAYER. Update your configuration file.
[INFO]: Saving runtime environment...
[INFO]: Preparing LEF files for the nom corner...
[INFO]: Preparing LEF files for the min corner...
[INFO]: Preparing LEF files for the max corner...
[INFO]: Running linter (Verilator) (log: ../work/runs/wokwi/logs/synthesis/linter.log)...
[INFO]: 0 errors found by linter
[INFO]: 0 warnings found by linter
[STEP 1]
[INFO]: Running Synthesis (log: ../work/runs/wokwi/logs/synthesis/1-synthesis.log)...
[STEP 2]
[INFO]: Running Single-Corner Static Timing Analysis (log: ../work/runs/wokwi/logs/synthesis/2-sta.log)...
[STEP 3]
[INFO]: Running Initial Floorplanning (log: ../work/runs/wokwi/logs/floorplan/3-initial_fp.log)...
[INFO]: Floorplanned with width 329.36 and height 220.32.
[STEP 4]
[INFO]: Running IO Placement (log: ../work/runs/wokwi/logs/floorplan/4-io.log)...
[INFO]: Applying DEF template...
[STEP 5]
[INFO]: Running Tap/Decap Insertion (log: ../work/runs/wokwi/logs/floorplan/5-tap.log)...
[INFO]: Power planning with power {VPWR} and ground {VGND}...
[STEP 6]
[INFO]: Generating PDN (log: ../work/runs/wokwi/logs/floorplan/6-pdn.log)...
[STEP 7]
[INFO]: Running Global Placement (log: ../work/runs/wokwi/logs/placement/6-global.log)...
[STEP 8]
[INFO]: Running Single-Corner Static Timing Analysis (log: ../work/runs/wokwi/logs/placement/8-gpl_sta.log)...
[STEP 9]
[INFO]: Running Placement Resizer Design Optimizations (log: ../work/runs/wokwi/logs/placement/9-resizer.log)...
[STEP 10]
[INFO]: Running Detailed Placement (log: ../work/runs/wokwi/logs/placement/10-detailed.log)...
[STEP 11]
[INFO]: Running Single-Corner Static Timing Analysis (log: ../work/runs/wokwi/logs/placement/11-dpl_sta.log)...
[STEP 12]
[INFO]: Running Clock Tree Synthesis (log: ../work/runs/wokwi/logs/cts/12-cts.log)...
[STEP 13]
[INFO]: Running Single-Corner Static Timing Analysis (log: ../work/runs/wokwi/logs/cts/13-cts_sta.log)...
[STEP 14]
[INFO]: Running Placement Resizer Timing Optimizations (log: ../work/runs/wokwi/logs/cts/14-resizer.log)...
[STEP 15]
[INFO]: Running Global Routing Resizer Design Optimizations (log: ../work/runs/wokwi/logs/routing/15-resizer_design.log)...
[STEP 16]
[INFO]: Running Single-Corner Static Timing Analysis (log: ../work/runs/wokwi/logs/routing/16-rsz_design_sta.log)...
[STEP 17]
[INFO]: Running Global Routing Resizer Timing Optimizations (log: ../work/runs/wokwi/logs/routing/17-resizer_timing.log)...
[STEP 18]
[INFO]: Running Single-Corner Static Timing Analysis (log: ../work/runs/wokwi/logs/routing/18-rsz_timing_sta.log)...
[STEP 19]
[INFO]: Running Global Routing (log: ../work/runs/wokwi/logs/routing/19-global.log)...
[ERROR]: during executing openroad script /openlane/scripts/openroad/repair_antennas.tcl
[ERROR]: Log: ../work/runs/wokwi/logs/routing/19-global.log
[ERROR]: Last 10 lines:
38# 0x00007F62E869BF1E in /lib64/libtcl8.5.so
39# Tcl_EvalEx in /lib64/libtcl8.5.so
40# Tcl_Eval in /lib64/libtcl8.5.so
41# sta::sourceTclFile(char const*, bool, bool, Tcl_Interp*) in openroad
42# ord::tclAppInit(Tcl_Interp*) in openroad
43# Tcl_Main in /lib64/libtcl8.5.so
44# main in openroad
45# __libc_start_main in /lib64/libc.so.6
46# 0x0000000000D85757 in openroad
child killed: segmentation violation

[ERROR]: Creating issue reproducible...
[INFO]: Saving runtime environment...
OpenLane TCL Issue Packager

EFABLESS CORPORATION AND ALL AUTHORS OF THE OPENLANE PROJECT SHALL NOT BE HELD
LIABLE FOR ANY LEAKS THAT MAY OCCUR TO ANY PROPRIETARY DATA AS A RESULT OF USING
THIS SCRIPT. THIS SCRIPT IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OR
CONDITIONS OF ANY KIND.

BY USING THIS SCRIPT, YOU ACKNOWLEDGE THAT YOU FULLY UNDERSTAND THIS DISCLAIMER
AND ALL IT ENTAILS.

Parsing config file(s)…
Setting up /work/runs/wokwi/issue_reproducible…
Done.
[INFO]: Reproducible packaged at '../work/runs/wokwi/issue_reproducible'.
Traceback (most recent call last):
  File "/home/mdb36/tt/tt06-tinyQV/tt/tt_tool.py", line 150, in <module>
    project.harden()
  File "/home/mdb36/tt/tt06-tinyQV/tt/project.py", line 481, in harden
    with open(os.path.join(self.local_dir, commit_id_json_path), "w") as f:
FileNotFoundError: [Errno 2] No such file or directory: './runs/wokwi/results/final/commit_id.json'

Screenshots

No response

Additional Context

This issue also caused failures for certain Tiny Tapeout 05 projects (that previously succeeded) when testing the 2024.03.12 tag of OpenLane:
https://github.com/TinyTapeout/tinytapeout-05-reharden/actions/runs/8264211005/job/22609263738
https://github.com/TinyTapeout/tinytapeout-05-reharden/actions/runs/8264211005/job/22609256508

The text was updated successfully, but these errors were encountered:

maliberty · 2024-04-01T04:31:14Z

@luis201420 is working on rewriting this chunk of code. I believe that will solve you problem but it will be a bit until the new code is ready.

maliberty · 2024-04-01T04:32:10Z

The general goal is to no longer call orderWires during antenna repair

Also update the sky130 PDK to the corresponding version. This solve Project/OpenROAD#4647: pins swapped incorrectly on and2b cell. It also introduces a new issue, causing the flow to crash for about 1/50 of the TT05 project dataset: The-OpenROAD-Project/OpenROAD#4878. However, we prefer that the flow crashes rather than silently produce incorrect results. GitHub action run where we tested the new version: https://github.com/TinyTapeout/tinytapeout-05-reharden/actions/runs/8525372820

dlmiles · 2024-04-05T14:24:14Z

When investigating this, the following assertion is triggered.

Is the frame posted before (containing db::orderWires(...) call a symptom, while this frame is closer to the cause ?

@maliberty no sign of db::orderWires() here, please re-evaluate if the removal of the call will address this problem, or if I should continue to investigate. Thanks

FWIW this problem/frame that looks like this can be demonstrated in many versions of OR going back the past 6 to 9 months. High congestion maybe a contributing factor.

[INFO GRT-0006] Repairing antennas, iteration 3.
[INFO GRT-0043] No OR_DEFAULT vias defined.
openroad: /openroad/src/odb/src/db/dbWireCodec.cpp:393: int odb::dbWireEncoder::addTechVia(odb::dbTechVia*): Assertion `0' failed.
Signal 6 received
Stack trace:
 0# 0x0000000004065175 in openroad
 1# 0x00007F5413F2B400 in /lib64/libc.so.6
 2# gsignal in /lib64/libc.so.6
 3# abort in /lib64/libc.so.6
 4# 0x00007F5413F241A6 in /lib64/libc.so.6
 5# 0x00007F5413F24252 in /lib64/libc.so.6
 6# odb::dbWireEncoder::addTechVia(odb::dbTechVia*) in openroad
 7# grt::RepairAntennas::addWireTerms(grt::Net*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >&, int, int, int, odb::dbTechLayer*, std::map<grt::RoutePt, grt::RoutePtPins, std::less<grt::RoutePt>, std::allocator<std::pair<grt::RoutePt const, grt::RoutePtPins> > >&, odb::dbWireEncoder&, std::map<int, odb::dbTechVia*, std::less<int>, std::allocator<std::pair<int const, odb::dbTechVia*> > >&, bool) in openroad
 8# grt::RepairAntennas::makeNetWire(odb::dbNet*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >&, std::map<int, odb::dbTechVia*, std::less<int>, std::allocator<std::pair<int const, odb::dbTechVia*> > >&) in openroad
 9# grt::RepairAntennas::makeNetWires(std::map<odb::dbNet*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >, grt::cmpById, std::allocator<std::pair<odb::dbNet* const, std::vector<grt::GSegment, std::allocator<grt::GSegment> > > > >&, int) in openroad
10# grt::RepairAntennas::checkAntennaViolations(std::map<odb::dbNet*, std::vector<grt::GSegment, std::allocator<grt::GSegment> >, grt::cmpById, std::allocator<std::pair<odb::dbNet* const, std::vector<grt::GSegment, std::allocator<grt::GSegment> > > > >&, int, odb::dbMTerm*, float) in openroad
11# grt::GlobalRouter::repairAntennas(odb::dbMTerm*, int, float) in openroad
12# grt::repair_antennas(odb::dbMTerm*, int, float) in openroad
...

maliberty · 2024-04-05T18:39:19Z

The problem is order wires and is understood.

The-OpenROAD-Project/OpenROAD#4878

workaround for The-OpenROAD-Project/OpenROAD#4878

MichaelBell · 2024-04-10T21:19:36Z

This is causing problems with a number of Tiny Tapeout 6 designs, now that Tiny Tapeout has moved to an OpenLane version with the bug present.

This patch seems to work (or at least resolve the crash) on at least a couple of designs, suggesting the problem might be happening in makeNetWires, as dlmiles suggested above. But none of us in the Tiny Tapeout community fully understand the implications. Any comment on whether this is a good idea would be welcome!

obriensp@d1f0b60

maliberty · 2024-04-10T21:23:13Z

We are in the midst of fixing another bug in the same spot of code so results will probably change again. In general such things are prone to fix one design and break another due to the brittle nature of order_wires. The real solution is still in progress but you should use whatever local methods you can until then.

dlmiles · 2024-04-16T16:28:55Z

obriensp@d1f0b60

This recent fix just landed in OR looks the same to me as the patch from this comment I quote here.

b98aca8#diff-02f10642344ceceb7764f552b99c58c78482dbd86eff6a8c6c17aae6407b7d70R304

This should solve The-OpenROAD-Project/OpenROAD#4878 Comparing the number of failures with previous version vs current version on tt05/tt06 project test set, this version performs slightly better: | Shuttle | Fails 2024.04.04 | Fails 2024.04.22 | |---------|------------------|------------------| | tt05 | 4 | 2 | | tt06 | 11 | 9 |

eder-matheus · 2024-06-05T14:15:09Z

@MichaelBell Could you try the latest version of OpenROAD? We've merged the new antenna checker code that don't use orderWires, so this crash should not be happening anymore.

I've tried to run your attached testcase, but it fails with a missing SDC file.

MichaelBell · 2024-06-05T15:07:19Z

Thanks, I'll take a look.

At a guess of what's going wrong with the SDC file - it should be using base.sdc from my src directory, but that does read_sdc $::env(OPENLANE_ROOT)/scripts/base.sdc, maybe OPENLANE_ROOT isn't set in your environment?

eder-matheus · 2024-06-05T15:09:33Z

Thanks, I'll take a look.

At a guess of what's going wrong with the SDC file - it should be using base.sdc from my src directory, but that does read_sdc $::env(OPENLANE_ROOT)/scripts/base.sdc, maybe OPENLANE_ROOT isn't set in your environment?

Yeah, that's the case. I tried to find the file under openlane/scripts but couldn't see it.

MichaelBell · 2024-06-05T15:47:51Z

Weird, it should be here: https://github.com/The-OpenROAD-Project/OpenLane/blob/master/scripts/base.sdc

eder-matheus · 2024-06-12T22:07:41Z

Weird, it should be here: https://github.com/The-OpenROAD-Project/OpenLane/blob/master/scripts/base.sdc

@MichaelBell It wasn't in the test case you attached. Either way, I'll close this issue for now, but if you have any other crashes, feel free to reopen it or create a new issue.

maliberty assigned luis201420 Apr 1, 2024

maliberty added the ant Antenna Checker label Apr 1, 2024

urish added a commit to urish/tt06-spell that referenced this issue Apr 9, 2024

fix: workaround for repairAntennas crash

13d5b9d

The-OpenROAD-Project/OpenROAD#4878

urish added a commit to urish/tt06-spell that referenced this issue Apr 9, 2024

fix: workaround for repairAntennas crash

2b975d4

The-OpenROAD-Project/OpenROAD#4878

urish added a commit to urish/tt06-spell that referenced this issue Apr 9, 2024

fix: disable Repair Antennas step

8b20b88

workaround for The-OpenROAD-Project/OpenROAD#4878

eder-matheus closed this as completed Jun 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Crash in GlobalRouter repairAntennas #4878

Crash in GlobalRouter repairAntennas #4878

MichaelBell commented Mar 30, 2024

maliberty commented Apr 1, 2024

maliberty commented Apr 1, 2024

dlmiles commented Apr 5, 2024 •

edited

maliberty commented Apr 5, 2024

MichaelBell commented Apr 10, 2024

maliberty commented Apr 10, 2024

dlmiles commented Apr 16, 2024 •

edited

eder-matheus commented Jun 5, 2024

MichaelBell commented Jun 5, 2024

eder-matheus commented Jun 5, 2024

MichaelBell commented Jun 5, 2024

eder-matheus commented Jun 12, 2024 •

edited

Crash in GlobalRouter repairAntennas #4878

Crash in GlobalRouter repairAntennas #4878

Comments

MichaelBell commented Mar 30, 2024

Describe the bug

Expected Behavior

OpenROAD Environment

OpenLane Environment

To Reproduce

Relevant log output

Screenshots

Additional Context

maliberty commented Apr 1, 2024

maliberty commented Apr 1, 2024

dlmiles commented Apr 5, 2024 • edited

maliberty commented Apr 5, 2024

MichaelBell commented Apr 10, 2024

maliberty commented Apr 10, 2024

dlmiles commented Apr 16, 2024 • edited

eder-matheus commented Jun 5, 2024

MichaelBell commented Jun 5, 2024

eder-matheus commented Jun 5, 2024

MichaelBell commented Jun 5, 2024

eder-matheus commented Jun 12, 2024 • edited

dlmiles commented Apr 5, 2024 •

edited

dlmiles commented Apr 16, 2024 •

edited

eder-matheus commented Jun 12, 2024 •

edited